Deep learning questions and resources

Question

Hi all,

I was wondering if anyone had any paper or book recommendations going into some of the advanced settings of the Torch add-in?

For example, what is X and Y slide sigma? How to pick the L1 value? etc.

Any recommended resources going over the combination of Image and Tabular models would also be greatly appreciated.

Best,

Clovis

russ_wolfinger · Accepted Answer

Thanks for the question @cweisbart  .   The add-in is a combination of several major components and there likely isn't a single great definitive reference currently, although we plan to continue expanding training materials over time.   Some ideas for now:

- Mousing directly over the label for any parameter should give a brief pop-up description.  e.g. for X Slide Sigma: "Standard deviation of Gaussian random shift along the X axis".

- In general, all of the advanced options in the third column are for dynamic image augmentation during training.   Augmentation is a powerful deep learning method for creating pseudo data for each batch. e.g. flipping a picture of a dog or cat horizontally still can be labeled as a dog or cat, and the model is able to see a wider range of possible images.   Likewise for other image manipulations like shifting, rotating, blurring.  Models trained with augmentation tend to generalize better.

- The L1 Penalty is analogous to the lasso penalty in GenReg and tends to sparsify the solution.    Good values to try are 1e-6 and 1e-5.   Likewise Weight Decay is like a ridge penalty, and the same two values often work well.

- The combination of Image and Tabular models is somewhat unique to this add-in.   We did it this way to provide ability to mix-and-match more flexibly.   The Tabular model is always run after the Image Model and takes embeddings from the Image Model as input.

- The doc (click Add-Ins > Torch Deep Learning > Help) has some additional details on pages 28-31.

- We've tried to use fairly standard terminology throughout so web searching a particular term will likely provide helpful results, background, and even examples.   Any hits from the PyTorch doc itself should be very solid.

Please feel free to send more feedback or questions anytime, and don't forget we have the TDL and new campfire servers available if you want to try some faster GPUs

russ_wolfinger · Accepted Answer

@cweisbart    Thanks for reminder--the "_optim.pt" files are the largest as they contain everything else as well as information on the training itself so it can be resumed.    They are used when specifying "Restore From" in the Torch dialog, but are not needed for deployment since we save the main model components separately.