Deep learning questions and resourcesHi all,
I was wondering if anyone had any paper or book recommendations going into some of the advanced settings of the Torch add-in?
For example, what is X and Y slide sigma? How to pick the L1 value? etc.
Any recommended resources going over the combination of Image and Tabular models would also be greatly appreciated.
Best,
Clovis
Re: Deep learning questions and resources@cweisbart Thanks for reminder--the "_optim.pt" files are the largest as they contain everything else as well as information on the training itself so it can be resumed. They are used when specifying "Restore From" in the Torch dialog, but are not needed for deployment since we save the main model components separately.
Re: Deep learning questions and resources@russ_wolfinger any feedback on the "_optim.pt" files would be super appreciated! Thanks in advance for the help!
Re: Deep learning questions and resourcesThanks for the response, very helpful! I was scanning the help pdf about the stored .pt files. I see that the "inet"s are the image backbones, the "tnet"s are the embedding layers, and the "linp"s the final layers. I also have some "_optim.pt" files. What do those represent? Thanks in advance!
Re: Deep learning questions and resourcesThanks for the question @cweisbart . The add-in is a combination of several major components and there likely isn't a single great definitive reference currently, although we plan to continue expanding training materials over time. Some ideas for now:
- Mousing directly over the label for any parameter should give a brief pop-up description. e.g. for X Slide Sigma: "Standard deviation of Gaussian random shift along the X axis".
- In general, all of the advanced options in the third column are for dynamic image augmentation during training. Augmentation is a powerful deep learning method for creating pseudo data for each batch. e.g. flipping a picture of a dog or cat horizontally still can be labeled as a dog or cat, and the model is able to see a wider range of possible images. Likewise for other image manipulations like shifting, rotating, blurring. Models trained with augmentation tend to generalize better.
- The L1 Penalty is analogous to the lasso penalty in GenReg and tends to sparsify the solution. Good values to try are 1e-6 and 1e-5. Likewise Weight Decay is like a ridge penalty, and the same two values often work well.
- The combination of Image and Tabular models is somewhat unique to this add-in. We did it this way to provide ability to mix-and-match more flexibly. The Tabular model is always run after the Image Model and takes embeddings from the Image Model as input.
- The doc (click Add-Ins > Torch Deep Learning > Help) has some additional details on pages 28-31.
- We've tried to use fairly standard terminology throughout so web searching a particular term will likely provide helpful results, background, and even examples. Any hits from the PyTorch doc itself should be very solid.
Please feel free to send more feedback or questions anytime, and don't forget we have the TDL and new campfire servers available if you want to try some faster GPUs
