building multiple neural network modelsI am using the neural platform to make predictions, and, being new to neural networking, I am only slightly familiar with all the input parameters: number of hidden layers, number of nodes/hidden layer, boosting models, learning rate, and tours. What I want to do is try to minimize RMSE and the validation model misclassification rate. What I've been doing is iteratively changing each parameter one by one, saving the model performance parameters, and pasting them into a new JMP table, but this is going to take days since there are so many combinations of layers, nodes, tours, etc. Would it be possible to write a script to where JMP Pro builds, say, 1,000 models and dumps the data into a table so that I don't have to manually change each model input parameter?
#Hidden layers: 1 or 2
#Sigmoidal nodes: 0, 1, 2, 3, or 4
#Linear nodes: 0, 1, 2, 3, or 4
#Radial nodes: 0, 1, 2, 3, or 4
#boosting models: no idea, 1-5 maybe?
#learning rate: 0-0.5 in 0.1 increments
#tours: 1, 5, 10, 20, 100
So that would be 2 x 5 x 5 x 5 x 5 x 5 x 5=~30,000.
I guess I could widdle a few of these down once I am more familiar with the dataset. Of course, having many nodes and a high number of boosting models doesn't make sense (nor do certain other combinations), but we're still talking about potentially hundreds of models worth testing. Surely this sort of model screening/comparing could be scripted, right?
Predictive Modeling and Machine LearningRe: building multiple neural network models@SDF1 I saw your message, but I got a strange error message when trying to respond: abm64@miami.edu.
BTW, after weeks of not using it, the last version of the GUI now works on my 16-inch Macbook Pro without any of those boundary box-related issues (using both continuous and categorical Y with all the different validation types). I don't recall updating anything on my computer, but hey, I'm just glad it works on this computer, too (it always worked on my Macbook Air, it's on its last legs!).
Re: building multiple neural network modelsHi @abmayfield ,
Interesting, well, it certainly seems as if this issue is truly related to the Macbook Pro (and perhaps OS that it's running). It possibly could be OS or maybe even hardware related -- like how JMP uses the graphics processing unit on a Pro is somehow different than how it uses it on an Air. Maybe check to see if Mac made the two computers with the same GPU or if could be a hardware issue. At any rate, I would imagine JMP developers would want to know more about the specifics you're facing, and bringing it to their attention at the Discover Summit would be helpful. At least it's not my incompetence! 
I'm glad you're still able to run it on your Air. I hope your Air continues to work so you can make your video and submit it for the Summit. Let me know you're time slot for the talk, would like to see it.
Thanks!,
DS
Re: building multiple neural network modelsYou know what I've been learning this week, which is literally the most time I've ever spent on my work Macbook Pro? There are LOTs of small, annoying, but mostly trivial bugs in JMP Pro 16.1 that are completely absent on my Macbook Air. And all have to do with aesthetics. For instance, you know how you can group scripts on the left, then click to expand the grouped folder? That doesn't work. Well, it works sometimes. You need to click the expand arrow, then click somewhere else on the screen, return, click again, and it eventually expands. There are other small, weird things like that they I never noticed on the same OS and same version of JMP Pro on my other Mac, so it must be some issue with JMP and this particular computer communicating with one another, which then affects your script since it seems to have something to do with appearance. In other words, I don't think it's because of your incompetence, haha, and I fully expect to still use the new GUI on my other computer. In fact, it is running through 2,000 NN models as I type this on my work computer. I just need to eventually figure out these Macbook Pro issues given the decaying nature of my Macbook Air. In fact, I may start logging all these little bugs and quirks and try to talk to some developers at the Discovery Summit. I wonder if it's truly MY computer or a general Macbook Pro issue. I don't want to upgrade to the newest Mac OS (Big Sur), but I wonder if this could actually be the solution.
Re: building multiple neural network modelsHi @abmayfield ,
Sorry to hear that you continue to get the error. The "GenAT_ver3.0.jsl" file I uploaded is the one I'm using. I have tried it on my work (JMP 16.0 Pro) and personal PC (JMP EA17), and it works on both. I don't get the error on either PC when trying to fit your data. It must be something about how the Macbook Pro deals with the fit report window. I am stumped on how to fix it without having access to the specific computer. I'd have to look at the tree structure of the report window and try and figure out what the difference is between a PC, Macbook, and Macbook Pro.
I went back and took a closer look at your log file and thought maybe it was something about the specific DOE that you generated, but even when I run that DOE and make a tuning table based on it, it works. At least it works on the PCs I'm using.
That's to bad, as it sounds like you might not be able to demo the GUI for your talk if it's not running on the Macbook Pro. I wish I could be of more help on this issue.
DS
Re: building multiple neural network modelsAnother thing that might be informative: even if I have a continuous Y, I get the same message (regardless of validation type). Strange!
Re: building multiple neural network modelsI just tried on the Macbook Pro and got the same outline box/arg 1 error, but I noticed something: the July version was named "3.0." Do you think you updated it but then accidentally uploaded the old version?
Re: building multiple neural network modelsAwesome! Let me try it now and report back. As it turns out, JMP support DID come to the same conclusion, so I feel confident it will work now.
Re: building multiple neural network modelsHi @abmayfield ,
Thanks for files, it helped me to figure out where the mistakes were coming in. It has to do with my mediocre programming and complete understanding of the tree structures within JMP (I'm all self taught at this). Anyway, the date for the latest version is just about right, I had updated things around early July to account for the fact that the fit statistics change their position in the tree structure based on what kind of modeling data the Y values are, e.g. numerical, ordinal, or categorical. I'm working with purely numerical responses almost 100% of the time, so I didn't see this error on a regular basis. In order to correctly provide information on the fit statistics for other modeling types of data, I needed to modify how it assigns that variable. Thankfully, it was all rather logical from the JMP side, so the modifications weren't that hard.
Attached is the ver3.0 that should fix this no matter your OS or JMP version you have. I don't think support will be able to help much since this is a "third party" code in a sense. Anyway, please give this ver3.0 a try and let me know if it continues to misbehave.
Thanks!,
DS
Re: building multiple neural network modelsI tried JMP Pro 15 and still got the same error message:
expected character arg 1 in access or evaluation of 'Outline Box' , Outline Box/*###*/(o_box)
I am using GUI version 3 from around July 10th or so. That was the most recent, right?
When I ran the debugger, it told me: "Object Button Box does not recognize the message send; maybe you mean one of these..." then it lists some alternatives. But the Button Box forms fine, and all the buttons work.
I would think your tree structure argument COULD be the issue but it's the same OS (Catalina). I'm going to reach out to technical support.
Re: building multiple neural network modelsI am attaching the log and the table with the GUI (the last script). I tried all the validation types that made sense for several of the things I am trying to predict (the Y's). Normally, I would just do everything on my personal Macbook (where everything works), but that computer is weeks away from biting the dust, so I hope i can get everything working on this newer, nicer Macbook Pro! In the mean time, I am going to try a few things. There is a slight chance that I might have just copied and older version of the GUI (that still had bugs), which would be a simple fix, though I don't think that is the issue since, regardless, it worked on the other computer. I'll be curious if you find anything weird, and thanks again for looking into this.
Re: building multiple neural network modelsHi @abmayfield ,
I have to say, that is some strange behavior for it to work with an Air and not a Pro. If I remember correctly, that o_box call is to extract some of the statistics of the fit and it can change depending on which type of validation method you're using: holdback, excluded rows, etc. It doesn't sound like it's a JMP version since you're running 16.1 on both. Are you using the same validation method for both computers? Which method(s) are you using?
My only guess is that if it is not working from one Macbook model to the next that there must be a change in the tree structure of the report and the JSL code is looking for a value to be in a specific location, but it's coming back empty or NAN, or something and it throws the error. This is maybe a bit beyond my coding skills -- to make it OS-independent might be beyond me. I'm running on a PC, Windows10. I have no doubt there are more skilled programmers at JMP that could figure out how to get around this issue so that it can run on different OSs and so forth.
If I can reproduce the error, then it's relatively easy for me to fix, but I don't have access to Macbook Pro. In the meantime, can you run it on the Macbook Pro and check the log file to see if it will tell you which line of code it is breaking down at? This might help me to narrow down the location of the bug and maybe see how to fix it.
Thanks!,
DS
Re: building multiple neural network modelsGreat. Thanks so much. I did notice something weird today: on my 13-inch Macbook Air running Catalina and JMP Pro 16.1, everything works. I then migrated over to my work Macbook Pro 16-inch, also running Catalina and JMP Pro 16.1, used the exact same data table and your GUI, and I get that weird "expected character arg 1 in access or evaluation of 'Outline Box', Outline Box/*###*/(o_box)" regardless of what I enter. Do you think it could actually be linked to the display (i.e., the larger screen)? Surely not. It seems strange that a script would run on one computer and not the other. Any ideas?
Re: building multiple neural network modelsHi @abmayfield ,
No problem, I'm glad it could help. Yeah, tuning the XGBoost can be a bit more involved. I found a blog here that is very useful -- last blog entry on that page, but for some reason link to the blog article is not really responding. It gives an example of how it can be done in Python, but it's essentially the same in JMP, once you've figured out the jargon for the model parameters. I also found this site and accompanying documentation VERY helpful in reading up on what XGBoost does and which parameters make it more or less conservative of a model. One thing I like about it, is that it's much more stable than boosted tree or random forest. Both of those have some stochastic behavior in R^2 resulting in models that differ slightly each time you run the fit with all the parameters being the same. XGBoost gives you the same fit quality each time you run the same set of parameters.
Please feel free to mention the NN GUI in your Discovery talk -- I plan on attending virtually, and was going to try and go to the in-person visit in Cary (I'm just an hour away), but I think with the continuing problems from the pandemic, they stopped the in-person meetup. I have posted the GUI to the add-ins page, so technically, I guess it's "published" and out there. Thank you for asking and thinking of maybe even demo-ing it. I hope it runs smoothly! I'm not the best programmer, but I can get things to work. If you run into any issues, please let me know and I'll try to correct it ASAP.
Thanks!,
DS
Re: building multiple neural network modelsWow! Thanks so much for the table and the explanation. I was thinking that, since I am finally familiar with all the NN jargon, XGBoost would be a breeze, but it seems like I still have a lot to learn (just to get a hang of the jargon for one thing)! It seems like XGBoost is sort of the "darling" of the machine learning world at the moment, but, more importantly, that it might actually be better suited for the types of data I generate, so I definitely need to do a deep dive into all this (for instance, while your GUI is running in the background!). Speaking of which, would you mind if I mentioned the NN GUI, and very likely even demo it, in my JMP Discovery talk (which I am recording this week)? I would be sure to give you credit, but I totally understand if you want to be the one to unveil it to a larger audience and wouldn't want to "steal your thunder." I would be using the neural platform only since I doubt I will be the master of XGBoost by Friday (when I record the talk), haha.
Re: building multiple neural network modelsHi @abmayfield ,
I have not upgraded to 16.1 yet, so I have not experienced the same error, however I will keep my eyes open in case it happens and post any update if needed.
As for XGBoost, i do have a tuning table to optimize that one. It's attached here. The XGBoost platform has many different tuning parameters that can be modified. Some are more influential than others. I typically follow a step-wise tuning of the model because it's just such a big parameter space. I do not have all possible parameters for the tuning table, just the most influential ones (you can always add more with the appropriately names columns). You can access the whole list of parameters in the platform by opening up the "Advanced Options" submenu in the XGBoost window. Since you mentioned that you have a highly imbalanced data set, you'll want to make sure that the scale_pos_weight parameter is set to 1.
In fact, you can even run a tuning design from the platform itself, where it will use the space filling DOE to generate the tuning table. It's pretty handy and flexible. However, I find that for CPU use and RAM minimization, running it like how my GUI works is better from within JMP.
I've found that for my situations, following these steps works really well for XGBoost:
1. Fix learning rate, max_depth, min_child_weight, gamma, subsample, and colsample_bytree to first find the right number of trees. These other parameters will be tuned later. I set them to their default values at first.
2. Tune max_depth and min_child_weight
3. Tune gamma (it's a good idea to re-tune the number of trees before going to step 4)
4. Tune subsample and colsample_bytree
5 Tune alpha and lambda, the regularization parameters
6. Tune Learning Rate: lower the learning rate and increase number of trees.
At each step, I generate a tuning table for the parameter that I wish to tune while keeping the others fixed at what they were for that step. I typically run through this sequence twice. It's critical that you use cross validation (either K-fold or validation column) when running XGBoost. You can run the fit with no cross validation, but then you risk overfitting the data and not really having that great of a predictive model in the end.
Hope this helps!,
DS
Re: building multiple neural network modelsNevermind on point #1. I think was something weird on my end, or that a suitable stratification of the samples into training, validation, and test. Basically, I have a totally unbalanced design in which one group I am trying to predict is much more common, so I think this is causing the issue!
Re: building multiple neural network modelsI started to use your NN GUI again after upgrading to JMP Po 16.1, and I stated to get the "subscript range" error again. Have you encountered this issue?
On another note, do you have a DOE script for the XGBoost tuning table? I am going to explore that approach with your GUI, as well! Thanks so much
Re: building multiple neural network modelsThanks so much for all the info. I indeed would totally have tried to build tons of models without having any idea what I was doing. Two years ago I did this with another approach and was on the verge of trying to write a Nature article, thinking I'd changed marine biology as we know it, before I realized that I was interpreting the model completely incorrectly, haha. Thankfully, AI and machine learning models are all the rage at the moment where I work, so, even though my motivation for using them was driven by the traditional approaches NOT cutting it (rather than my wanting to do AI because it's "sexy"), I hope I can use this wave of interest in these sorts of analyses to try to learn all the ins and outs.
Re: building multiple neural network modelsHi @abmayfield ,
As far as the XGBoost add-in, you'll need to download and install the add-in separately. It works from JMP Pro 15 and up. You can find the add-in here. Be sure to read the documentation and try some of the test modeling that is explained by Russ in the post before moving on to doing your own modeling. Right now, that platform is still one which requires a fair bit more understanding from the user on how to modify/manipulate the hyperparameters of the fit. I found this blog to be very helpful when it comes to tuning the XGBoost platform (there are MANY different parameters to tune, which makes the whole space quite cumbersome). Although it's written for python users, the same basic idea works well for the JMP platform. This video is also helpful, and you'll also want to read up on what each parameter in the model does, what its working ranges are, and why you may or may not want to modify it from the default value here. It is definitely not a platform for just diving into without doing some background homework. I found this out the hard way by first diving in and realized I didn't know what I was doing. So, I did some homework and it helped a lot. Now, I can tune models pretty quickly using the framework from the blog I listed above.
Good luck!,
DS
Re: building multiple neural network modelsThanks so much for explaining that, and I'm glad I asked because my assumption was wrong. It makes more sense, though, that each "run" would be a random shuffling in terms of which samples are in the training model and which are held back. As you stated before, with my small, wide dataset, it makes more sense to manually exclude rows, or else create a validation column, so really I'm just trying to make sure I "master" all the various options since hopefully in the future, I WILL have a dataset in which holdback and kfold could be employed. I especially think the latter is attractive/clever.
I have un about 15,000 models with your GUI, no exaggeration. Now I think I'm going to play around with comparing NN vs. XGBoost, a technique I never heard of at all until your GUI. I don't even know if it's in JMP Pro 16 (though maybe it's hidden somewhere)!
Re: building multiple neural network modelsHi @abmayfield ,
Sorry for the late reply, I was out on vacation last week.
With regard to your question about the holdback portion, it is my understanding that JMP will randomly select the percentage of rows based on your holdback value. So, if you have 100 rows and select 0.3 as the holdback portion, each time JMP runs the NN platform, it will select a new random set of 30 rows as the validation portion. So, in terms of using the tuning table, the platform is called each iteration where it sends to the NN platform a new set of model parameters. If you've chosen the holdback option for validation and put a number in ranging from 0 to 1, then each new run in the tuning table, JMP will select a different set of rows according to the holdback portion. However, if you are running multiple tours during a single NN model fit, then it will use the same holdback rows during each tour. Each time the NN platform is called, it will generate a new subset for validation. This is my understanding, at least.
For the above situation, it is not possible to know which rows are held back (at least not that I'm aware of). If you wanted to do something like that, then I would choose the excluded rows holdback validation option. You can then select which rows in your data table should be excluded and only those will be used for validation. Presumably, you could feed the NN platform a specific random seed, say 10191, for each tuning run with a holdback portion and it will use that random number each time to generate the subset for validation. Since it's starting from the same random seed each time the NN platform is called, in theory JMP should end up selecting the same rows for each tuning run. However, in this case you still wouldn't know which rows those were. Again, this is my understanding of how it it works.
Hope this helps!,
DS
Re: building multiple neural network modelsI have now run literally 10,000 odd models thanks to your epic GUI. I did have one question regarding holdback. Say my tuning table has 100 rows (i.e., different models to test). Does each row get a random holdback based on the percentage or do the same X number of samples get held back for each model? I am guessing it's the latter. In other words, if samples A, B, C, and D are held back from model #1, those same four samples get held back from model #2. I guess regardless my followup question would be: is it possible to KNOW which samples get held back? In any event, as we discussed, in my case having a stratified validation column really makes more sense because I can ensure that at least one sample from each treatment is included as training, validation, and test, whereas a random 30% holdback could theoretically ONLY hold back samples from the SAME treatment (which would bias the prediction in favor of said treatment!
Re: building multiple neural network modelsHi @abmayfield ,
I'm really glad to hear that it's working for you and I'm hopeful the bugs have been ironed out. I wish I had a better solution for the top of the pile data table issue. I honestly don't know how to get around that.
Anyway, I do believe you are right about Kfold xvalidation. That method typically works better with tall tables and not so great with wide tables. If you have very few data entries, then using a validation column stratified on the response is a good idea. Or, if you use the GenReg platform, you can use the leave-one-out approach. But typically, Kfold is not the best if you have few data points to consider.
I would definitely recommend looking into XGBoost, you can download the add-in from the add-in page on JMP. It's a very stable and robust algorithm. XGBoost works really well with a validation column.
You might consider also looking into support vector machines, which JMP Pro has. It was originally developed for classification problems, but can also handle regression problems. In any case, it's always a good idea to compare different models using a hold out data set that has not been used in training or xvalidation of the model. I typically build several different models and then look at how they compare on a completely new data set to see which is the best at predicting the outcome.
Glad I could help!,
DS
Re: building multiple neural network modelsThis is SO awesome. I am doing batches of 100 iterations for each validation method and have yet to have encountered any error. I have not tried "excluded rows" yet but am confident it will work.
kfold always give me the same, perfect (overfit?) solution, though I think this is a testament to the size/format of my dataset, rather than any sort of error with the script.
Scientifically, I have been doing some research, and it appears if my "small" datasets (dozens of samples and hundreds of potential predictors) are probably more amenable to XGBoost, or else neural nets may be overkill, so I am going to try that platform, too. I didn't even know XGBoost was supported in JMP at all (whereas I have seen random forests and likely the other approach in your GUI). In other words, I look forward to taking advantage of ALL components of this excellent research tool! Thanks so much for your help (again), and I'll let you know if I encounter any other errors.
Re: building multiple neural network modelsHi @abmayfield ,
So, I found out what the problem was in my coding. It wasn't an issue from one version of JMP to another, it has to do with how many levels there are for a nominal or ordinal response. This changes the location of the column number box in the report window. I needed to adjust the code to reflect that when going from a response that has 2 or 3, or 4, or 6 levels, the statistics for the fit changes location within the report. The nice thing is that is changes at a fixed amount related to how many levels there are, so I was able to relatively easily fix the code. It took me a while to figure out that bug though because I had used the PowderMetallurgy file from the sample data, but it is only two levels, not 4 like yours is.
Attached is the updated version that is more generalized to different levels of the response variable. The code should already work for 2-level or 3-level validation columns as well as nominal, ordinal, and continuous responses.
I tested out all fit platforms: NN, BT, Bootstrap Forest, and XGBoost with your data set and it works properly across all data types and the different validation options. Please note that at present, JMP can only fit a two-level response (nominal or ordinal) using the boosted tree platform. The other platforms can handle more levels than that.
I'm pretty sure it's now working as intended (except for defaulting to the last opened data table -- I do not know how to get around that). However, if you do find a bug in it, please let me know.
Thanks, and good luck!,
DS
Re: building multiple neural network modelsHi @abmayfield ,
I have that same issue with my data tables. I think you're right, if you don't run it directly from the data table your modeling from, it somehow defaults to the last opened data table in your JMP home window. I do not know how to get around this.
Hmm, the code should work with a validation column that is only training and validation or all three. I wrote it to check the number of levels in the validation column and adjust accordingly, meaning to record the appropriate fit statistics whether there are two levels or 3. I will double check that one tomorrow. I think the subscript range error is likely due to the changes between 15 and 16. When 16 came out, I did not go back and double check the code to make sure it worked, but I ran it through multiple tests in 15 and everything worked. Nonetheless, I will fix this and get things corrected so that it works as expected, I just won't have time until Friday morning to get to it.
Again, thanks for helping me debug the code, and sorry for the inconvenience. I'll get back to you when I've checked it for all the data types and code types like I did for submitting it as a JMP Add-in.
Thanks!,
DS
Re: building multiple neural network modelsAWESOME! Now we're cooking with fire. The only thing I see now is that, even though the GUI asks you to tell it which is the data table and which is the tuning table, it still sometimes gets them mixed up. I think it defaults to whichever table is closer to the "top of the pile" (I tend to have 10-20 tables open, or else this would likely not be a problem).
Would it be possible to add a simple "rule" that says "If the validation column has only training and validation terms, do not calculate the test data" or something along those lines? I guess the alternative is to go through the code manually and put the "//" in front of all the "test" options. I ask because sometimes I use training and validation ONLY, rather than training/validation/test (since my sample sizes are small). if you enter a validation column with ONLY training and validation terms now, you get the "subscript range" error, and I think it's because it's looking for "test" data that are not there!
Re: building multiple neural network modelsHi @abmayfield ,
Thanks again for catching that and letting me know -- my models tend to have numerical responses, so I don't usually need to edit things for nominal data types.
Anyway, I think there might have been a change from JMP 15 to 16 that lead to different number column boxes in the report windows because I needed to change all the same values from what they were to what they are now, and it worked before in JMP 15, I recall testing it out. I've re-checked, and this updated version correctly runs with the different validation methods for NN: excluded rows holdback, holdback, kfold, and validation column. Sorry once again for the inconvenience. I am glad you can test it out and let me know of the bugs so that it can get better and help others. Please let me know if you find anything else!
Thanks!,
DS
Re: building multiple neural network modelsThanks for looking into that, and it does work now with a validation column, but I get the same error code when I use holdback or kfold. I wonder if it's because those methods don't necessarily stratify the data to where each bin I am trying to predict is represented? I'll try it with a bigger dataset and see if I get the same "subscript range" error.
Re: building multiple neural network modelsHi @abmayfield ,
Sorry it took a bit to get back to you. I was able to reproduce the error you were getting and found where the mistake was in my code. lines 348 and 349 had the wrong values for the variables ncol_box_v and ncol_box test. They should be 10 and 19, respectively. I've corrected that and re-ran the NN with the validation column, and it works as it should. I'm attaching the corrected code. Sorry for the inconvenience. Thanks for catching that, and if you find any other bugs, please let me know.
Hope this helps!,
DS
Re: building multiple neural network modelsWow, this GUI has really gotten an upgrade! The only thing is that now I get an error of: "
Subscript Range in access or evaluation of 'V_stats[ /*###*/5]' , V_stats[/*###*/5" regardless of the validation type I use. I wonder if it's because I have such a small sample size for trying to have training, validation, and test samples for each of four groups (as Y's)?
Re: building multiple neural network modelsHi @abmayfield ,
No, I'm guessing that it's because you selected to have a validation column for the radio button selection (vs holdback or holdback excluded, etc.) but forgot to include the validation in the list of columns when you cast the columns into their selected roles. That's my guess, but if it's not that, then I might need you to provide a step-by-step description.
I have also updated the code somewhat to correct for a few other mistakes that were being made. My code includes the progress window, which I think you didn't want, but anyway, I'm attaching the new GenAt.jsl file so you can have the source code. I use it all the time to assist with my modeling -- NN, boosted tree, random forest, and XGBoost. I like it, but it does require some knowledge/understanding of what you're doing from a modeling perspective and what the different platforms allow you to do.
I don't know if this link will work, but I tried writing up a description on how to use it for the Add-ins repository. I know I can get to that link through my SAS account, just not sure if others can. It's not yet made public on the repository, though.
Hope this helps!,
DS
Re: building multiple neural network modelsHello!
I am back at this NN work after a 6-month hiatus, and I noticed that when I try to run your NN GUI, I get a new error message: "Send expects Scriptable Object in access of evalution of 'list', {/*###*/"Validation"})." Do you think this might be because of some change between JMP 15 and 16 (i.e., certain JMP 15 scripts need to be modified for JMP 16?
Re: building multiple neural network modelsThat is great! Let me know when you post it. I will definitely try it out, especially since I'm still in the early stages of figuring out this dataset (though NN was the clear winner over all other model types I tested on JMP Pro 16). I do see the value of not going DIRECTLY to NN because of the reasons you stated AND because it's not great for response variable reduction. In my case, I would need to repeatedly measure my same suite of 30-40 proteins in every sample (with no missing data, as you pointed out), which will drive up costs versus doing a more traditional, candidate biomarker approach (with one or just a few predictors). So I basically make simple, stripped down models with just a few biomarkers (typically using Gen-reg) and compare them to the optimized NN model from your script. In THIS instance, NN is superior, but that's not to say it ALWAYS will be from every future dataset I will collect! Looking forward to trying out the GUI, bugs and all.
Re: building multiple neural network modelsHi @abmayfield ,
I definitely agree with you about not having overly complex models. That's absolutely an issue that should be avoided.
My take on things is that NN modeling is not too dissimilar than other platforms. Each one has it's benefits and drawbacks. For example, NNs are not that great at handling missing data, they're not very computationally scalable, and their notoriously difficult to interpret. For people using such models to help build theories, that interpretation is important. NNs also can't handle all types of data sets or irrelevant inputs. But, there are no assumptions going into the model, and they tend to have a high predictive ability.
But, since each modeling method has a different set of benefits/drawbacks, I tend to model across different platforms and then compare the best of the models from each platform against each other on a new data set to see which perform the best at predicting the new data set and use that one.
That being said, I just finished Ver 1.0 of a GUI interface that can be run from a JMP menu hot button (not just from within the data table). You feed it your data table and tuning table, and depending on which of the four methods you choose (NN, boosted tree, bootstrap forest, XGBoost), it will run the number of fits you send it based on the tuning table. It'll work across the different data types for the Y input and across different validation column types, even multiple validation columns for the XGBoost platform. It's by no means bug free, but I'm pretty excited I got it to work within a week or so of posting the first GUI for the NN platform. It does require the user to have knowledge of which tuning table to use for each platform and other platform specific requirements -- like one must know that the boosted tree platform in JMP doesn't work with more than 2-level nominal/ordinal data (at least with JMP 15) or that the XGBoost platform can use multiple validation columns, but each must be only 2-levels.
It was fun working on this project and will help me in future efforts. Thanks!
Good luck,
DS
Re: building multiple neural network modelsI think part of this may have to do with the fact that, for many modeling types, building ever-more complex models is NOT what you want to do: if a simple model fits your data well, and the validation or test RSME values are low, then just stop there. But for NN, adding layers, nodes, boosts, tours, etc., is not necessarily a BAD thing because, although more complex, they are also more flexible. So I can kind of see why having this GUI for ALL modeling platforms may lead those unfamiliar with modeling to essentially data mine exhaustively and to their detriment. From my limited understanding of NN, though, if one model has a respectable validation RSME under the default conditions of TanH=3, yet another has 3 TanH, 3 Linear, and 3 Gaussian nodes across two hidden layers, yet has a similar RSME, you wouldn't automatically reject the latter model on account of it being more complex yet equally adequate in terms of RSME because it might end up being MORE flexible to future test datasets. Therefore, it could very well be that most people WOULD do well to build potentially hundreds to thousands of NN models. It is entirely possible, though, that I am wrong and NN is no different from other modeling types, in which, all else being equal, simple is better. I'm just spit-balling as to why you don't see more people trying to optimize NN input parameters: is it out of ignorance or fear of data mining?!
Re: building multiple neural network modelsHi @abmayfield ,
Thanks for your feedback and kind words. Yeah, I had that problem too, about it pulling up the wrong data table if it's not "on top". I'm not sure why it does that at all. There's also something funky about how it responds to when I call certain things in the new window for passing on to the fit portion later. The "cancel" button is not working like I intend, and the "help" button doesn't go anywhere right now, but it's a start! I think that a scripting expert could easily fix those bugs and have it work as intended. Perhaps when I get things set up for it to run on more platforms and to not have those bugs in it, I could post it to the add-in page. Would be an honor to have it incorporated into JMP.
It does help me with my work as it'll make the process of tuning my models even more automated and simplified. I had wanted to do something for the NN platform, but was putting it off until your post, and now that I've made the GUI, I'm wanting to expand on that to make it more cross-functional. The ability to tune a large set of models and focus on just the stats really helps to narrow down the hyperparameter space so that it's easier to compare models across different platforms and see which is the best at predicting new data. Then, you can go back and run the optimal settings in the standard platform and get all the rich graphical feedback JMP provides to visualize the model and all the other stuff.
Good luck!,
DS
Re: building multiple neural network modelsWow! Now this is truly great. The only slight glitch is that, if the script window is directly above the "tuning" table, it "calls" the tuning table in the model window instead of the data table. As long as the data table and script are at the forefront of your screen, it loads the model terms correctly. In fact, I just ran 100 models in just a few minutes. I wonder if you can put this on the "add-in" depot (forgot its actual name) on the JMP website so that it gets more widespread attention. I'm certain this would help MANY people (if not immediately, then in the future as more see the utility of NN in model building). Hell, it should probably make its way in some form into JMP Pro 17 (might be too late for 16). Thanks so much for all your help, and I hope this helps your own research, as well.
Re: building multiple neural network modelsHi @abmayfield ,
I hope the JMP and/or computer re-set did the trick. If not, I also went back and built a GUI for the NN modeling -- I plan on expanding it to work across different platforms. It doesn't work perfectly, and I get some strange errors at times (an expert scripter can probably debug it within seconds). But, the nice thing is that it will work with any type of input data column -- ordinal, nominal, and continuous, and it works with the acceptable validation methods for NN -- excluded rows, holdback, Kfold, and validation column. It will work with both validation columns that have Test data in them and ones that don't. It's the script called NN GUI in your data table (attached).
For those interested, I borrowed heavily on a blog from JMP staff .@juliagong (retired) here and a support note from .@Wendy_Murphrey here. I plan on creating a GUI that can automate this for the other platforms I have also automated from within data tables: boosted tree, bootstrap forest, and XGBoost. I plan on posting it in the community, but it could take some time -- especially to make sure it's fully debugged.
The only change from the below code to the GUI in the data table is changing dt = Data Table ("...") to dt = Current Data Table().
I hope this helps to make progress on your work. It does for me as I can improve this to make my modeling efforts easier, too!
Good luck,
DS
NN GUI code:
Names Default To Here( 1 );
//Clear Symbols();
dt = Data Table( "Neural networking Ofav 2017 dataset" );
dt_parms = Data Table( "NN Tuning 8" );
nc = N Col( dt );
dtp_name = dt_parms << get name;
dt_name = dt << get name;
valid_list = {"Excluded Rows Holdback", "Holdback", "KFold", "Validation Column"};
//the width of the column list box in pixels
lbWidth = 110;
/* Expression to clear all current settings */
clearRoles = Expr(
colListY << RemoveAll;
colListV << RemoveAll;
);
/* Expression to store the current settings in global variables */
recallRoles = Expr(
::ycolRecall = colListY << GetItems;
::vcolRecall = colListV << GetItems;
);
VmethodDlg = New Window( "Choose Validation Type",
<<RETURN result=""> 0,
VMeth_Insert = "Validation(" || ":" || validCol[1] || ")";
If(
coltype == "Ordinal",
set = 1;
o_box = 3;
ncol_box_v = 10;
ncol_box_test = 19;
dt_results << New Column( "Generalized R² Training" );
dt_results << New Column( "Generalized R² Validation" );
dt_results << New Column( "Generalized R² Test" );
dt_results << New Column( "Entropy R² Training" );
dt_results << New Column( "Entropy R² Validation" );
dt_results << New Column( "Entropy R² Test" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
dt_results << New Column( "RMSE Test" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
dt_results << New Column( "Mean Abs Dev Test" );
dt_results << New Column( "Misclassification Rate Training" );
dt_results << New Column( "Misclassification Rate Validation" );
dt_results << New Column( "Misclassification Rate Test" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
dt_results << New Column( "-LogLiklihood Test" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
dt_results << New Column( "Sum Freq Test" );,
coltype == "Nominal",
set = 2;
o_box = 3;
ncol_box_v = 10;
ncol_box_test = 19;
dt_results << New Column( "Generalized R² Training" );
dt_results << New Column( "Generalized R² Validation" );
dt_results << New Column( "Generalized R² Test" );
dt_results << New Column( "Entropy R² Training" );
dt_results << New Column( "Entropy R² Validation" );
dt_results << New Column( "Entropy R² Test" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
dt_results << New Column( "RMSE Test" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
dt_results << New Column( "Mean Abs Dev Test" );
dt_results << New Column( "Misclassification Rate Training" );
dt_results << New Column( "Misclassification Rate Validation" );
dt_results << New Column( "Misclassification Rate Test" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
dt_results << New Column( "-LogLiklihood Test" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
dt_results << New Column( "Sum Freq Test" );,
coltype == "Continuous",
set = 3;
o_box = 3;
ncol_box_v = 2;
ncol_box_test = 3;
dt_results << New Column( "R² Training" );
dt_results << New Column( "R² Validation" );
dt_results << New Column( "R² Test" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
dt_results << New Column( "RMSE Test" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
dt_results << New Column( "Mean Abs Dev Test" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
dt_results << New Column( "-LogLiklihood Test" );
dt_results << New Column( "SSE Training" );
dt_results << New Column( "SSE Validation" );
dt_results << New Column( "SSE Test" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
dt_results << New Column( "Sum Freq Test" );
);
);
If( VMeth == "Validation Column" & lvl == 0,
VMeth_Insert = "Validation(" || ":" || validCol[1] || ")";
If(
coltype == "Ordinal",
set = 1;
o_box = 3;
ncol_box_v = 10;
dt_results << New Column( "Generalized R² Training" );
dt_results << New Column( "Generalized R² Validation" );
dt_results << New Column( "Entropy R² Training" );
dt_results << New Column( "Entropy R² Validation" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
dt_results << New Column( "Misclassification Rate Training" );
dt_results << New Column( "Misclassification Rate Validation" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );,
coltype == "Nominal",
set = 2;
o_box = 3;
ncol_box_v = 10;
ncol_box_test = 19;
dt_results << New Column( "Generalized R² Training" );
dt_results << New Column( "Generalized R² Validation" );
dt_results << New Column( "Entropy R² Training" );
dt_results << New Column( "Entropy R² Validation" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
dt_results << New Column( "Misclassification Rate Training" );
dt_results << New Column( "Misclassification Rate Validation" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );,
coltype == "Continuous",
set = 3;
o_box = 3;
ncol_box_v = 2;
ncol_box_test = 3;
dt_results << New Column( "R² Training" );
dt_results << New Column( "R² Validation" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
dt_results << New Column( "SSE Training" );
dt_results << New Column( "SSE Validation" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
);
);
imax = N Row( dt_parms );
For( i = 1, i <= imax, i++,
Nlayer = dt_parms:N_Layers[i];
If( dt_results:N_Layers[i] == 1,
dt_results:N_TanH_2[i] = 0;
dt_results:N_Linear_2[i] = 0;
dt_results:N_Gauss_2[i] = 0;
);
If( dt_results:N_Layers[i] == 2,
dt_results:N_Boosts[i] = 0;
dt_results:Learn_Rate[i] = 0.1;
);
NTH1 = dt_results:N_TanH_1[i];
NTH2 = dt_results:N_TanH_2[i];
NL1 = dt_results:N_Linear_1[i];
NL2 = dt_results:N_Linear_2[i];
NG1 = dt_results:N_Gauss_1[i];
NG2 = dt_results:N_Gauss_2[i];
Nboosts = dt_results:N_Boosts[i];
LR = dt_results:Learn_Rate[i];
TCov = dt_results:T_Cov[i];
RFit = dt_results:Robust_Fit[i];
PMethod = dt_results:Penalty_Method[i];
Ntours = dt_results:N_Tours[i];
str = Eval Insert(
"report = (dt << Neural(
Y( response_col ),
X(
:OFA...22_c0_g2_i1.p1,
:OFA...47_c0_g1_i1.p1,
:OFA...73_c1_g1_i3.p1,
:OFA...72_c1_g1_i1.p1,
:OFA...5_c1_g2_i15.p1,
:OFA...78_c2_g1_i3.p1,
:OFA...51_c4_g1_i1.p1,
:OFA...38_c0_g1_i8.p1,
:OFA...19_c0_g1_i1.p1,
:OFA...22_c2_g1_i4.p1,
:OFA...27_c1_g3_i5.p1,
:OFA...82_c3_g2_i3.p1,
:OFA...11_c1_g1_i7.p1,
:OFA...50_c2_g4_i1.p3,
:OFA...18_c1_g1_i7.p1,
:OFA...66_c2_g1_i3.p1,
:OFA...85_c0_g1_i2.p1,
:OFA...99_c5_g1_i6.p1,
:OFA...03_c2_g1_i1.p1,
:SYM...24_c0_g1_i1.p1,
:SYM...75_c0_g2_i1.p1,
:SYM...42_c0_g1_i1.p1,
:SYM...04_c0_g1_i1.p1,
:SYM...97_c0_g1_i1.p1,
:SYM...13_c0_g1_i1.p1,
:SYM...66_c0_g1_i1.p1,
:SYM...72_c0_g6_i1.p1,
:SYM...13_c0_g1_i1.p1 2,
:SYM...33_c0_g1_i1.p1 3,
:SYM...43_c0_g1_i1.p1,
:SYM...51_c0_g1_i1.p1,
:SYM...65_c0_g1_i1.p1 2
),
^VMeth_insert^,
Informative Missing(0),
Transform Covariates(^TCov^),
Fit(
NTanH(^NTH1^),
NLinear(^NL1^),
NGaussian(^NG1^),
NTanH2(^NTH2^),
NLinear2(^NL2^),
NGaussian2(^NG2^),
Transform Covariates(^TCov^),
Penalty Method(\!"PMethod\!"),
Number of Tours(^Ntours^),
N Boost(^Nboosts^),
Learning Rate(^LR^)
),
Go,
invisible
)) << Report;"
);
Eval( Parse( str ) );
w = Window( dt_name || " - " || "Neural of " || y_resp[1] );
T_stats = w[Outline Box( o_box ), Number Col Box( 1 )] << Get;
V_stats = w[Outline Box( o_box ), Number Col Box( ncol_box_v )] << Get;
If( VMeth == "Validation Column" & lvl > 0,
Test_stats = w[Outline Box( o_box ), Number Col Box( ncol_box_test )] << Get
);
report << Close Window;
If( coltype == "Ordinal",
dt_results:Generalized R² Training[i] = T_stats[1];
dt_results:Entropy R² Training[i] = T_stats[2];
dt_results:RMSE Training[i] = T_stats[3];
dt_results:Mean Abs Dev Training[i] = T_stats[4];
dt_results:Misclassification Rate Training[i] = T_stats[5];
dt_results:Name( "-LogLiklihood Training" )[i] = T_stats[6];
dt_results:Sum Freq Training[i] = T_stats[7];
dt_results:Generalized R² Validation[i] = V_stats[1];
dt_results:Entropy R² Validation[i] = V_stats[2];
dt_results:RMSE Validation[i] = V_stats[3];
dt_results:Mean Abs Dev Validation[i] = V_stats[4];
dt_results:Misclassification Rate Validation[i] = V_stats[5];
dt_results:Name( "-LogLiklihood Validation" )[i] = V_stats[6];
dt_results:Sum Freq Validation[i] = V_stats[7];
If( VMeth == "Validation Column" & lvl > 0,
dt_results:Generalized R² Test[i] = Test_stats[1];
dt_results:Entropy R² Test[i] = Test_stats[2];
dt_results:RMSE Test[i] = Test_stats[3];
dt_results:Mean Abs Dev Test[i] = Test_stats[4];
dt_results:Misclassification Rate Test[i] = Test_stats[5];
dt_results:Name( "-LogLiklihood Test" )[i] = Test_stats[6];
dt_results:Sum Freq Test[i] = Test_stats[7];
);
);
If( coltype == "Nominal",
dt_results:Generalized R² Training[i] = T_stats[1];
dt_results:Entropy R² Training[i] = T_stats[2];
dt_results:RMSE Training[i] = T_stats[3];
dt_results:Mean Abs Dev Training[i] = T_stats[4];
dt_results:Misclassification Rate Training[i] = T_stats[5];
dt_results:Name( "-LogLiklihood Training" )[i] = T_stats[6];
dt_results:Sum Freq Training[i] = T_stats[7];
dt_results:Generalized R² Validation[i] = V_stats[1];
dt_results:Entropy R² Validation[i] = V_stats[2];
dt_results:RMSE Validation[i] = V_stats[3];
dt_results:Mean Abs Dev Validation[i] = V_stats[4];
dt_results:Misclassification Rate Validation[i] = V_stats[5];
dt_results:Name( "-LogLiklihood Validation" )[i] = V_stats[6];
dt_results:Sum Freq Validation[i] = V_stats[7];
If( VMeth == "Validation Column" & lvl > 0,
dt_results:Generalized R² Test[i] = Test_stats[1];
dt_results:Entropy R² Test[i] = Test_stats[2];
dt_results:RMSE Test[i] = Test_stats[3];
dt_results:Mean Abs Dev Test[i] = Test_stats[4];
dt_results:Misclassification Rate Test[i] = Test_stats[5];
dt_results:Name( "-LogLiklihood Test" )[i] = Test_stats[6];
dt_results:Sum Freq Test[i] = Test_stats[7];
);
);
If( coltype == "Continuous",
dt_results:R² Training[i] = T_stats[1];
dt_results:RMSE Training[i] = T_stats[2];
dt_results:Mean Abs Dev Training[i] = T_stats[3];
dt_results:Name( "-LogLiklihood Training" )[i] = T_stats[4];
dt_results:SSE Training[i] = T_stats[5];
dt_results:Sum Freq Training[i] = T_stats[6];
dt_results:R² Validation[i] = V_stats[1];
dt_results:RMSE Validation[i] = V_stats[2];
dt_results:Mean Abs Dev Validation[i] = V_stats[3];
dt_results:Name( "-LogLiklihood Validation" )[i] = V_stats[4];
dt_results:SSE Validation[i] = V_stats[5];
dt_results:Sum Freq Validation[i] = V_stats[6];
If( VMeth == "Validation Column" & lvl > 0,
dt_results:R² Test[i] = Test_stats[1];
dt_results:RMSE Test[i] = Test_stats[2];
dt_results:Mean Abs Dev Test[i] = Test_stats[3];
dt_results:Name( "-LogLiklihood Test" )[i] = Test_stats[4];
dt_results:SSE Test[i] = Test_stats[5];
dt_results:Sum Freq Test[i] = Test_stats[6];
);
);
);</RETURN>Re: building multiple neural network modelsHmmm, I downloaded your new table and now I get the outline box error every time. I am going to try a JMP reboot, then a computer reboot. It seems strange that it would work on your end but not on mine!
Re: building multiple neural network modelsHi @abmayfield ,
I'm reposting this as it seems my original post that I tried to edit got lost somehow. Thanks for sharing the data table, it helped to fix the issues.
Anyway, I made some changes, so a few things to note:
- I made it so a single code can run on Ordinal, Nominal, and Continuous modeling types. Two different versions are no longer needed.
- You will need to edit the variable y_resp, which is the Y response column by entering the column name between the quotes (see line 8). You no longer need to edit it in the call within Neural(). You'll still have to edit the list of X()'s, though.
- The code then tests the modeling type of y_resp column and sets some variables and generates the appropriate columns in the output data table.
- The output data table is now given a name depending on the validation type and host data table. This way you can run all the validation types and then compare without having weird naming issues from JMP.
- I edited each of the different validation type scripts you have in your data table, and they can easily be copy/pasted to new data tables. I tested this on your previous table you sent and it worked just fine.
- I fixed the error my code was writing to the log file each time it ran, had to move a parenthesis to a new location.
- I found out that the reason your holdback and my k-fold weren't actually working correctly, is that when you do that type, you need to change Validation() to Validation Method(). I've done that for your different methods and the scripts are fixed.
So, it's improving bit by bit, so that's nice. I'd really prefer to have it test the response column type and validation method before it generates the extra columns, but I'm not sure how to do that yet. Maybe I'll edit that for the next roll out!
Below is the code example for the Kfold validation. I've also attached your new data table with the corrected scripts saved to it.
Names Default To Here( 1 );
dt = Current Data Table();
dt_parms = Data Table( "NN Tuning 8" );
dtp_name = dt_parms << get name;
dt_name = dt << get name;
//edit the line below so that it has the name of the column you want to model.
y_resp = "health designation";
response_col = Column( dt, Eval( Eval Expr( Expr( y_resp ) ) ) );
dt_results = dt_parms << Subset( Output Table( "Output of " || dtp_name || " with Kfold" || " for " || dt_name ), All rows, Selected Columns Only( 0 ) );
//This tests the response column type to automatically set the values for pulling the data out of the report window.
//It also will create the appropriate columns depending on the modeling type of the response column.
coltype = response_col << Get Modeling Type;
If( coltype == "Ordinal",
o_box = 3;
ncol_box_v = 10;
ncol_box_test = 19;
dt_results << New Column( "Generalized R² Training" );
dt_results << New Column( "Generalized R² Validation" );
//dt_results << New Column( "Generalized R² Test" );
dt_results << New Column( "Entropy R² Training" );
dt_results << New Column( "Entropy R² Validation" );
//dt_results << New Column( "Entropy R² Test" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
//dt_results << New Column( "RMSE Test" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
//dt_results << New Column( "Mean Abs Dev Test" );
dt_results << New Column( "Misclassification Rate Training" );
dt_results << New Column( "Misclassification Rate Validation" );
//dt_results << New Column( "Misclassification Rate Test" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
//dt_results << New Column( "-LogLiklihood Test" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
//dt_results << New Column( "Sum Freq Test" );
);
If( coltype == "Nominal",
o_box = 3;
ncol_box_v = 10;
ncol_box_test = 19;
dt_results << New Column( "Generalized R² Training" );
dt_results << New Column( "Generalized R² Validation" );
//dt_results << New Column( "Generalized R² Test" );
dt_results << New Column( "Entropy R² Training" );
dt_results << New Column( "Entropy R² Validation" );
//dt_results << New Column( "Entropy R² Test" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
//dt_results << New Column( "RMSE Test" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
//dt_results << New Column( "Mean Abs Dev Test" );
dt_results << New Column( "Misclassification Rate Training" );
dt_results << New Column( "Misclassification Rate Validation" );
//dt_results << New Column( "Misclassification Rate Test" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
//dt_results << New Column( "-LogLiklihood Test" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
//dt_results << New Column( "Sum Freq Test" );
);
If( coltype == "Continuous",
o_box = 3;
ncol_box_v = 2;
ncol_box_test = 3;
dt_results << New Column( "R² Training" );
dt_results << New Column( "R² Validation" );
//dt_results << New Column( "R² Test" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
//dt_results << New Column( "RMSE Test" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
//dt_results << New Column( "Mean Abs Dev Test" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
//dt_results << New Column( "-LogLiklihood Test" );
dt_results << New Column( "SSE Training" );
dt_results << New Column( "SSE Validation" );
//dt_results << New Column( "SSE Test" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
//dt_results << New Column( "Sum Freq Test" );
);
imax = N Row( dt_parms );
For( i = 1, i <= imax, i++,
Nlayer = dt_parms:N_Layers[i];
If( dt_results:N_Layers[i] == 1,
dt_results:N_TanH_2[i] = 0;
dt_results:N_Linear_2[i] = 0;
dt_results:N_Gauss_2[i] = 0;
);
If( dt_results:N_Layers[i] == 2,
dt_results:N_Boosts[i] = 0;
dt_results:Learn_Rate[i] = 0.1;
);
NTH1 = dt_results:N_TanH_1[i];
NTH2 = dt_results:N_TanH_2[i];
NL1 = dt_results:N_Linear_1[i];
NL2 = dt_results:N_Linear_2[i];
NG1 = dt_results:N_Gauss_1[i];
NG2 = dt_results:N_Gauss_2[i];
Nboosts = dt_results:N_Boosts[i];
LR = dt_results:Learn_Rate[i];
TCov = dt_results:T_Cov[i];
RFit = dt_results:Robust_Fit[i];
PMethod = dt_results:Penalty_Method[i];
Ntours = dt_results:N_Tours[i];
str = Eval Insert(
"report = (dt << Neural(
Y( response_col ),
X(
:OFA...22_c0_g2_i1.p1,
:OFA...47_c0_g1_i1.p1,
:OFA...73_c1_g1_i3.p1,
:OFA...72_c1_g1_i1.p1,
:OFA...5_c1_g2_i15.p1,
:OFA...78_c2_g1_i3.p1,
:OFA...51_c4_g1_i1.p1,
:OFA...38_c0_g1_i8.p1,
:OFA...19_c0_g1_i1.p1,
:OFA...22_c2_g1_i4.p1,
:OFA...27_c1_g3_i5.p1,
:OFA...82_c3_g2_i3.p1,
:OFA...11_c1_g1_i7.p1,
:OFA...50_c2_g4_i1.p3,
:OFA...18_c1_g1_i7.p1,
:OFA...66_c2_g1_i3.p1,
:OFA...85_c0_g1_i2.p1,
:OFA...99_c5_g1_i6.p1,
:OFA...03_c2_g1_i1.p1,
:SYM...24_c0_g1_i1.p1,
:SYM...75_c0_g2_i1.p1,
:SYM...42_c0_g1_i1.p1,
:SYM...04_c0_g1_i1.p1,
:SYM...97_c0_g1_i1.p1,
:SYM...13_c0_g1_i1.p1,
:SYM...66_c0_g1_i1.p1,
:SYM...72_c0_g6_i1.p1,
:SYM...13_c0_g1_i1.p1 2,
:SYM...33_c0_g1_i1.p1 3,
:SYM...43_c0_g1_i1.p1,
:SYM...51_c0_g1_i1.p1,
:SYM...65_c0_g1_i1.p1 2
),
Validation Method ( \!"Kfold\!", 5 ),
Informative Missing(0),
Transform Covariates(^TCov^),
Fit(
NTanH(^NTH1^),
NLinear(^NL1^),
NGaussian(^NG1^),
NTanH2(^NTH2^),
NLinear2(^NL2^),
NGaussian2(^NG2^),
Transform Covariates(^TCov^),
Penalty Method(\!"PMethod\!"),
Number of Tours(^Ntours^),
N Boost(^Nboosts^),
Learning Rate(^LR^)
),
Go,
invisible
)) << Report;"
);
Eval( Parse( str ) );
w = Window( dt << GetName || " - " || "Neural of " || y_resp );
T_stats = w[Outline Box( o_box ), Number Col Box( 1 )] << Get;
V_stats = w[Outline Box( o_box ), Number Col Box( ncol_box_v )] << Get;
//Test_stats = w[Outline Box( o_box ), Number Col Box( ncol_box_test )] << Get;
report << Close Window;
If( coltype == "Ordinal",
dt_results:Generalized R² Training[i] = T_stats[1];
dt_results:Entropy R² Training[i] = T_stats[2];
dt_results:RMSE Training[i] = T_stats[3];
dt_results:Mean Abs Dev Training[i] = T_stats[4];
dt_results:Misclassification Rate Training[i] = T_stats[5];
dt_results:Name( "-LogLiklihood Training" )[i] = T_stats[6];
dt_results:Sum Freq Training[i] = T_stats[7];
dt_results:Generalized R² Validation[i] = V_stats[1];
dt_results:Entropy R² Validation[i] = V_stats[2];
dt_results:RMSE Validation[i] = V_stats[3];
dt_results:Mean Abs Dev Validation[i] = V_stats[4];
dt_results:Misclassification Rate Validation[i] = V_stats[5];
dt_results:Name( "-LogLiklihood Validation" )[i] = V_stats[6];
dt_results:Sum Freq Validation[i] = V_stats[7];
//dt_results:Generalized R² Test[i] = Test_stats[1];
//dt_results:Entropy R² Test[i] = Test_stats[2];
//dt_results:RMSE Test[i] = Test_stats[3];
//dt_results:Mean Abs Dev Test[i] = Test_stats[4];
//dt_results:Misclassification Rate Test[i] = Test_stats[5];
//dt_results:Name( "-LogLiklihood Test" )[i] = Test_stats[6];
//dt_results:Sum Freq Test[i] = Test_stats[7];
);
If( coltype == "Nominal",
dt_results:Generalized R² Training[i] = T_stats[1];
dt_results:Entropy R² Training[i] = T_stats[2];
dt_results:RMSE Training[i] = T_stats[3];
dt_results:Mean Abs Dev Training[i] = T_stats[4];
dt_results:Misclassification Rate Training[i] = T_stats[5];
dt_results:Name( "-LogLiklihood Training" )[i] = T_stats[6];
dt_results:Sum Freq Training[i] = T_stats[7];
dt_results:Generalized R² Validation[i] = V_stats[1];
dt_results:Entropy R² Validation[i] = V_stats[2];
dt_results:RMSE Validation[i] = V_stats[3];
dt_results:Mean Abs Dev Validation[i] = V_stats[4];
dt_results:Misclassification Rate Validation[i] = V_stats[5];
dt_results:Name( "-LogLiklihood Validation" )[i] = V_stats[6];
dt_results:Sum Freq Validation[i] = V_stats[7];
//dt_results:Generalized R² Test[i] = Test_stats[1];
//dt_results:Entropy R² Test[i] = Test_stats[2];
//dt_results:RMSE Test[i] = Test_stats[3];
//dt_results:Mean Abs Dev Test[i] = Test_stats[4];
//dt_results:Misclassification Rate Test[i] = Test_stats[5];
//dt_results:Name( "-LogLiklihood Test" )[i] = Test_stats[6];
//dt_results:Sum Freq Test[i] = Test_stats[7];
);
If( coltype == "Continuous",
dt_results:R² Training[i] = T_stats[1];
dt_results:RMSE Training[i] = T_stats[2];
dt_results:Mean Abs Dev Training[i] = T_stats[3];
dt_results:Name( "-LogLiklihood Training" )[i] = T_stats[4];
dt_results:SSE Training[i] = T_stats[5];
dt_results:Sum Freq Training[i] = T_stats[6];
dt_results:R² Validation[i] = V_stats[1];
dt_results:RMSE Validation[i] = V_stats[2];
dt_results:Mean Abs Dev Validation[i] = V_stats[3];
dt_results:Name( "-LogLiklihood Validation" )[i] = V_stats[4];
dt_results:SSE Validation[i] = V_stats[5];
dt_results:Sum Freq Validation[i] = V_stats[6];
//dt_results:R² Test[i] = Test_stats[1];
//dt_results:RMSE Test[i] = Test_stats[2];
//dt_results:Mean Abs Dev Test[i] = Test_stats[3];
//dt_results:Name( "-LogLiklihood Test" )[i] = Test_stats[4];
//dt_results:SSE Test[i] = Test_stats[5]
//dt_results:Sum Freq Test[i] = Test_stats[6];
);
);
Let me know if you find any other issues and I'll try to fix them.
Hope this helps!,
DS
Re: building multiple neural network modelsHi @abmayfield ,
Thanks for sharing the data table. I was able to find a few things.
- Found out where the error was that it keeps writing to the log window and fixed that. It's in the str call. Had to move a parenthesis to a different location.
- I found out that when doing K-fold or Holdback, you need to change the Validation() to Validation Method(). Then it works and proportions the data appropriately.
- I recoded how it tests for the modeling type of the column and made some associated changes.
- I also wrote it so the same code can be used for Nominal, Ordinal, and Continuous. This can be done simply by changing the variable y_resp -- the Y response column name. It does all the rest automatically.
- I also added a title to the results data table that changes with the "host" data table and the validation type so you can run all three methods and there will be no naming interference from JMP.
- The three NN fitting scripts in your data table can now by copy-pasted from one data table to the next without issues. I tested it with the first data table you sent, and it worked just fine.
It's getting a little more automated now, and is a bit improved. I would prefer to have it check the validation method and then generate the extra columns and assign values accordingly. Maybe in the next roll-out!
Here's what the code looks like in general. Notice that I was testing on the temperature column to check for continuous modeling types.
Names Default To Here( 1 );
dt = Current Data Table();
dt_parms = Data Table( "NN Tuning 8" );
dtp_name = dt_parms << get name;
dt_name = dt << get name;
//edit the line below so that it has the name of the column you want to model.
y_resp = "temperature (1, 2, 3)";
response_col = Column( dt, Eval( Eval Expr( Expr( y_resp ) ) ) );
dt_results = dt_parms << Subset( Output Table( "Output of " || dtp_name || " with (validation type)" || " for " || dt_name), All rows, Selected Columns Only( 0 ) );
//This tests the response column type to automatically set the values for pulling the data out of the report window.
//It also will create the appropriate columns depending on the modeling type of the response column.
coltype = response_col << Get Modeling Type;
If( coltype == "Ordinal",
o_box = 3;
ncol_box_v = 10;
ncol_box_test = 19;
dt_results << New Column( "Generalized R² Training" );
dt_results << New Column( "Generalized R² Validation" );
dt_results << New Column( "Generalized R² Test" );
dt_results << New Column( "Entropy R² Training" );
dt_results << New Column( "Entropy R² Validation" );
dt_results << New Column( "Entropy R² Test" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
dt_results << New Column( "RMSE Test" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
dt_results << New Column( "Mean Abs Dev Test" );
dt_results << New Column( "Misclassification Rate Training" );
dt_results << New Column( "Misclassification Rate Validation" );
dt_results << New Column( "Misclassification Rate Test" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
dt_results << New Column( "-LogLiklihood Test" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
dt_results << New Column( "Sum Freq Test" );
);
If( coltype == "Nominal",
o_box = 3;
ncol_box_v = 10;
ncol_box_test = 19;
dt_results << New Column( "Generalized R² Training" );
dt_results << New Column( "Generalized R² Validation" );
dt_results << New Column( "Generalized R² Test" );
dt_results << New Column( "Entropy R² Training" );
dt_results << New Column( "Entropy R² Validation" );
dt_results << New Column( "Entropy R² Test" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
dt_results << New Column( "RMSE Test" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
dt_results << New Column( "Mean Abs Dev Test" );
dt_results << New Column( "Misclassification Rate Training" );
dt_results << New Column( "Misclassification Rate Validation" );
dt_results << New Column( "Misclassification Rate Test" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
dt_results << New Column( "-LogLiklihood Test" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
dt_results << New Column( "Sum Freq Test" );
);
If( coltype == "Continuous",
o_box = 3;
ncol_box_v = 2;
ncol_box_test = 3;
dt_results << New Column( "R² Training" );
dt_results << New Column( "R² Validation" );
dt_results << New Column( "R² Test" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
dt_results << New Column( "RMSE Test" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
dt_results << New Column( "Mean Abs Dev Test" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
dt_results << New Column( "-LogLiklihood Test" );
dt_results << New Column( "SSE Training" );
dt_results << New Column( "SSE Validation" );
dt_results << New Column( "SSE Test" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
dt_results << New Column( "Sum Freq Test" );
);
imax = N Row( dt_parms );
For( i = 1, i <= imax, i++,
Nlayer = dt_parms:N_Layers[i];
If( dt_results:N_Layers[i] == 1,
dt_results:N_TanH_2[i] = 0;
dt_results:N_Linear_2[i] = 0;
dt_results:N_Gauss_2[i] = 0;
);
If( dt_results:N_Layers[i] == 2,
dt_results:N_Boosts[i] = 0;
dt_results:Learn_Rate[i] = 0.1;
);
NTH1 = dt_results:N_TanH_1[i];
NTH2 = dt_results:N_TanH_2[i];
NL1 = dt_results:N_Linear_1[i];
NL2 = dt_results:N_Linear_2[i];
NG1 = dt_results:N_Gauss_1[i];
NG2 = dt_results:N_Gauss_2[i];
Nboosts = dt_results:N_Boosts[i];
LR = dt_results:Learn_Rate[i];
TCov = dt_results:T_Cov[i];
RFit = dt_results:Robust_Fit[i];
PMethod = dt_results:Penalty_Method[i];
Ntours = dt_results:N_Tours[i];
str = Eval Insert(
"report = (dt << Neural(
Y( response_col ),
X(
:OFA...22_c0_g2_i1.p1,
:OFA...47_c0_g1_i1.p1,
:OFA...73_c1_g1_i3.p1,
:OFA...72_c1_g1_i1.p1,
:OFA...5_c1_g2_i15.p1,
:OFA...78_c2_g1_i3.p1,
:OFA...51_c4_g1_i1.p1,
:OFA...38_c0_g1_i8.p1,
:OFA...19_c0_g1_i1.p1,
:OFA...22_c2_g1_i4.p1,
:OFA...27_c1_g3_i5.p1,
:OFA...82_c3_g2_i3.p1,
:OFA...11_c1_g1_i7.p1,
:OFA...50_c2_g4_i1.p3,
:OFA...18_c1_g1_i7.p1,
:OFA...66_c2_g1_i3.p1,
:OFA...85_c0_g1_i2.p1,
:OFA...99_c5_g1_i6.p1,
:OFA...03_c2_g1_i1.p1,
:SYM...24_c0_g1_i1.p1,
:SYM...75_c0_g2_i1.p1,
:SYM...42_c0_g1_i1.p1,
:SYM...04_c0_g1_i1.p1,
:SYM...97_c0_g1_i1.p1,
:SYM...13_c0_g1_i1.p1,
:SYM...66_c0_g1_i1.p1,
:SYM...72_c0_g6_i1.p1,
:SYM...13_c0_g1_i1.p1 2,
:SYM...33_c0_g1_i1.p1 3,
:SYM...43_c0_g1_i1.p1,
:SYM...51_c0_g1_i1.p1,
:SYM...65_c0_g1_i1.p1 2
),
Validation ( :Validation with test ),
Informative Missing(0),
Transform Covariates(^TCov^),
Fit(
NTanH(^NTH1^),
NLinear(^NL1^),
NGaussian(^NG1^),
NTanH2(^NTH2^),
NLinear2(^NL2^),
NGaussian2(^NG2^),
Transform Covariates(^TCov^),
Penalty Method(\!"PMethod\!"),
Number of Tours(^Ntours^),
N Boost(^Nboosts^),
Learning Rate(^LR^)
),
Go,
invisible
)) << Report;"
);
Eval( Parse( str ) );
w = Window( dt << GetName || " - " || "Neural of " || y_resp );
T_stats = w[Outline Box( o_box ), Number Col Box( 1 )] << Get;
V_stats = w[Outline Box( o_box ), Number Col Box( ncol_box_v )] << Get;
Test_stats = w[Outline Box( o_box ), Number Col Box( ncol_box_test )] << Get;
report << Close Window;
If( coltype == "Ordinal",
dt_results:Generalized R² Training[i] = T_stats[1];
dt_results:Entropy R² Training[i] = T_stats[2];
dt_results:RMSE Training[i] = T_stats[3];
dt_results:Mean Abs Dev Training[i] = T_stats[4];
dt_results:Misclassification Rate Training[i] = T_stats[5];
dt_results:Name( "-LogLiklihood Training" )[i] = T_stats[6];
dt_results:Sum Freq Training[i] = T_stats[7];
dt_results:Generalized R² Validation[i] = V_stats[1];
dt_results:Entropy R² Validation[i] = V_stats[2];
dt_results:RMSE Validation[i] = V_stats[3];
dt_results:Mean Abs Dev Validation[i] = V_stats[4];
dt_results:Misclassification Rate Validation[i] = V_stats[5];
dt_results:Name( "-LogLiklihood Validation" )[i] = V_stats[6];
dt_results:Sum Freq Validation[i] = V_stats[7];
dt_results:Generalized R² Test[i] = Test_stats[1];
dt_results:Entropy R² Test[i] = Test_stats[2];
dt_results:RMSE Test[i] = Test_stats[3];
dt_results:Mean Abs Dev Test[i] = Test_stats[4];
dt_results:Misclassification Rate Test[i] = Test_stats[5];
dt_results:Name( "-LogLiklihood Test" )[i] = Test_stats[6];
dt_results:Sum Freq Test[i] = Test_stats[7];
);
If( coltype == "Nominal",
dt_results:Generalized R² Training[i] = T_stats[1];
dt_results:Entropy R² Training[i] = T_stats[2];
dt_results:RMSE Training[i] = T_stats[3];
dt_results:Mean Abs Dev Training[i] = T_stats[4];
dt_results:Misclassification Rate Training[i] = T_stats[5];
dt_results:Name( "-LogLiklihood Training" )[i] = T_stats[6];
dt_results:Sum Freq Training[i] = T_stats[7];
dt_results:Generalized R² Validation[i] = V_stats[1];
dt_results:Entropy R² Validation[i] = V_stats[2];
dt_results:RMSE Validation[i] = V_stats[3];
dt_results:Mean Abs Dev Validation[i] = V_stats[4];
dt_results:Misclassification Rate Validation[i] = V_stats[5];
dt_results:Name( "-LogLiklihood Validation" )[i] = V_stats[6];
dt_results:Sum Freq Validation[i] = V_stats[7];
dt_results:Generalized R² Test[i] = Test_stats[1];
dt_results:Entropy R² Test[i] = Test_stats[2];
dt_results:RMSE Test[i] = Test_stats[3];
dt_results:Mean Abs Dev Test[i] = Test_stats[4];
dt_results:Misclassification Rate Test[i] = Test_stats[5];
dt_results:Name( "-LogLiklihood Test" )[i] = Test_stats[6];
dt_results:Sum Freq Test[i] = Test_stats[7];
);
If( coltype == "Continuous",
dt_results:R² Training[i] = T_stats[1];
dt_results:RMSE Training[i] = T_stats[2];
dt_results:Mean Abs Dev Training[i] = T_stats[3];
dt_results:Name( "-LogLiklihood Training" )[i] = T_stats[4];
dt_results:SSE Training[i] = T_stats[5];
dt_results:Sum Freq Training[i] = T_stats[6];
dt_results:R² Validation[i] = V_stats[1];
dt_results:RMSE Validation[i] = V_stats[2];
dt_results:Mean Abs Dev Validation[i] = V_stats[3];
dt_results:Name( "-LogLiklihood Validation" )[i] = V_stats[4];
dt_results:SSE Validation[i] = V_stats[5];
dt_results:Sum Freq Validation[i] = V_stats[6];
dt_results:R² Test[i] = Test_stats[1];
dt_results:RMSE Test[i] = Test_stats[2];
dt_results:Mean Abs Dev Test[i] = Test_stats[3];
dt_results:Name( "-LogLiklihood Test" )[i] = Test_stats[4];
dt_results:SSE Test[i] = Test_stats[5];
dt_results:Sum Freq Test[i] = Test_stats[6];
);
);Let me know if there are any further issues.
Thanks!,
DS
Re: building multiple neural network modelsIt's weird. If I just keep running the script, it will eventually work!
Another strange thing: no matter what I change the holdback percentage to, it always gives me 11 training vs. 9 validation (~55/45%). Is it not possible to do, say, 70/30%? I have a strong feeling this is because I simply entered in the command incorrectly, rather than it being impossible!
Re: building multiple neural network modelsHi @abmayfield ,
The outline box error is probably from when it tries to grab the statistic values from the NN window. Can you share the new data table and code with me? I can troubleshoot it and see if I can make it easier to port it from one data table to another.
I think it might have to do with how I've set up the part to check the data column type and to make the ncol_box variable a different value. If it's nominal this value is different than if it's ordinal, or even numerical.
If you can share the data table and code that you're getting this error with, I'm sure I can fix it. If you want, you could anonymize the data table: Tables > Anonymize.
DS
Re: building multiple neural network modelsOK, great. This answers a few additional questions I had. Now the only thing I'm struggling with is, when I copy, paste, and run the script in a new data table, I get an outline box error, which you allude to in the script comments. "expected character arg 1 in access or evaluation of outline box." What does that mean?? I've increased and decreased the numbers, but I get the same error message every time, and the script will not run. I don't even know what an outline box is. It's weird that the tree structure is identical. I just changed the file name. The file name is longer. Maybe that means I need a larger outline box?Re: building multiple neural network modelsHi @abmayfield ,
If you go back and review the script, the second line defines what the data table is to be used during the analysis. The JSL code dt = Current Data Table() defines a local variable "dt" that is then used in the rest of the script instead of having to call the data table by name. Similarly, dt_parms is the local variable that is assigned to the NN tuning table, "NN tuning". Those two tables are really the only things that you feed to the script. The rest is all internal.
If you copy/paste the JSL code into a new data table it should work automatically because you're running it locally in that data table.
You should not modify the call dt << Nerual(), except to define your Y()'s and X()'s, and maybe the Validation() method. The JSL code dt << Nerual() is a command that is telling JMP to perform NN modeling on the data table "dt". Hence, the definition of what dt is at the beginning of the code.
You will also need to modify the w = Window() call to reflect what the title of the NN report window will be, as well as the Regex() below it. Finally, depending on whether or not you actually will have the "test" data set as part of a validation column, then you'll either want to comment out (with the // command at the start of the line) or remove the comment out at the beginning of the code when it creates the Test result columns as well as the Test_stats definition and where the code writes the results to the new output data table.
Other than than that, you shouldn't have to edit much at all.
With regards to your last question, it kind of depends on how you're implementing the code. If you're running it from within the data table, i.e. that it's saved in the script portion of the data table and you're running it via the green hot button, then you would only need to change the Y() and X()'s that you're using as predictors and response to match the column names from the new data table. You will of course need to have the "NN tuning" table open. On the other hand, if you're running this as a separate script, then you might want to change the definition of dt from dt = Current Data Table() to dt = Data Table (" "), where you'd insert the name of the new data table between the quotes.
The "Neural" you're referring to is not a data table but the JMP command to run a Neural Net model on a specified data table. The "<<" command means to perform the action which is to the right of the << on the object to the left. Near the start of the code you can see that with the new column command. Where it says dt_results << New Column(), the code is telling JMP to create a new column for the object dt_results, which is the subset data generated from the dt_parms data table.
Hope this helps!,
DS
Re: building multiple neural network modelsI have a (likely simple) scripting question. Say I want to change the input dt to another one. Shouldn't I just be able to copy and paste the script, and substitute the dt from "Neural" to something else?
str = Eval Insert(
"report = (dt << Neural (
Y( :health designation ),
X(
If I delete the original "Neural" file and substitute it with another table with the exact same information and a different name, the script won't work! Is it necessary to have the exact same data in two data tables, one of which is named "Neural?" Maybe I missed that part!
Re: building multiple neural network modelsGlad to hear it's working out for you. Good luck with the modeling! You might want to check other model platforms as well and then compare models on a test data set to see which is the best predictor.
DSRe: building multiple neural network modelsWow, I can't believe that I've run 300 neural network models before lunch thanks to your scripting help. I hope this gets picked up and "kudo'd" as I imagine when more and more people start seeing the value of the NN platform, they will gravitate towards something like this, especially if it's a brand new dataset and you don't really know where to "dive in" with respect to the bewildering amount of NN model input parameters that can be slightly tweaked (sometimes resulting in huge differences even when just increasing, for instance, the learning rate from 0.05 to 0.1). I will play around with the different validation approaches next (validation column vs. holdback vs. KFold). Definitely need to be careful given my (currently) small sample size (hoping to use these preliminary data to get funds to add more "rows" i.e., corals).
Re: building multiple neural network modelsHi @abmayfield ,
Kfold is also another great way to validate the short data set. Yes, the row exclusion you would set as a row state, which you could do by selecting the "test" data set and set the row state to "excluded". If you use Kfold or Holdback, JMP uses those for the validation of the data, not as testing of the data. Hopefully you also have a separate data set you can use as the test data and not have it in the same data table as the training and validation data.
I would just use Kfold for validation and then comment out the parts that you don't need to run the script. Below is the edited version with a 5-Kfold validation setting for the NN platform. Should work, it did for me.
Names Default To Here( 1 );
dt = Current Data Table();
dt_parms = Data Table( "NN Tuning" );
dt_results = dt_parms << Subset( All rows, Selected Columns Only( 0 ) );
dt_results << New Column( "Generalized R² Training" );
dt_results << New Column( "Generalized R² Validation" );
//dt_results << New Column( "Generalized R² Test" );
dt_results << New Column( "Entropy R² Training" );
dt_results << New Column( "Entropy R² Validation" );
//dt_results << New Column( "Entropy R² Test" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
//dt_results << New Column( "RMSE Test" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
//dt_results << New Column( "Mean Abs Dev Test" );
dt_results << New Column( "Misclassification Rate Training" );
dt_results << New Column( "Misclassification Rate Validation" );
//dt_results << New Column( "Misclassification Rate Test" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
//dt_results << New Column( "-LogLiklihood Test" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
//dt_results << New Column( "Sum Freq Test" );
imax = N Row( dt_parms );
For( i = 1, i <= imax, i++,
Nlayer = dt_parms:N_Layers[i];
If( dt_results:N_Layers[i] == 1,
dt_results:N_TanH_2[i] = 0;
dt_results:N_Linear_2[i] = 0;
dt_results:N_Gauss_2[i] = 0;
);
If( dt_results:N_Layers[i] == 2,
dt_results:N_Boosts[i] = 0;
dt_results:Learn_Rate[i] = 0.1;
);
NTH1 = dt_results:N_TanH_1[i];
NTH2 = dt_results:N_TanH_2[i];
NL1 = dt_results:N_Linear_1[i];
NL2 = dt_results:N_Linear_2[i];
NG1 = dt_results:N_Gauss_1[i];
NG2 = dt_results:N_Gauss_2[i];
Nboosts = dt_results:N_Boosts[i];
LR = dt_results:Learn_Rate[i];
TCov = dt_results:T_Cov[i];
RFit = dt_results:Robust_Fit[i];
PMethod = dt_results:Penalty_Method[i];
Ntours = dt_results:N_Tours[i];
str = Eval Insert(
"report = (dt << Neural(
Y( :health designation ),
X(:OFA...22_c0_g2_i1.p1 ),
Validation ( \!"KFold\!", 5 ),
Informative Missing(0),
Transform Covariates(^TCov^),
Fit(
NTanH(^NTH1^),
NLinear(^NL1^),
NGaussian(^NG1^),
NTanH2(^NTH2^),
NLinear2(^NL2^),
NGaussian2(^NG2^),
Transform Covariates(^TCov^),
Penalty Method(\!"PMethod\!"),
Number of Tours(^Ntours^),
N Boost(^Nboosts^),
Learning Rate(^LR^)
)),
Go,
invisible
) << Report;"
);
Eval( Parse( str ) );
w = Window( dt << GetName || " - " || "Neural of health designation by OFA...22_c0_g2_i1.p1" );
If( !Is Missing( Regex( w << Get Window Title, "health designation by OFA...22_c0_g2_i1.p1" ) ) == 1,
ncol_box_v = 10;
//ncol_box_test =19;
);
T_stats = w[Outline Box( 3 ), Number Col Box( 1 )] << Get;//You might need to adjust the Outline Box () and Number Col Box() values depending on your tree structure
V_stats = w[Outline Box( 3 ), Number Col Box( ncol_box_v )] << Get;//You might need to adjust the Outline Box () and Number Col Box() values depending on your tree structure
//Test_stats = w[Outline Box( 3 ), Number Col Box( ncol_box_test )] << Get;
report << Close Window;
dt_results:Generalized R² Training[i] = T_stats[1];
dt_results:Entropy R² Training[i] = T_stats[2];
dt_results:RMSE Training[i] = T_stats[3];
dt_results:Mean Abs Dev Training[i] = T_stats[4];
dt_results:Misclassification Rate Training[i] = T_stats[5];
dt_results:Name( "-LogLiklihood Training" )[i] = T_stats[6];
dt_results:Sum Freq Training[i] = T_stats[7];
dt_results:Generalized R² Validation[i] = V_stats[1];
dt_results:Entropy R² Validation[i] = V_stats[2];
dt_results:RMSE Validation[i] = V_stats[3];
dt_results:Mean Abs Dev Validation[i] = V_stats[4];
dt_results:Misclassification Rate Validation[i] = V_stats[5];
dt_results:Name( "-LogLiklihood Validation" )[i] = V_stats[6];
dt_results:Sum Freq Validation[i] = V_stats[7];
//dt_results:Generalized R² Test[i] = Test_stats[1];
//dt_results:Entropy R² Test[i] = Test_stats[2];
//dt_results:RMSE Test[i] = Test_stats[3];
//dt_results:Mean Abs Dev Test[i] = Test_stats[4];
//dt_results:Misclassification Rate Test[i] = Test_stats[5];
//dt_results:Name( "-LogLiklihood Test" )[i] = Test_stats[6];
//dt_results:Sum Freq Test[i] = Test_stats[7];
);
Good luck!,
DS
Re: building multiple neural network modelsMore excellent suggestions, especially regarding the type of validation. I see that Neural supports holdback, Kfold, or row exclusion (whereby I'm guessing I'd just randomly choose a few rows to leave out). Were I to want to change the script to change the type of validation, would I just change the "Validation ( :Validation with test )," line? And would the held back samples be equivalent in the eyes of JMP as validation samples? If not, I'd need to remove the validation columns (or rename them), and the test columns would have to go, as well.
Re: building multiple neural network modelsHi @abmayfield ,
Glad that it fixed your problem.
With the tree-based modeling (XGBoost, boosted trees, and bootstrap forest), you can easily use the tuning table GUI within JMP to run many models. But, one downside is that it chews up a lot of RAM. The nice thing about running it this way is that you can run 10's of thousands of different parameter settings and it doesn't eat up much ram, just CPU consumption. I've run up to 40k different settings with bootstrap forest (which is slow) and it can take a few days on my work laptop (32 GB RAM and 2GHz CPU).
I am also an early adopter for JMP 16, and the model screening is nice, but as you point out, it's not tuning any of the models, which can be a bit misleading. It just uses the default settings, which aren't always the best settings to create a predictive model. It could at least help you to determine which platforms are worth the time effort to tune further and which aren't worth looking into.
You might want to also consider the SVM platform to model your nominal data as this is supposed to also be a good method for that kind of data.
As for the simulator, that is something different and is meant as a way to test the estimates with different initial conditions (randomization). This gives a distribution of values for the estimate and gives you a confidence interval for the original estimate and you can then decide on whether that model with the given estimates is good or not.
As for the DOE, you could do a factorial or even run as a space filling design with the factors set appropriately, and I think that platform also allows for conditional constraints. Also, you might consider doing a predictor screening or bootstrapping on the predictors to see if you actually need all 89 of them or if you can really only use a smaller subset.
As a last note, since you have a short and wide data table, you might want to consider instead of using a validation column, do a Leave one Out approach for the validation. If you do that, you'll have to edit the script for that.
Good luck!,
DS
Re: building multiple neural network modelsAha! Thanks so much for re-doing it for me. I clearly have no knowledge of scripting, so I just changed "Neural" to the name of my data table.....which obviously wouldn't work. Once I changed the name of my data table to "Neural," added the Test columns, and made the other changes you recommended, it worked.
Now what I might do is use DOE to make a conditionally factorial design (since, as you noted, you can't boost with multiple hidden layers) and run a few hundred models on my work computer (so as not to overly tax my 8-GB RAM laptop).
I feel like your script needs to become a GUI or add-in (any takers?). JMP Pro 16, which I am beta-testing, has a really great "model screening" tool, but I think it tests across modeling types (e.g., PLS vs. Gen-Reg), not WITHIN the potentially thousands of models that could be built within each modeling type (or else it may never stop running). I have heard of the "simulator," but I think that is doing something different (not generating tons of simulated models), though I could be wrong. Again, thanks so much for your help. This will surely save me DAYS of time!
Re: building multiple neural network modelsHi @abmayfield ,
Thanks for sharing the data table, it helped to debug the changes you made in the script.
A few thing I noticed:
- You needed to add additional columns for the "Test" data set statistics you want to record.
- You accidentally removed the definition for imax, which is the number of rows of the parameter table
- You also accidentally changed the call to the NN platform where in the script it says dt << Neural(), this is where it sends the command to perform the neural net modeling on the data table "dt". You had changed it to something else.
- The definition of the window "w" needs to have the full name of the NN window, and if you're only running a single X, then apparently JMP puts that in as part of the window name, so it needs to be in both the GetName command and the IF statement below it. But that's only true when modeling with a single X. If using more than one, then you don't need to have the "by......" in the window name or IF statement.
- The last part is to make sure that the stats for the Training, Validation, and Test data are correctly called in the Number Col Box() command.
The below code works on your data table (also attached with modified script). It works with the NN Tuning table I gave earlier.
If you are really going to use all 86 columns, you need to put them in manually in the X(col1, col2,...) part of the script. You should then also edit the GetName part because it won't include all 86 names, it'll just be the window name with the response column.
Names Default To Here( 1 );
dt = Current Data Table();
dt_parms = Data Table( "NN Tuning" );
dt_results = dt_parms << Subset( All rows, Selected Columns Only( 0 ) );
dt_results << New Column( "Generalized R² Training" );
dt_results << New Column( "Generalized R² Validation" );
dt_results << New Column( "Generalized R² Test" );
dt_results << New Column( "Entropy R² Training" );
dt_results << New Column( "Entropy R² Validation" );
dt_results << New Column( "Entropy R² Test" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
dt_results << New Column( "RMSE Test" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
dt_results << New Column( "Mean Abs Dev Test" );
dt_results << New Column( "Misclassification Rate Training" );
dt_results << New Column( "Misclassification Rate Validation" );
dt_results << New Column( "Misclassification Rate Test" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
dt_results << New Column( "-LogLiklihood Test" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
dt_results << New Column( "Sum Freq Test" );
imax = N Row( dt_parms );
For( i = 1, i <= imax, i++,
Nlayer = dt_parms:N_Layers[i];
If( dt_results:N_Layers[i] == 1,
dt_results:N_TanH_2[i] = 0;
dt_results:N_Linear_2[i] = 0;
dt_results:N_Gauss_2[i] = 0;
);
If( dt_results:N_Layers[i] == 2,
dt_results:N_Boosts[i] = 0;
dt_results:Learn_Rate[i] = 0.1;
);
NTH1 = dt_results:N_TanH_1[i];
NTH2 = dt_results:N_TanH_2[i];
NL1 = dt_results:N_Linear_1[i];
NL2 = dt_results:N_Linear_2[i];
NG1 = dt_results:N_Gauss_1[i];
NG2 = dt_results:N_Gauss_2[i];
Nboosts = dt_results:N_Boosts[i];
LR = dt_results:Learn_Rate[i];
TCov = dt_results:T_Cov[i];
RFit = dt_results:Robust_Fit[i];
PMethod = dt_results:Penalty_Method[i];
Ntours = dt_results:N_Tours[i];
str = Eval Insert(
"report = (dt << Neural(
Y( :health designation ),
X(
:OFA...22_c0_g2_i1.p1
),
Validation ( :Validation with test ),
Informative Missing(0),
Transform Covariates(^TCov^),
Fit(
NTanH(^NTH1^),
NLinear(^NL1^),
NGaussian(^NG1^),
NTanH2(^NTH2^),
NLinear2(^NL2^),
NGaussian2(^NG2^),
Transform Covariates(^TCov^),
Penalty Method(\!"PMethod\!"),
Number of Tours(^Ntours^),
N Boost(^Nboosts^),
Learning Rate(^LR^)
)),
Go,
invisible
) << Report;"
);
Eval( Parse( str ) );
w = Window( dt << GetName || " - " || "Neural of health designation by OFA...22_c0_g2_i1.p1" );
If( !Is Missing( Regex( w << Get Window Title, "health designation by OFA...22_c0_g2_i1.p1" ) ) == 1,
ncol_box_v = 10;
ncol_box_test =19;
);
T_stats = w[Outline Box( 3 ), Number Col Box( 1 )] << Get;//You might need to adjust the Outline Box () and Number Col Box() values depending on your tree structure
V_stats = w[Outline Box( 3 ), Number Col Box( ncol_box_v )] << Get;//You might need to adjust the Outline Box () and Number Col Box() values depending on your tree structure
Test_stats = w[Outline Box( 3 ), Number Col Box( ncol_box_test )] << Get;
report << Close Window;
dt_results:Generalized R² Training[i] = T_stats[1];
dt_results:Entropy R² Training[i] = T_stats[2];
dt_results:RMSE Training[i] = T_stats[3];
dt_results:Mean Abs Dev Training[i] = T_stats[4];
dt_results:Misclassification Rate Training[i] = T_stats[5];
dt_results:Name( "-LogLiklihood Training" )[i] = T_stats[6];
dt_results:Sum Freq Training[i] = T_stats[7];
dt_results:Generalized R² Validation[i] = V_stats[1];
dt_results:Entropy R² Validation[i] = V_stats[2];
dt_results:RMSE Validation[i] = V_stats[3];
dt_results:Mean Abs Dev Validation[i] = V_stats[4];
dt_results:Misclassification Rate Validation[i] = V_stats[5];
dt_results:Name( "-LogLiklihood Validation" )[i] = V_stats[6];
dt_results:Sum Freq Validation[i] = V_stats[7];
dt_results:Generalized R² Test[i] = Test_stats[1];
dt_results:Entropy R² Test[i] = Test_stats[2];
dt_results:RMSE Test[i] = Test_stats[3];
dt_results:Mean Abs Dev Test[i] = Test_stats[4];
dt_results:Misclassification Rate Test[i] = Test_stats[5];
dt_results:Name( "-LogLiklihood Test" )[i] = Test_stats[6];
dt_results:Sum Freq Test[i] = Test_stats[7];
);
Hope this helps!,
DS
Re: building multiple neural network modelsWell, I failed, but I think it's probably because of some tiny scripting mistake. I used your NN tuning table with my data (attached) and made all the necessary updates (I used three training+validation+test samples). If you run the last script ("neural network comparison"), it will build the new table, but no analyses are run. I just included a few X terms, but it will actually be all 86 of the columns in the "standardized data" group. When I use the debugger, it says something is going wrong with the
"For( i = 1, i <= imax, i++," area. I am hoping/guessing a more competent script writer can probably find the issue in seconds!
Re: building multiple neural network modelsYes! This is exactly what I want: a table with all the NN input parameters (e.g., number of TanH nodes, number of layers, etc.) that syncs with the NN platform. I have not tried it yet, and I do imagine it will take a while, but I am using a small 20 sample x 90 predictor "practice" dataset, so it might be doable. In reality, I've really only found that the type of activation and the number of nodes (as well as number of tours) actually lower my RMSE, with boosting not helping very much. But even then, this will be much faster than doing all ~500 models I planned to run (not nearly as many as I had intended to run yesterday thankfully since I want to stay under 4 nodes/activation type). I will try it out tomorrow and let everyone know how it goes.
Re: building multiple neural network modelsHi @abmayfield ,
This was something that I've been wanting to script up for a while and was really fun, and your post gave the fire to get it going, thanks! I've been wanting to have a way to tune the NN platform for a while as it is very sensitive to the initial conditions, and it's rather slow to do it manually.
So, I adapted some scripts that I have made for other model platforms to automate their tuning processes, for example, the boosted tree, bootstrap forest, and XGBoost platforms. The scripts are meant to be saved into your data table and run with the green hot button.
Hopefully you know some about scripting and will know where to adapt these scripts to your specific situations/data structure/column names, etc. I have entered in some comments to try and help explain what I'm doing in each larger group. I'm attaching two scripts -- one for continuous Y's and one for Nominal/Ordinal Y's. I did have to include a test to see if the output Y is nominal or ordinal because the Number Col Box() value changes depending on this. I used the Big Class Families.jmp data table in the sample data as a test data set. I'm also including two output data tables I generate using either the numerical NN tuning or nominal/ordinal NN tuning. I did nominal on :sex using :age, :weight:, :height, and then I changed :age from Ordinal to Continuous and modeled age with :weight and :sex for the continuous NN tuning. The fits and stats are horrible, but it's intended only for the purpose of seeing if the JSL works or not. I also added a :Validation column to the data table by stratifying on :age, just to check that it all worked correctly.
I'm sure someone else could script it up more elegantly to test the Y() input column to be modeling type Continuous, Nominal, or Ordinal and then have the appropriate Number Col Box() value inserted to extract the data appropriately using only a single JSL code.
You will need to edit the Y() and X() inputs to the dt << Nerual() section as well as the part where you get the name of the report window, as it's named after the variable you're modeling.
The NN tuning tables could be treating like a space filling DOE in principle where the lower and upper settings are whatever you want to set it to, except for some of them, like N_Layers can only be 1 or 2 because JMP only allows 1 or 2 layers. Penalty_Method is only the options that are valid in the NN platform, and the Transform Covariate and Robust Fit only take on values of 0 or 1 (off or on). I'm including the NN tuning table as well.
If you have any questions, let me know.
Here's my code for the Nominal/Ordinal modeling type:
Names Default To Here( 1 );
dt = Data Table( "Big Class Families" );//enter the name of the current data table
dt_parms = Data Table( "NN Tuning" );//enter name of parameter data table for tuning the NN
dt_results = dt_parms << Subset( All rows, Selected Columns Only( 0 ) );//this copies the tuning table with all the different rows in it
//these commands create columns to record the fit results
dt_results << New Column( "Generalized R² Training" );
dt_results << New Column( "Generalized R² Validation" );
dt_results << New Column( "Entropy R² Training" );
dt_results << New Column( "Entropy R² Validation" );
dt_results << New Column( "RMSE Training" );
dt_results << New Column( "RMSE Validation" );
dt_results << New Column( "Mean Abs Dev Training" );
dt_results << New Column( "Mean Abs Dev Validation" );
dt_results << New Column( "Misclassification Rate Training" );
dt_results << New Column( "Misclassification Rate Validation" );
dt_results << New Column( "-LogLiklihood Training" );
dt_results << New Column( "-LogLiklihood Validation" );
dt_results << New Column( "Sum Freq Training" );
dt_results << New Column( "Sum Freq Validation" );
//This part just creates a separate modelling progress, delete if you don't want it.
i = 1;
imax = N Row( dt_parms );
dlgStatus = New Window( "Overall Modeling Progress",
V List Box(
Text Box( " Overall Modeling Progress ", <<SET font="" size=""></SET>
One thing that is strange that I don't understand is that I get this warning about the log window I create when I run it for the NN platform, but I've never gotten it from my other platforms. I can't follow where that warning comes from, but it still runs correctly. Maybe someone can fix that part of my code?
Unexpected ",". Perhaps there is a missing ")".
Trying to parse operand for "<<" operator.
Line 23 Column 10: ))►,
As a last note: be careful about the settings as it can take a VERY long time to run the modeling if you have huge numbers of boosts or tours or transfer functions. Just be cautious not to go overboard.
Hope this helps!,
DS
Re: building multiple neural network modelsBTW, the "option" trick to get all relevant features to show the same information as the feature you are actively selecting is actually the "command" trick on a Mac (hence why that didn't previously work).
Re: building multiple neural network modelsMark, Thanks for your suggestions. The second one I've tried and failed. I know the "make into combined table" option from other platforms, but maybe it was left out of the neural platform for some reason. Strangely, the "option" trick, whereby you hold down the option key, make a selection, and it applies to all parts of the report, does not work either in the neural platform, or else that could speed things up.
That would be cool if I could have my neural network parameters (# nodes, #tours, etc.) as X's in a DOE, with the goal of minimizing RSME, but it seems to not be possible because of the way you must enter the parameters in the particular boxes in the neural model launch. You'd have to do some really advanced scripting, and even then I'm not sure if DOE and Neural could communicate properly. But if anyone out there knows of a way in which I could get the neural platform to basically run the models reflected in this screenshot (being a sub-selection of the grand total), I would greatly appreciate it. In other words, the values in the first screenshot table would need to go to the appropriate input boxes in the second screenshot (i.e., the neural model launch).
Re: building multiple neural network modelsYou could design an experiment with fewer levels to fit a second-order model of RMSE and then use the model to optimize the settings. Also, you can make each model in turn and then extract all the results at once by right-clicking on one of the results table and selecting Make Into Combined Table.
