FDE question about predicting a Y variable for new spectrum dataOk so here's my question.
Let's say I ran an FDE analysis on some spectrum data I had. I created a model, saved the function summaries in a new data table, and created a model that links the Functional PCAs to the Y variable of interest.
Now I have new spectrum data, and I would like to predict the Y variable for a new particular curve.
What should I do? Usually, with regular data, I can add a row with the X variable and the predicted Y column will be automatically populated. However, with functional data, I don't see where I can save the functional PCA values from the spectrum data so that I can use that in the prediction.
Am I missing something here? what would be your approach?
Re: FDE question about predicting a Y variable for new spectrum data@yasmine_hajar Interesting example and question. I guess one answer is to add the new rows to the original data and rerun FDE from a saved script or workflow, retraining the predictive model if the Functional PCAs are affected.
I was curious to see how well Torch performs using the original spectral features to predict FFA Value. The attached table splits Manda_DataFile into 23 rows and creates stratified 5-fold variables. Torch results with its default MLP over 100 epochs appear very good with out-of-fold correlation 0.99 and RMSE 0.39.
However, if I am reading the file names correctly, it looks like there are really only 7 effective experimental units with 3-4 reps on each. This is also apparent from the repeated values of FFA Value. I created a new column Unit and tried it as the Validation variable (in Torch, click red-triangle > Change Variables), creating a 7-fold scheme that holds out each unit in turn. The out-of-fold correlation drops to 0.9 and RMSE increases to 1.2. This latter scheme may provide a more realistic assessment of generalizability to new data.
Re: FDE question about predicting a Y variable for new spectrum dataThank you @russ_wolfinger ! I was hoping to find the embedded formulas for the FPCAs columns but rerunning the script will work for now.
I will come back to you with my questions about the Torch approach!
Thanks again!
Re: FDE question about predicting a Y variable for new spectrum dataWith the inclusion of Supplementary variables, FDOE can be run and then prediction formulas saved from the FDOE portion of the report. Then new rows added will automatically compute the prediction.
I'm not super familiar with the FDE platform, but it looks like if you don't have any supplementary variables, I don't see a way to save a prediction formula based on the FPCs out. That sounds like a feature request.
Re: FDE question about predicting a Y variable for new spectrum data@yasmine_hajar the best way to do that now is to load the new spectra with a validation column, that way you can ensure they are not used in the original sample. We have an open item to save a scoring script for this purpose that I will up the priority on.
Hope this helps!
Re: FDE question about predicting a Y variable for new spectrum data@eclaassen I would use FDOE if the supplementary parameter I had was a factor not a response .
But in my case, the spectrum is the factor and the supplementary information (the FFA column) is the prediction. Hence I created a model with PCA1,2,..5 in the model summaries table
