Application for Functional Data on Infrared Spectroscopy and ChromatographyI have an inquiry about using Functional Data Explorer on infrared spectroscopy and chromatography. Example waveforms are pasted below and I'm trying to get the customer to send me a data table. The application is to use these as responses from DOE. For infrared spectrography, the presence of peaks at specific wavenumbers is interesting. For chromatography, the area under the curve at points in time (the x-axis is time) is interesting. There was also discussion about aligning the x-axis and subtracting a basis response.
https://www2.chemistry.msu.edu/faculty/reusch/virttxtjml/spectrpy/infrared/infrared.htm
https://www.researchgate.net/figure/High-performance-liquid-chromatography-HPLC-chromatograms-of-the-flavonoids-in-CN30_fig1_281614653
Re: Application for Functional Data on Infrared Spectroscopy and ChromatographyHey Martin,
Your suggestion P-Splines is something I have been playing with using FDE. I have not tried FT as of yet, but I am also skeptical that it will do any better than a spline fit....at least for spectral data. One thing I have found that seems to work well is to use the Rows as Functions instead of stacking the data. I have had good results with NIR and Mass Spec data the Rows as Functions and P-Splines. I did run into some difficulty with NMR data, but I need to revisit that data to try some new things.
Ultimately though, I use PLS or Gen Reg to fit the predictive models and I am using the scanning Elastic Net in Gen Reg to find some really good predictive models that are on par with or better than PLS models for both continuous and binomial response data.
Re: Application for Functional Data on Infrared Spectroscopy and ChromatographyI believe that JMP Development have recently had some good discussions with customers with data like this. So they have a good understanding of why spectral and chromatography data (and similar such as XRD) are difficult for FDE as it is now. Hopefully they will be able to find solutions that could open this up to new applications and new users of JMP Pro!
Re: Application for Functional Data on Infrared Spectroscopy and ChromatographyThough already sa long time ago:
Just to add on Phil/Mike/Bill's comments: This kind of data will be difficult to fit with FDE standard knots. you either want to try fourier (though this could overswing on one side) or P-Splines where you adjust the knots closer around the peak variations. This will help to eliminate the effect of smoothing. Often this is not important but if you are interested in the different positions of the peaks (hight, auc, ...) this might become important.
From my past work in with seismographic and signal processing customers filtering the noise was important, there FDE still lacks of good filtering techniques. This would help in the data processing a lot I believe.
However, I would give it a try and see how good you can be (currently). Though I like nonlinear and curfe fit platform I'm not that convinced they will do a decent good job either
Re: Application for Functional Data on Infrared Spectroscopy and ChromatographyFrom my experience with customers, IR spectra and Chromatography are useful applications of FDE. GSK have presented a couple of times on how they used FDE to optimise a chromatography method (although they could not share the presentation). I have another customer looking at FTIR spectra.
In both cases they were not looking at the whole spectrum/chromatogram. They focus on particular regions/peaks of interest.
The examples above look a bit "busy" to me and I think FDE could struggle. But it really depends on the situation - what the nature of the function-to-function variation is and what they want to find out from the data.
Subtracting the baseline is common data prep in these situations. There is no easy way to do this in FDE just now but hopefully will be added.
Have a look at Root Integrated Function Square as one of the FDE summary statistics that could be helpful for area under the curve.Re: Application for Functional Data on Infrared Spectroscopy and ChromatographyHey!
Maybe a different perspective from @Bill_Worley. The NIR data from the PLS book is a great example of an IR spectra that will work really well with FDE. When the spectra peaks get more sharp the ability of doing a really good trace goes down. The function tries to round over the top/bottom spikey bit. I've had this problem with some XRD spectra, Mass Spec and NMR. That said, you can still get a lot of good information out of the trace (identifying active regions in the spectra, etc... see Pete's Discovery Talk...) I also find that it's a lot easier for people new to to the topic to wrap their heads around that PLS. I've been able to get traction with FDE where PLS just goes over their heads.
For Chromatography, particularly AUC calc's, you're going to want to go with Nonlinear or Fit Curve (IMO). Though, you might be able to use FDE to address the noise Bill mentions... kind of as a data clean-up step.
Best,
MRe: Application for Functional Data on Infrared Spectroscopy and ChromatographyHyde,
The IR spectra you show is for a pure compound. If they are looking at pure compounds they might be able to develop a calibration curve. PLS has been used very effectively with spectral data in the past. If you have the PLS book there is a great description on determining the octane in gas. That all said, Pete Hersh showed and example of working with NIR data in FDE at Discovery and you could contact him on that one.
For chromatography the time points are the elution times for specific compounds. Unless you wanted to compare curves from different lots I am not sure FDE is the right answer here, but I have been wrong before. There is typically a lot of baseline noise with chromatography as well which you mention. Happy to work through this with you if you get some data from your customer.
Best,
Bill
