Are there limits to how much data FDE can handle?Hi All,
I’ve had a customer ask me about the extent that FDE can be scaled up.
Essentially, they have a set of curve data from some genomics work where there is curve data (made up of about 8 points) for approx. 20,000 genes. What they’d like to do is use FDE to model and screen these but wanted to know if there is a limit (other than potential compute time) to the extent of data that FDE can handle?
My assumption from this discussion: Solved: FDE: Do we have any numbers on performance? - JMP User Community is that this should be achievable, albeit with it being a lengthy process, but thought I’d reach out to you all to help confirm or deny this.
Thanks,
Stuart
Re: Are there limits to how much data FDE can handle?Hey, Stuart!
TL;DR:
Show them with Anodic Bonding and the standard "memory limit" statement about JMP probably holds in this case.
The soap box version:
You might have a look at the Anodic bonding data set in the sample data as a way to demonstrate that we can handle problems at this scale. It's got 122000 records, 61 data points per function with 2000 functions. It's a semiconductor data set, but it's in the same ball park for sample size. And, it runs pretty quick.
Also, your case has some advantages over the Anodic Bonding situation: You can't really get a complex shape with 8 data points per trace so you're not going to have a lot of FPC's. The number of knots is also going to be small (which is similar to the Anodic Bonding data). That also should make it pretty easy to tackle even larger data sets.
In the example that Martin and I were discussing (you linked to it) the traces were probably oversampled during acquisition. (It happens a lot.) Another strategy we could have used in that case would have been to downsample the data (i.e. using moving averages or something). But again - not a problem with this type of data.
Best,
M
