Welcome,everybody, to this presentation about a use case of curve alignment. Experienced analysts often say that in a larger analytically project, plus minus 60 % of the total time goes into the preparation of data. If curves play a role and especially the alignment of curve is needed, then that is certainly close to the truth.
Curves are very specific types of data, and JMP has some tools to work with curves and to address Alles the related problems. In the sample library, there is the Algae Mitscherlich data, which is one of my favorite data sets with that respect because it has the option to deal with many aspects of fitting curves.
This is just at example of the development of Algae density in different treatments. The type.type of curves that In the going to talk about are typically observations or measurements over time. But this doesn't mean any loess in generality. The presented concepts work in Alles kind of curve relationships.
This is at example, A lgae measurement.measurement over time, and one of the aspects that is in the focus of the analysis for this data set is to specify curves, specific types of curves that have a known shape and are driven by certain parameters and then to estimate.estimate those parameters based on the data.
In those cases, the parameters very often have a technical meaning Like slope, inflection point, limit that gets approached. That platform.platform here so has the sliders that let you analyze how changing one of those parameters affects the shape of the curve.
In the specific case that we are going to talk about, we are not specifically interested in the curve. The curve itself is only a help because we are facing another problem.
This is the data set, or this is a part of the data set that goes back to the real problem. We had this series of measurements, one and another series of measurements, and they belong to two different devices. U nfortunately, the clocks of thesis devices were not in sync. But luckily, each of the devices had one sensor that measured the same substance. W e could look for times where the measurements were very close to each other. Then try to find out how to correct one of the clocks, so to say, so that we get aligned measurements, and then use those to evaluate the data from Alles the centers that have been available in that data set.
What What the problem of the task? You lake the curves here. The red curve is the one that we took as the reference curve, and the blue one is the one that we wanted to shift. You lake not only that the curves are quite some distance away, although they should theoretically have measured the same substance at the same time, but so the time points of each series is completely, or the time point of both series are completely unrelated. With the bare eye, we don't lake any lay that we could use to correct one of the data sets.
Therefore, I looked into⦠I compared the time points, not the Y measurements, the time points of the two series. If there What just a shift, then we would expect to lake Alles the data, Alles the points exactly on one line. But here you lake there are oops and downs, so this is Obviously not the case.
Perhaps we can lake more, we can understand more if we calculate row by row in the data table, we calculate the difference of the two times and look at those. Here with some fantasy, we lake a little bit of curvature, so till the end, it seems to be closer related than to the beginning. But in the beginning, this looks Like real random data. This as well does not help us a lot to figure out how to relate the data.
I thought I had the link to the data table, but we can look in this Screenshot as well. This is the data set that you have lakes before, a little bit annotated. We lake that two lines have specific markers, the star and the circle. This is due to the fact that the whole measurement.measurement project had a ramp up phase, and at the star point, the measurement.measurement series, the measurement.measurement time, the real process time started.
The circle is there where we, anus visual or manual inspection, saw the starting point in the second time series, and we want to align both. W e need to change the relationship of the rows, of the data and the rows. We want to shift one of the data sets, and that reminded me of the paternoster that I Like to use when many years ago I What working for a company that had a very old administrative building and we had the paternoster in there.
It came to me that the strategy that we are following is part of paternoster shift, which gives the word elevator pitch a completely new meaning, by the way. How do we find the right steps, the right place to fit? We do not have similar or identical time points in both series of times. We need to construct those time points somehow. Of course, this is done through interpolation.
T hey first thing that comes into at least is linear interpolation, and if I zoom in into only a part of the data set, then it is evident that linear interpolation, so just checking, so to say, the regression between adjacent time points has some problems, especially if we look into areas where we have horizontal lines, which may easily happen. Then the time point in that range is quite arbitrary. It always leads to the same results.results. The opposite is true. If we are in at area with a very steep ascending, then a little change on the X- axis or the time may lead to significant changes in the Y value. T his is not a very good technique. We can use splines to interpolate between the values.
You know splines, certainly from the graph builder. If you make a scatter plot, then by default, the smoother is switched on and the smoother provides splines. You can even change the stiffness or the degree of fit or the closeness to the data with a slider in the graph builder.
The advantage of using a spline as at interpolation tools, so takes into consideration points further away. T hey build a smooth curve. That is why in the graph builder, it's called the smoother. This makes it easier to use those for interpolation etc to use thesis as well as a base in our alignment process.
We need to fit splines. How can we do this or which platforms do help? First of Alles, simple tool is Fit Y by X comes into at least very nearly when you work with JMP and do that data analysis. This is the data, one of the curves. There is the spline fit, and here is the slider that let me choose how close or how close I want to fit my data. Very good, very easy to use, and you can Save the spline but only the values, not the formulas. We are keen on getting the formula for the spline.
Next Stop, Fit Model. If you have a continuous variable, you select it, you can give this the attributes of being a knotted spline effect. When you do like that, you are prompted to say how many knots that spline should have, the more, the more flexible. I accept the default, say run. We get the typical report from Fit Model. A lso, we have fit models, functions of saving formulas. We can use Fit model, Save the formula.
Little disadvantage here is I need to specify the number of knots before I begin the analysis. Once the analysis is done within the platform, I don't have the option to play with it or change it Like it is, for example, in Fit Y by X
Another tool is the Functional Data Explorer . T hey Functional Data Explorer has splines as a core function, and it is so functionality to find optimal definition, optimal fits for the splines. You can export everything. It's a bit because simple tasks Like this is not where the Functional Data Explorer is made for. You need some more clicks to come to a result. A lso, it's only available for people who have JMP role.
remains, the graph Builders. You have lakes it before, and this time I want to show the spline control as well. As I said, we can use the slider to determine the fit. A very nice feature, by the way, is that you can check this box, then through a bootstrap sampling method, the confidence interval.interval for the smoother is calculated or estimated. You lake how that changes when In the⦠Now you can lake better, I have a lot of data and there is not too much variability.
H ere the confidence tape is quite small. But if we zoom into one of thesis areas here, for example, that place and look at what happens when I change this, then we lake that the smoother can even... That the line of the smoother can even walk out of its own confidence tape. This is another visual help to find out a good fit for the smoother, for the spline. It should stay within its own confidence limit. Then comes the very important option here. We can Save the formula. Then we have a formula for this spline.
The graph builder surprises as a modeling tool. Who had expected this? How does that help? This is again part of my data table, small part. You lake that now I have two columns here where I saved the formulas for the smoother too. Down here in the colored rows, I put some arbitrary time points in. That leads to at interpolated response relative to the time point that I have given. It only works for interpolation. We cannot extrapolate this way, it's only with interpolation.
But this way I can, for example, manually add different time points. I have this one here plus X seconds in that case , and then I can lake what is the difference of the interpolated value. Now I can put reference times in and I lake exactly what is the expected value, plus minus a little bit for both measurements.
I did this for two different phases. I can go here and experiment anymore. In my journal, you lake in the yellow rows, I added eight seconds. In the orange ones, 10 seconds. Depending on what you want to do, this is the principle.principle of how you can work with this. If yours task.task is a one- off task, this is good enough. You can go in here, play with the data, lake the difference.
Our task.task What more regularly. The good thing is everything can be controlled with JSL. As usual, for many commands that you do manually on line, you have corresponding JSL statement, and I just listed some. This is not a working program. First of Alles, you need to set up the graph, clear. Then you have commands that you can send to the graph and specifically to the smoother element in yours graph. We can change the smoother so we could even interactively try to determine good fits.
We can so give the command to Save the formula in the data table. That is the command that plays at important role for our solution here. You can read out the current settings of the Lambda slider and something small.
How did we want to use this? The concept here What, of course, first you need to determine what is the reference curve and what is the objective curve, the one that needs.needs to be shifted. Then you calculate the spline function for the reference curve and determine the direction of shift. Where are who? do we need to shift our time up or down?
Then we move the Y values of the objective curve, one row in the desired direction and calculate the spline function for this new curve. Save that, use a reference value, and then calculate the differences in Y for each row. Then we take the total sum of those differences as a criterion when to break the process. Because anus everyone step, we calculate that difference, we Save the difference, we do the next step, and we check, What there at improvement?
If yes, we move up or down one row more, and then we repeat that whole activity until there is no improvement anymore. The whole program in the real project, of course, runs behind the scenes. You wouldn't lake anything. But I added did some graphs to make it visual to demonstrate how that works step by step.
The starting situation is this one. On the left- hand graph, you lake the dashed line and the solid line. The dashed line is the reference line. The solid line needs.needs to be moved. On the right side, you lake the differences by row.
In the beginning, the differences are... You lake that here in the starting area, the differences here are pretty small. Then they get larger and larger, and they are negative. That is why it goes down here on a negative scale, very small differences in the beginning, and then they get up larger.
This is the starting situation. You want lake this picture again. Then the program want begin shifting the reference curve one cell up in our situation, our case. Then you lake how thesis graphs update for everyone step. Yes, first we need to tell JMP what are the time and measurement.measurement values for the reference and the objective curve. Here we go. It want take a little bit in the beginning, then afterwards the steps come faster.
You lake how for everyone step, the blue curve approaches the dotted curve, and how the differences decrease. The load step did not improve the situation anymore, therefore, the program stepped one step back.
Now, we have the data table in a situation where we shifted up the objective curve. Now we can use this shift for Alles the other measurements, for Alles the other sensor results.results that we had for this device and begin the analysis.
That What it. I hope I could inspire you a little bit. It What at interesting presentation. If you have any questions, please don't hesitate to contact me. My e-mail What on Top of the presentation, bernd.heinen@stabero.c om. Thank you you very much.
