Calculating linear regression slope in formulaHello,
I was wondering if there was a way to calculate the linear regression slope of column data within a data table formula.
A simple example of the table is below. The data is ordered and I would like to calculate the slope of a fit line of volume over time using the current row's data and previous points to perform the regression.
The idea is that I would like to see the how the calculated slope changes with increasing the number of data points. From doing this manually, the slope values end up approaching a given value and stay within +/-5% after 20 or so points.
| Instance | Time (s) | Volume (mL) | Slope (mL/s) |
| 1 | 0 | 0 | n/a |
| 2 | 5 | 0.5 | m1 |
| 3 | 10 | 0.8 | m2 |
| ... | ... | ... | ... |
| N | T | V | mN-1 |
If that is possible, my next challenge would be to set this up so that I can calculate the slope formula so that it uses N data points on either side of the current row for a symmetrical moving calculation instead.
Thank you in advance.
Re: Calculating linear regression slope in formulaThis is interesting.. if I wanted to apply a p-value to this... based on null hypothesis (slope = 0)... is there a way to integrate this into forumula?
In other words, I would like to see when the moving slope is changing significantly based on confidence interval around slope
Re: Calculating linear regression slope in formulaYou said, "Many times, we do not know when the correlation between the X's have changed, so this is one attempt to look into the impact of time for any analysis at hand." Have you tried the Functional Data Explorer in JMP Pro? It might be useful for detection a change in the shape through the functional principal components.
Re: Calculating linear regression slope in formulaHello JMP team ...
Nelson's approach is very likely what I was looking for. I am conversing w/ him on a similar need with mann kendall tau test.
My input is not in a single row.
My data is typically contains several parameters each measured at a different time step. Tidy data format; parameters are in columns, each row is a different time state - i.e. chronology is in rows.
I am looking into many statistical functions measuring trend in one of these or a fit versus some other X in time but as a function of rolling windows. So instead of regressing the whole history of X vs Y for instance, I like a logic that looks into regression of the two within constant past N time steps. Each row then re-evaluates and gets me a new result. These could a slope from linear regression, or a tau test from a non-parametric MK test or some other stat output.
Many times, we do not know when the correlation between the X's have changed, so this is one attempt to look into the impact of time for any analysis at hand. N mentioned above can be changed to various values to examine short term vs mid/long terms trends.
I am ok w/ scripts, they are not my preference unless the computations can great benefit a method offered only by scripts.
My preference is generally to put the calculations into formulaes and have them maintained within the data frames.
Until today, I did not know that script structures could be deployed within formulaes.
That probably was the main cause of my confusion.
Appreciate all your help.
Re: Calculating linear regression slope in formulaI know.
My point is that the best way to start learning about functions that you might use in a formula is with the Scripting Index.
Re: Calculating linear regression slope in formulaCan you give us some more background on what you're looking for?
Formulas, generally speaking, work across rows of a data table, evaluating one per row with column values from that row as inputs.
So, by asking for a regression function in the formula editor, one would expect that you had a matrix of input values in each row of your data table. Perhaps that's what you've got but that's an unusual structure.
So, with some more information we may be able to help you understand how you could use a formula for this, or perhaps, point you in a direction outside of the formula editor.
Re: Calculating linear regression slope in formulaA JMP formula can pretty much be anything you want it to be. My pointing you to the Linear Regression() function in the Scripting Index, happens to show it's example in the form of a script, but the function can also be used in a formula. Below is a silly example, of calculating the regression for first rows 1-3, then 1-4, 1-5, etc. for all of the rows in the Big Class data table. Here is the formula...….attached in the data table with the new formula column added
Re: Calculating linear regression slope in formulaI am not looking for a script.
I am looking for a formula application (i.e. using formula builder)Re: Calculating linear regression slope in formulaSelect Help > Scripting Index. Enter "regression" in the filter box. You should find the Linear Regression() function in the second list. Select it to see information about this function on the right.
Re: Calculating linear regression slope in formulahi ...
You mentioned "Linear Regression() function, which could be used in a formula" ...and getting a slope from a formula fitting to provided data set being the final objective, where would one go find Linear Regression() function or any help ?
Do you have any example ?
thanks
Re: Calculating linear regression slope in formulaJ() is a function that creates a matrix of identical values. The default is 1.
|| is an operator that indicate concatenation. So in Mark's example he is concatenating 2 matrices.
These functions and operators are fully documented in the Scripting Guide
Help==>JMP Document Library ….. Scripting Guide
Re: Calculating linear regression slope in formula@Mark_Bailey Hi mark, I am new to JMP and am trying to do something similar to bio_grad where I perform linear regression on 3 points, then 4 points, then so on in a column in my data table. I am trying to understand what your answer is actually doing and had a couple of questions. What does || mean? also what is J?Re: Calculating linear regression slope in formulaDear @txnelson, can you show how to tweak the above code to run fit models on rows selections based on each 'wafer' id/value, instead of the random sampling? Thanks!Re: Calculating linear regression slope in formulaHi,
You mention a "span of 3", if we want to modify it to be moving slope of "N" samples, which value will need to change?
Thanks!
Re: Calculating linear regression slope in formulaTake a look at the Linear Regression() function, which could be used in a formula. It might be something you can use.
Re: Calculating linear regression slope in formulaHi,
Is there any plan to put slope and intercept into formula builder of a future JMP version w/ a more straight forward direct function ?
thx.
Re: Calculating linear regression slope in formulaExcellent, that works, thank you
Re: Calculating linear regression slope in formulaThis modified formula produces a moving slope of span 8:
If( Row() > 7,
x = J( 8, 1, 1 ) || :height[Index( Row() - 7, Row() )]`;
y = :weight[Index( Row() - 7, Row() )]`;
(Inv( x` * x ) * x` * y)[2];
)
Re: Calculating linear regression slope in formulaHi
I am trying to apply your formula to calculate the moving slope with a window of 8, by which I mean (for Row() > 7), I want to calculate the slope of the linear regression for the first set of x and y data points [1 to 8], and then the next set of 8 data points incrementing by 1 [2 to 9], and so on. I am trying to calculate the maximum slope to get the initial rate of an enzymatic reaction.
Please can you tell me how you would modify your script to achieve this? I have tested your formula and I obtain the slope for all data points from Row 1 to Row n, rather than a moving window of defined size.
Thank you in advance
Re: Calculating linear regression slope in formulawow~~ Awesome!!!!!!
Thank you so much!!!!! : )
Re: Calculating linear regression slope in formulaThis is exactly what I was looking for. I had no idea these operations were available!
Thank you!
Re: Calculating linear regression slope in formulaYou could also use a column formula. I opened Big Class.jmp from the Sample Data folder. (Select Help > Sample Data Library > Big Class.jmp > Open) I crated a new numeric column called moving slope. This example computes the moving slope with a span of 3 for Y = weight and X = height with the following column formula:
If( Row() > 1,
x = J( Row(), 1, 1 ) || :height[Index( 1, Row() )]`;
y = :weight[Index( 1, Row() )]`;
(Inv( x` * x ) * x` * y)[2];
,
Empty()
)
Then plot moving slope versus row in Graph Builder.
Re: Calculating linear regression slope in formulaTry plugging this formula into a new formula column (this will calculate the slope up to the current row):
(Row() * Summation( i = 1, Row(), :Time[i] * :Volume[i] )
-Summation( i = 1, Row(), :Time[i] ) * Summation( i = 1, Row(), :Volume[i] )) / (
Row() * Summation( i = 1, Row(), :Time[i] ^ 2 )
-Summation( i = 1, Row(), :Time[i] ) ^ 2)
To do the second part of your question would only require changing the index on the summation, i.e. from i=Row()-N to Row()+N.
Hope this helps.
-Jerry
Re: Calculating linear regression slope in formulaHere is a script that gives you the output you want, it just does it a different way. It uses the Fit Model Platform and loops through it several time, increasing the N.
Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA\semiconductor capability.jmp" );
// Create an output table to hold the results
dtOut = New Table( "summary",
New Column( "Instance" ),
New Column( "N" ),
New Column( "RSquare" ),
New Column( "RSM" ),
New Column( "Intercept" ),
New Column( "b1" ),
New Column( "b2" )
);
Loop Counter = 0;
// Loop across the different sizes
For( i = 5, i <= 111, i = i + 5,
Loop Counter++;
// Create a random sample data table from the original data table
dt << New Column( "random", formula( Random Uniform() ) );
dt << sort( by( dt:random ), , Order( Ascending ), replace table( 1 ) );
dt << delete columns( "Random" );
dt << select where( Row() <= i );
dtSample = dt << subset( invisible, selected rows( 1 ), selected columns( 0 ) );
// Run the regression
fm = dtSample << Fit Model(
Y( :NPN1 ),
Effects( :PNP1, :PNP2 ),
Personality( "Standard Least Squares" ),
Emphasis( "Minimal Report" ),
Run(
:NPN1 << {Summary of Fit( 1 ), Analysis of Variance( 1 ), Parameter Estimates( 1 ),
Lack of Fit( 0 ), Plot Actual by Predicted( 0 ), Plot Regression( 0 ),
Plot Residual by Predicted( 0 ), Plot Effect Leverage( 0 )}
)
);
// Capture the output data
dtOut << Add Rows( 1 );
dtOut:Instance[N Rows( dtOut )] = Loop Counter;
dtOut:N[N Rows( dtOut )] = i;
dtOut:RSquare[N Rows( dtOut )] = (Report( fm )["Summary of Fit"][1][2] << get)[1];
dtOut:RSM[N Rows( dtOut )] = (Report( fm )["Summary of Fit"][1][2] << get)[3];
dtOut:Intercept[N Rows( dtOut )] = (Report( fm )["Parameter Estimates"][1][3] << get)[1];
dtOut:b1[N Rows( dtOut )] = (Report( fm )["Parameter Estimates"][1][3] << get)[2];
dtOut:b2[N Rows( dtOut )] = (Report( fm )["Parameter Estimates"][1][3] << get)[3];
// Clean up after our selfs
fm << close window;
Close( dtSample, nosave );
);Re: Calculating linear regression slope in formulaFantastic solution to EXACTLY what I wanted to do. Thank you!!
Re: Calculating linear regression slope in formulaFolks interested in this problem may want to consider supporting the wish idea:
https://community.jmp.com/t5/JMP-Wish-List/Moving-Rolling-Window-Statistics/idi-p/722620
