28662_en-US

liliconcom

Normal Distributions and Transformations

Hi Everyone,

 

I have some measured data and when I try a continuous normal fit, I can see that my data is not normal.  However, I can see from the Goodness-of-Fit Test that the data is from the Johnson Su distribution.

 

This distribution has two shape, one location and one scale parameter.  From my research online, I can see how to calculate variance from these parameters and from that the standard deviation.  I used Excel to calculate that, but is there a way in JMP to do this?  From my understanding, the Summary Statics table from the "Distributions" analysis calculates these statistics assuming the data is from the normal distribution.

 

Thanks in advance!

 

Natalie

Re: Normal Distributions and TransformationsThank you for the reply, txnelson. I didn't think of that and I wouldn't have expected that to be the case. I am wondering what the value of that transformation is then if it makes it impossible to reverse the transformation with accuracy. If I develop a model off of the transformed data, I need to know what the output of that model is in the original data form. It seems silly, then, that JMP would even include this transformation as an option. Kudos on the original script. I will save it for future use as needed.Re: Normal Distributions and Transformations

The script will not work for the Denisity Transformation.  The function produces multiple values for the same density value and the code can not handle that.  See graph below

 

Re: Normal Distributions and Transformations

Perhaps it is my dataset then? In short, I was picking a value from my data set and pulling out the lognormal transformed value for testing/validating the code. My data set is below. A value of 7 corresponds to a transformed value of 0.112832236739791, therefore I should be able to use the transformed value as the target in the code and the script should spit out 7. Correct?

 

 

Names Default to Here(1);
dt=New Table( "Test",
 Add Rows( 30 ),
 New Column("PNP3", 
  Numeric,
  "Continuous",
  Format( "Best", 10 ),
  Set Values(
   [12, 4, 12, 13, 7, 5, 5, 15, 5, 5, 5, 15, 7, 7, 16, 5, 9, 11, 18, 8, 16, 12, 7, 14, 6, 11, 7, 10, 14, 10]
  )
 ),
 New Column( "LogNormal Density PNP3",
  Numeric,
  "Continuous",
  Format( "Best", 12 ),
  Formula(
   Normal Density( Log( :PNP3 ), 2.17970974904887, 0.43804625261609 ) / :PNP3
  )
 )
);

/***********************************************************************/
/*                                                                     */
/* The getformula column retrieves the formula from the translation    */
/* column and replaces the Original Column name in the formula with    */
/* string "__value__".  The value of this variable is what will be     */
/* evaluated in the successive approximations done by the script.      */
/*                                                                     */
/***********************************************************************/

getformula = Function( {ColName, FormulaColName},
 {ColName, FormulaColName, TheFormula, coloncolname}, 
 //__value__ = .;
 // Get the transformed data columns formula as a literal string
 TheFormula=Column( FormulaColName ) << Get Formula ;
 
 // Check to see that a formula was found
 If( Is Empty( TheFormula ) == 1 ,
  Dialog(
   "   The column specified as",
   "the Transformed Column does",
   "      not contain a formula. ",
   " ",
   "    Please rerun and select",
   "       the correct column"
  );
  Throw();
 );
 TheFormula=char( Column( FormulaColName ) << Get Formula );

 // Get the actual name of the orignal column since
 // the upper,lower case and spacing is critical in determining
 // where in the formula the column name actually occurs
 ColName = Column( ColName ) << Get Name;
 

 // Determine if the reference to the column name in the 
 // formula is a simple :colname reference or a complex
 // reference :Name(\!"colname\!")
 // If the column name isn't found set the return code to -1
 If(
  Contains( TheFormula, ":" || ColName ), ColonColName = ":" || ColName, // Else
  Contains( TheFormula, ":Name(\!"" || ColName || "\!")" ), ColonColName = ":Name(\!"" || ColName || "\!")", // Else
  rc = -1
 );
  
 // Replace all of the column references in the formula with
 // the string "(__value__)" so that when the formula is 
 // evaluated later, it will take the then value of the memory
 // variable called __value__ and use it in the formula
 If( Contains( TheFormula, ColonColName ) > 0,
  While( Contains( TheFormula, ColonColName ) > 0, TheFormula = Munger( TheFormula, 1, ColonColName, "(__Value__)" ) ),
  Dialog(
   "   The column specified as",
   "the Transformed Column does",
   "     not contain a reference",
   "      to the original column.",
   "           in it's formula.",
   " ",
   "    Please rerun and select",
   "       the correct column"
  );
  Throw();
 );
 TheFormula;
);

/***********************************************************************/
/*                                                                     */
/* The gettrans function evaluates the formula in the transformed      */
/* and converts the formula into a generic form for repeated use in    */
/* running of the script.                                              */
/*                                                                     */
/***********************************************************************/

gettrans = Function( {ColName, FormulaColName, TheTarget, Theformula},
//colname="PNP3";formulacolname="Johnson Sl Transform PNP3"; Thetarget=johnsonmean;theformula=myformula;
 {ColName, FormulaColName, TheFormula, High, Low, TheTarget, TheMax, Themin, __value__}, 

 // The program uses successive approximations to determine the different 
 // parametrics.  The way it works is that it calculates the needed parameter
 // such as Mean, or Standard Deviation, and then by using successive 
 // approximations from the original column's values, and passing those
 // values through the columns formula, when the approximation value matches
 // the calculated value from the transformed column, the retransformed value
 // has been found

 // Set the extreem values
 High = Col Maximum( If( Excluded( Row State( Empty() ) ) == 0, Column( ColName ), . ) );
 Low = Col Minimum( If( Excluded( Row State( Empty() ) ) == 0, Column( ColName ), . ) );
 
 // Make a guess at the first value 
 __value__ = Mean( High, Low );

 // Iterate the guessing for up to 100 times, adjusting by 1/2 on each loop
 For( i = 1, i <= 100, i++,
  TheResult = Eval( Parse( theformula ) );

  If(
   TheResult > TheTarget, High = __value__,
   TheResult < TheTarget, Low = __value__,
   Break()
  );
  If( High == Low, Break() );
  __value__ = Mean( High, Low );
 );
 
 __value__; // Expose the return value
); // End of function gettrans

myFormula = getformula( "PNP3","LogNormal Density PNP3" );

MatchValue = 7;

TargetValue = 0.112832236739791;

dt<<RUN formulas=""></RUN>

 

After running the code, here is the log output

 

MatchValue = 7;
TargetValue = 0.112832236739791;
Gettrans("PNP3", "LogNormal Density PNP3", 0.112832236739791, myFormula) = 18;

 

The output of the Gettrans function should be 7. The value of 18 that the code is outputting corresponds to a transformed value of 0.0135701784925184.

Re: Normal Distributions and Transformations

I don't understand what the issue is?  Since all the code is doing is taking taking values from the original column and passing them through the formula until the result is equal to the target value from the transformed column(typically the mean of the new distribution).  I reworked the script from my previous responses, but this time, using the Log Normal Density formula determined by the Distribution platform, and the code seems to work for me

Names Default to Here(1);
dt=New Table( "Test",
 Add Rows( 100 ),
 New Column("PNP3", 
  Numeric,
  "Continuous",
  Format( "Best", 10 ),
  Set Values(
   [130.378809500886, 132.736937590235, 136.831952164704, 136.969154239163,
   136.622623419893, 137.480356512481, 138.094011176166, 142.10586919757,
   134.750041237898, 129.719685093372, 136.520172900026, 134.778624216716,
   136.991131754497, 140.420940666186, 143.257827228986, 132.213765487819,
   144.671172276421, 134.703176752174, 136.744212473486, 137.187637284245,
   139.788226018039, 139.372617199359, 136.112484488985, 142.809097034809,
   137.799370430979, 138.482493403385, 135.236982575217, 136.251777781494,
   130.768356509183, 138.248554728086, 139.77894292052, 134.25405167366,
   147.680687116943, 131.351711991517, 132.84274608728, 129.925216236015,
   133.47206414316, 143.339607103893, 145.341236691691, 139.200187547183,
   142.775409342827, 140.276696563388, 130.623979847275, 140.899814366103,
   136.839389290019, 137.239125319, 133.5265281641, 139.356927352471,
   130.278640163464, 144.604061001983, 135.286715550332, 134.465744849174,
   131.37612790407, 131.830655714309, 140.69724979219, 142.88152043774,
   135.253839945611, 127.349434776131, 129.499730399113, 128.447533754611,
   130.916853702805, 134.599575929218, 140.761701916093, 136.870661473033,
   138.253066182015, 140.403627077024, 134.522643679098, 124.842978178703,
   131.803059455053, 125.886786494664, 133.013566701611, 136.940299158936,
   133.263913979648, 144.695171321015, 149.541020434399, 144.503845528521,
   136.086324063453, 139.530943158798, 138.421460162451, 133.180943784947,
   142.166796633818, 142.676541730822, 135.723943440339, 143.957996114985,
   145.712158558794, 138.38937502716, 140.535304753392, 142.140619481175,
   131.379414509756, 144.949299702964, 133.349854687882, 139.470639804255,
   140.160558367008, 137.130662662612, 145.692632562344, 131.870848869966,
   136.391733804566, 134.219740661271, 139.021827550389, 147.958038547157]
  )
 ),
 New Column( "LogNormal Density PNP3",
  Numeric,
  "Continuous",
  Format( "Best", 12 ),
  Formula(
   Normal Density( Log( :PNP3 ), 4.92072961984847, 0.0370684539915594 ) / :PNP3
  )
 )
);

/***********************************************************************/
/*                                                                     */
/* The getformula column retrieves the formula from the translation    */
/* column and replaces the Original Column name in the formula with    */
/* string "__value__".  The value of this variable is what will be     */
/* evaluated in the successive approximations done by the script.      */
/*                                                                     */
/***********************************************************************/

getformula = Function( {ColName, FormulaColName},
 {ColName, FormulaColName, TheFormula, coloncolname}, 
 //__value__ = .;
 // Get the transformed data columns formula as a literal string
 TheFormula=Column( FormulaColName ) << Get Formula ;
 
 // Check to see that a formula was found
 If( Is Empty( TheFormula ) == 1 ,
  Dialog(
   "   The column specified as",
   "the Transformed Column does",
   "      not contain a formula. ",
   " ",
   "    Please rerun and select",
   "       the correct column"
  );
  Throw();
 );
 TheFormula=char( Column( FormulaColName ) << Get Formula );

 // Get the actual name of the orignal column since
 // the upper,lower case and spacing is critical in determining
 // where in the formula the column name actually occurs
 ColName = Column( ColName ) << Get Name;
 

 // Determine if the reference to the column name in the 
 // formula is a simple :colname reference or a complex
 // reference :Name(\!"colname\!")
 // If the column name isn't found set the return code to -1
 If(
  Contains( TheFormula, ":" || ColName ), ColonColName = ":" || ColName, // Else
  Contains( TheFormula, ":Name(\!"" || ColName || "\!")" ), ColonColName = ":Name(\!"" || ColName || "\!")", // Else
  rc = -1
 );
  
 // Replace all of the column references in the formula with
 // the string "(__value__)" so that when the formula is 
 // evaluated later, it will take the then value of the memory
 // variable called __value__ and use it in the formula
 If( Contains( TheFormula, ColonColName ) > 0,
  While( Contains( TheFormula, ColonColName ) > 0, TheFormula = Munger( TheFormula, 1, ColonColName, "(__Value__)" ) ),
  Dialog(
   "   The column specified as",
   "the Transformed Column does",
   "     not contain a reference",
   "      to the original column.",
   "           in it's formula.",
   " ",
   "    Please rerun and select",
   "       the correct column"
  );
  Throw();
 );
 TheFormula;
);

/***********************************************************************/
/*                                                                     */
/* The gettrans function evaluates the formula in the transformed      */
/* and converts the formula into a generic form for repeated use in    */
/* running of the script.                                              */
/*                                                                     */
/***********************************************************************/

gettrans = Function( {ColName, FormulaColName, TheTarget, Theformula},
//colname="PNP3";formulacolname="Johnson Sl Transform PNP3"; Thetarget=johnsonmean;theformula=myformula;
 {ColName, FormulaColName, TheFormula, High, Low, TheTarget, TheMax, Themin, __value__}, 

 // The program uses successive approximations to determine the different 
 // parametrics.  The way it works is that it calculates the needed parameter
 // such as Mean, or Standard Deviation, and then by using successive 
 // approximations from the original column's values, and passing those
 // values through the columns formula, when the approximation value matches
 // the calculated value from the transformed column, the retransformed value
 // has been found

 // Set the extreem values
 High = Col Maximum( If( Excluded( Row State( Empty() ) ) == 0, Column( ColName ), . ) );
 Low = Col Minimum( If( Excluded( Row State( Empty() ) ) == 0, Column( ColName ), . ) );
 
 // Make a guess at the first value 
 __value__ = Mean( High, Low );

 // Iterate the guessing for up to 100 times, adjusting by 1/2 on each loop
 For( i = 1, i <= 100, i++,
  TheResult = Eval( Parse( theformula ) );

  If(
   TheResult > TheTarget, High = __value__,
   TheResult < TheTarget, Low = __value__,
   Break()
  );
  If( High == Low, Break() );
  __value__ = Mean( High, Low );
 );
 
 __value__; // Expose the return value
); // End of function gettrans

myFormula = getformula( "PNP3","LogNormal Density PNP3" );

dt<<RUN formulas=""></RUN>

Re: Normal Distributions and TransformationsHi txnelson,

I have a question about the program/script you linked. It will reverse a johnson transormation easily, but it does not work for other transformations. In particular, the logNormal Density transformation. Would you mind if I PM you to see if there is a way the script could be modified to work?Re: Normal Distributions and Transformations

The column called "Johnson Sl Transform X By Timepoint" is the transformed data.  It is this column that contains the normalized data.  You would run your analyses based upon this column.  

Therefore, when you run an anlysis to determine if there is a difference between Timepoint1 and Timepoint2 data, you would run it on the transformed data column.  Now the issue is, that it will report values in the scale of the transformed data.  Thus, in order to relate it back to the original column X, you run the script, entering in the data point value from the analysis, and getting back, what the value in terms of X.

Re: Normal Distributions and Transformations

Of course the script should be run separately as I posted before. I just enclosed the single file (Test.jmp) for convenience. Shouldn't your script record the normal data after Johnson SI transformation in the third column called Y (in that data table file) in order to run it and see its normal shape?

The issue is that there is no command line in your script to save the normal data in that data table. That's why I have been asked you about it.

If you look one of those split files and run your script, the third column Y is still in blank. I thought each split data table could be completed by the normal data after Johnson SI transformation in the third column Y because after that I would like to combine them again into one file to proceed my stats analysis.

 

Timepoint 1:

JohnsonMean = -3.80945275324507e-15;

Col Mean(:X) = -7.4569375;

Gettrans("X", "Johnson Sl Transform X By Timepoint", JohnsonMean, myFormula) = -10.7617217296089;

 

Timepoint 2:

JohnsonMean = 7.27196081129478e-15;

Col Mean(:X) = 3.257375;

Gettrans("X", "Johnson Sl Transform X By Timepoint", JohnsonMean, myFormula) = 1.87687074670016;

 

Thank you for your help.

Re: Normal Distributions and Transformations

Your question's solution has to bring in a human decision about whether or not you know for a fact that the different Timepoint groups of data are from different overall distributions?  My assumption was that the best determination of whether or not the overall distribution is from a normal distribution was to combine all of the data into a single evaluation.  And the follow on from that is to use the single transformation for all of the data, and then to perform subsequent analyses on that transformed data.  Any true differences should be identified within the transformed data.  Now, if you have an independent apriori knowlege of the Timepoint groups being from separate distributions, then you will need to subset the data into the separate Timepoint groups, and run the script separately on the different subsets.

Re: Normal Distributions and Transformations

Hi Jim,

I think there is a misunderstanding here. Although, the Log file shows: 

JohnsonMean = 1.72084568816899e-15;
Col Mean(:X) = -2.09978125;
Gettrans("X", "Johnson Sl Transform X By Timepoint", JohnsonMean, myFormula) = 1.1;

Your script doesn't save the normal data from "Johnson SI Transformation By Timepoint" column into the Y column in my data table.

Best,

Reinaldo

Re: Normal Distributions and Transformations

Go back to the data table with the combined timepoints.  Then go to Distribution and run it against the transformed data.

 

Then, go to 

     Help==>Books==>Discovering JMP

 

You have to learn more about how JMP works!

Re: Normal Distributions and Transformations

I created two subsets: one per timepoint. Where do I use the outputs from Log file to get and plot the normal data?

Re: Normal Distributions and Transformations

1. Okay, I will run it separately.

2. I have those outputs you mentioned (Log file) and the Johnson SI Transform X By Timepoint (= non-normal distribution), but no data assigned to Y Column.

Re: Normal Distributions and Transformations

1. If you need to get separate results for each timepoint, you will need to subset the data table into separate tables for each timepoint, and then run the script on each of them separately.

2. You now need to proceed with your analysis using the transformed data column, as if it was your Y column..

3. Read and digest the document

     Help==>Books==>Discovering JMP

Re: Normal Distributions and Transformations

Sorry, I didn't tell you the Log file because I couldn't find it. I am beginner in JMP. Now I can find it: View -> Log.

Okay, I added that command line and I got the same result!

What's the next step?

 

PS: Doesn't the script consider two timepoint separately to calculate those parameters? Perhaps, the new parameter called "Timepoint" would be added in those functions "getformula" and "gettrans".

Re: Normal Distributions and Transformations

1. I had requested in one of my last responses, if you had looked in the Log to see if there were any errors.  I ran the script as you had included in the attached data table from your last response.  In looking in the log, it specified:

The namespace "dt" is not defined in access or evaluation of 'dt:Johnson Sl Transform X By Timepoint' , 
dt:Johnson Sl Transform X By Timepoint/*###*/

Taking that, and looking into the script, the "dt" reference is in the line

JohnsonMean = Col Mean( dt:Johnson Sl Transform X By Timepoint );

dt in this context is looking for a variable called dt, which is pointing to the data table to read the data from.  So the correction to the program, is to add the following line prior to the use of the reference in the code. 

dt=current data table();

I added it in my copy of the code, as the first line in the script.

Once I did that, in the log I got the following results from the script

JohnsonMean = 1.72084568816899e-15;
Col Mean(:X) = -2.09978125;
Gettrans("X", "Johnson Sl Transform X By Timepoint", JohnsonMean, myFormula) = 1.1;

Please Note......the results were in the log file.......

 

 

Re: Normal Distributions and Transformations

Thank you for your post, Jim! I think now the Capability Analysis is understood!

 

Regarding the transformation from non-normal data to normality, I created an example (Test.jmp) which contains two timepoints and X as the non-normal variable. I followed your procedure:

 

1. Analyse -> Distribution

2. X as "Y, Columns" and Timepoint as "By"

3. I clicked on Distributions Timepoint -> Stack

4. For each timepoint, I clicked on the red triangle and "Continuous Fit -> Johnson SI"

5. I clicked on "Fitted Johnson SI -> Save Transformed"

 

Then JMP generated the column "Johnson SI Transform X By Timepoint"

 

So, that's the point: I tried to run your script that is in the top left of the Test.jmp, but nothing happened.

 

May you explain what I did wrong in that script, please?

 

PS: I would like that the Y column represents the normality of X after Johnson SI transformation.

 

Many thanks!

Re: Normal Distributions and Transformations

They typically are applied to manufacturing processes

Re: Normal Distributions and Transformations

Am I correct to think the Cp and Cpk aren't applied to the Psychology field, but to Engineering, Process Control and so on?

Re: Normal Distributions and Transformations

Cp/Cpk analysis is completely dominated by the specification limits.  And yes, you can get 3k spec limits generated, however, using them on such a limited measurment space seems to me to be only and exercise in producing numbers, not anything of any real meaning. 

Re: Normal Distributions and Transformations

Regarding the Capability Analysis, I saw it set up in one of your previous post when you explained "the steps of how to transform non to normal data" to me in the URL: https://community.jmp.com/t5/Discussions/Normal-Distributions-and-Transformations/td-p/28662/page/2?lightbox-message-images-53315=9889i529042A0AA61277A  (page 2)

 

It seems the "Capability Analysis" is measuring if a system can meet a set of specifications or requirements in a statistical process control, isn't it? So it doesn't apply to my case.

 

 

 

Re: Normal Distributions and Transformations

I will ask again, but more directly, why do you think you need to run a Capability Analysis on the data?

 

That aside, concering the script not providing you with any visible output, are there any messages in the Log, indicating any errors?

Re: Normal Distributions and Transformations

1. Although, I read the Capability Analysis "measures the conformance of a process to defined specification limits", unfortunately I didn't understand exactly its meaning in Statistics.

 

2. Therefore, when the new column "Johnson SI" is created, I really need to run your script. (Initially, I thought it wasn't necessary to run it). Anyway, I tried to run it. In my case, the data table is in column format. I attempted to adapt your script deleting the first part, which created the data table, and inserting the names of columns rather than PNP3 and "Johnson SI Transform PNP3", and I saved it on the top left box (below the name of the data file) inside the data file. When I ran it, nothing happened. (I don't know if I saved it correctly or occurred any parsing error in regards to the column names since they weren't in the script, but in the data file). Please, Jim (@txnelson), how can I run your script using a column-format data file? 

 

Thank you very much!

Re: Normal Distributions and Transformations

1.  You will have to tell me what you want to do with the Capability Analysis.  My background is in both psycology and engineering.  You could use Cp/Cpk to get somewhat of an estimate of the capability is, but since you are using it within a single set of data, and not based upon real specification limits, I would find the statistic somewhat suspect.

2. The purpose of transforming the data is to transform the data into a distribution that is normally distributed.  The rational for this, is that many of the statistical tests are based upon the data being normally distributed.  The Wilconon  statistic is a NonParametric test.  It does not require a normal distribution.

Re: Normal Distributions and Transformations

I understood that I needed to run your script. If I don't run it I can see that column normalised, but...

 

1. In which situation should I use those parameters from Capability Analysis?

2. When I ran the non-normal data in a repeated-measures design I could find a significant effect on the variable that I analysed across the timepoints through the results from the nonparametric muliple comparison test (e.g., Wilcoxon Each Pair). However, when I ran the data, which column had been transformed using Johnson SI, that significant effect disappeared across the timepoints. So, which result should I trust?

Re: Normal Distributions and Transformations

Yes.....

Re: Normal Distributions and Transformations

Okay. After that should I run your script to transform that Johnson SI column to normality?

Re: Normal Distributions and Transformations

Select "Save Transformed" from the red triangle in the Fitted Johnson SI outline box.  It will create a new column of data using the Johnson SI transformation

 

Re: Normal Distributions and Transformations

Hi Jim (@txnelson),

 

Thank you for your post. I read that link you had suggested. As I understood, I could fit the distribution using Johnson SI through Capability Analysis, selecting "Johnson SI" on the or I could run the Fit Distribution and then clicking on the red triangle Capability Analysis I could find the Spec Limits as you suggested.

 

I agree with you that Spec Limits are traditionally determined from the knowing of the measurement data. However, I believe that it applies to the engineering field. In Psychology field, it's hard to have any idea about those limits, but only the data collected. In this way, I think I should run the Fit Distribution for Johnson SI and then click on the red triangle ==> Set Spec Limits for K Sigma, selecting K value = 3.

 

When I do it, I get the Quantile Sigma and the fitting plot for Johnson SI. However, I haven't got the normal shape of my data, but only the parameters (Spec Limits) from Capability Analysis. What's the next step, please?

 

Thank you.

Re: Normal Distributions and Transformations

1. I suggest that you go to the JMP Webpage and read the documentation on the Distribution Platform

     JMP Webpage==Support==>Online Documentation==Basic Analysis==>Distributions

It will give you a very good education on what the tool can do for you.  You stated that you are using JMP 10.  The most recent JMP version is 14(to be released shortly) and the documentation on the web is JMP 14.  However, you will find, almost all of what you will read about in the documentation was available in JMP 10.

2. The determination of what the shape of your data's distribution (i.e. Normal, Johnson SI, Log Normal, etc.) is not determined by the Capability Analysis.  It is actually the opposite.  The shape of the distribution determines what formulas to use to calcuate the Capability of the data.

3. You seem to not understand what the limits are in a capability analysis.  Spec Limits are traditionally determined from the knowledge of the measurement data.  That is, if you are measuring voltage of an electrical component, the design of the part would state that for the part to work properly, the voltage needs to be between the Lower Specification Limit (LSL) and the Upper Specification Limit(USL).  It would be these limits that would be used in the determination of how Capable the process is.

4. In many cases all you want to get out of the Distribution Platform, is to get the determination if the data are normally distributed.  Why this is important, is because many statistical analyses have the assumption that the data are normally distributed.  So the purpose of transforming the data is to change the data into a normal distribution, so the statistical tests can provide more accurate results.

Re: Normal Distributions and Transformations

In addition, I am a beginner in JMP and this is my first transformation I try to do with an additional complexity that it refers to a repeated-measures design.

 

I couldn't find out the relationship between "Capability Analysis" and "Fitted Johnson SI" yet. I mean If I select the "Capability Analysis" first and assuming I enter those aforementioned parameters (my previous post) correctly, then I will select the "Continuous Fit==>All", choosing Johnson SI. In this way, does JMP take into account that information entered in "Capability Analysis" to evaluate the "Fitted Johnson SI"?

 

PS: My data is in tall (or long) format: rows are represented by subjects; the first column is Timepoints (T1, T2, ...) and the second column is OUT. When I clicked on Fitted Johnson SI==>Save Transformed for each timepoint, the same column called "Johnson SI Transform OUT by Timepoints" was completed with those transformed scores. After that I tried to run the stats analysis in Fit Model using that "Johnson SI Transform OUT by Timepoints" column in the "Pick Role Variables" box and {Timepoints, subject& Random, subject*Timepoints & Random} in the "Contruct Model Effects" box, but I couldn't get any relevant result.

 

I suppose I would need to run your script immediatly after getting the "Johnson SI Transform OUT by Timepoints" (transformed data) column. Am I right?

 

That's my next doubt: understanding your procedure described as solution. I ran your example and it was amazing! I need to learn how to "take the std from the transformed data, calculate what the values above and below the mean are for 1, 2, 3, etc. stds, and then reverse the transformation back to the original data.".

 

Thank you very much for your attention and valuable help!

Re: Normal Distributions and Transformations

Hi Jim ( @txnelson ), thank you very much for your explanation!

 

I have the following problem:

 

1. For my study case, the non-normal outcome called OUT contains different timepoints (T1, T2, ...) because I have a repetead-measure design. So, when I do Analysis==>Distribution, I suppose I need to select that variable OUT as "Y, Columns" and the between-subject (e.g., Timepoints) as "By" in the "Cast Selected Columns into Roles" dialog, don't I?

 

2. If I do it then I will have a plot OUT vs. each timepoints such as OUT vs. T1; OUT vs. T2 and so on. In JMP v.10, I need to follow your instruction for each plot because it cannot do it automatically. It's okay. I select on Distributions Timepoint=T1==>Stack, and all plots are shown in the horizontal axis.

 

Doubt: Should I first select "Capability Analysis" or "Continuous Fit==>All"? I mean when I select "Capability Analysis" the dialog appears to enter the following parameters: "Lower Spec Limit", "Target" and "Upper Spec Limit", and I don't know which values I have to define. Are they based on the Box plot, excluding the outliers?

 

Thank you.

 

Re: Normal Distributions and Transformations

The Distribution Platform allows one to evaluate what Distribution a given column is, and then, it may have a method to transform the data to a normal distribution.  Here are the steps

1. Run the Distribution Platform, selecting the desired column(s).  My example comes from the Semiconductor Capability sample data table installed when JMP is installed.

     Analyze==>Distribution

2. Once the output is displayed, select from the 

     Continuous Fit==>All

3.  The platform will give you, in order, what distributions best fit the data.  In this case, the LogNormal is selected.  Unfortunatly, the LogNormal does not have the ability to create a transformed version of the data, so unselect it, and select Johnson SI.

4. Now, click on the red triangle for "Fitted Johnson SI and select Save Transformed

 

JMP has now saved a transformed version of the data into a new column, which you can now use for your analyses

Re: Normal Distributions and Transformations

Hi Jim (@txnelson),

I couldn't understand how to transform a non-normal data to a normal distribution in JMP. Please may you explain the steps from a raw non-normal data to me?

Thank you.

Re: Normal Distributions and Transformations

Here is a complete program that returns a value based upon a statistic result from a transformed set of data.  I hope this help

Names Default to Here(1);
dt=New Table( "Test",
 Add Rows( 100 ),
 New Column("PNP3", 
  Numeric,
  "Continuous",
  Format( "Best", 10 ),
  Set Values(
   [130.378809500886, 132.736937590235, 136.831952164704, 136.969154239163,
   136.622623419893, 137.480356512481, 138.094011176166, 142.10586919757,
   134.750041237898, 129.719685093372, 136.520172900026, 134.778624216716,
   136.991131754497, 140.420940666186, 143.257827228986, 132.213765487819,
   144.671172276421, 134.703176752174, 136.744212473486, 137.187637284245,
   139.788226018039, 139.372617199359, 136.112484488985, 142.809097034809,
   137.799370430979, 138.482493403385, 135.236982575217, 136.251777781494,
   130.768356509183, 138.248554728086, 139.77894292052, 134.25405167366,
   147.680687116943, 131.351711991517, 132.84274608728, 129.925216236015,
   133.47206414316, 143.339607103893, 145.341236691691, 139.200187547183,
   142.775409342827, 140.276696563388, 130.623979847275, 140.899814366103,
   136.839389290019, 137.239125319, 133.5265281641, 139.356927352471,
   130.278640163464, 144.604061001983, 135.286715550332, 134.465744849174,
   131.37612790407, 131.830655714309, 140.69724979219, 142.88152043774,
   135.253839945611, 127.349434776131, 129.499730399113, 128.447533754611,
   130.916853702805, 134.599575929218, 140.761701916093, 136.870661473033,
   138.253066182015, 140.403627077024, 134.522643679098, 124.842978178703,
   131.803059455053, 125.886786494664, 133.013566701611, 136.940299158936,
   133.263913979648, 144.695171321015, 149.541020434399, 144.503845528521,
   136.086324063453, 139.530943158798, 138.421460162451, 133.180943784947,
   142.166796633818, 142.676541730822, 135.723943440339, 143.957996114985,
   145.712158558794, 138.38937502716, 140.535304753392, 142.140619481175,
   131.379414509756, 144.949299702964, 133.349854687882, 139.470639804255,
   140.160558367008, 137.130662662612, 145.692632562344, 131.870848869966,
   136.391733804566, 134.219740661271, 139.021827550389, 147.958038547157]
  )
 ),
 New Column( "Johnson Sl Transform PNP3",
  Numeric,
  "Continuous",
  Format( "Best", 12 ),
  Set Property( "Notes", "Fitted Johnson Sl" ),
  Formula(
   (Log( (:PNP3 - 44.6529505888426) / 1 ) * 15.3554291860315 + (
   -69.5611150866502)) * 1
  )
 )
);

/***********************************************************************/
/*                                                                     */
/* The getformula column retrieves the formula from the translation    */
/* column and replaces the Original Column name in the formula with    */
/* string "__value__".  The value of this variable is what will be     */
/* evaluated in the successive approximations done by the script.      */
/*                                                                     */
/***********************************************************************/

getformula = Function( {ColName, FormulaColName},
 {ColName, FormulaColName, TheFormula, coloncolname}, 
 //__value__ = .;
 // Get the transformed data columns formula as a literal string
 TheFormula=Column( FormulaColName ) << Get Formula ;
 
 // Check to see that a formula was found
 If( Is Empty( TheFormula ) == 1 ,
  Dialog(
   "   The column specified as",
   "the Transformed Column does",
   "      not contain a formula. ",
   " ",
   "    Please rerun and select",
   "       the correct column"
  );
  Throw();
 );
 TheFormula=char( Column( FormulaColName ) << Get Formula );

 // Get the actual name of the orignal column since
 // the upper,lower case and spacing is critical in determining
 // where in the formula the column name actually occurs
 ColName = Column( ColName ) << Get Name;
 

 // Determine if the reference to the column name in the 
 // formula is a simple :colname reference or a complex
 // reference :Name(\!"colname\!")
 // If the column name isn't found set the return code to -1
 If(
  Contains( TheFormula, ":" || ColName ), ColonColName = ":" || ColName, // Else
  Contains( TheFormula, ":Name(\!"" || ColName || "\!")" ), ColonColName = ":Name(\!"" || ColName || "\!")", // Else
  rc = -1
 );
  
 // Replace all of the column references in the formula with
 // the string "(__value__)" so that when the formula is 
 // evaluated later, it will take the then value of the memory
 // variable called __value__ and use it in the formula
 If( Contains( TheFormula, ColonColName ) > 0,
  While( Contains( TheFormula, ColonColName ) > 0, TheFormula = Munger( TheFormula, 1, ColonColName, "(__Value__)" ) ),
  Dialog(
   "   The column specified as",
   "the Transformed Column does",
   "     not contain a reference",
   "      to the original column.",
   "           in it's formula.",
   " ",
   "    Please rerun and select",
   "       the correct column"
  );
  Throw();
 );
 TheFormula;
);

/***********************************************************************/
/*                                                                     */
/* The gettrans function evaluates the formula in the transformed      */
/* and converts the formula into a generic form for repeated use in    */
/* running of the script.                                              */
/*                                                                     */
/***********************************************************************/

gettrans = Function( {ColName, FormulaColName, TheTarget, Theformula},
//colname="PNP3";formulacolname="Johnson Sl Transform PNP3"; Thetarget=johnsonmean;theformula=myformula;
 {ColName, FormulaColName, TheFormula, High, Low, TheTarget, TheMax, Themin, __value__}, 

 // The program uses successive approximations to determine the different 
 // parametrics.  The way it works is that it calculates the needed parameter
 // such as Mean, or Standard Deviation, and then by using successive 
 // approximations from the original column's values, and passing those
 // values through the columns formula, when the approximation value matches
 // the calculated value from the transformed column, the retransformed value
 // has been found

 // Set the extreem values
 High = Col Maximum( If( Excluded( Row State( Empty() ) ) == 0, Column( ColName ), . ) );
 Low = Col Minimum( If( Excluded( Row State( Empty() ) ) == 0, Column( ColName ), . ) );
 
 // Make a guess at the first value 
 __value__ = Mean( High, Low );

 // Iterate the guessing for up to 100 times, adjusting by 1/2 on each loop
 For( i = 1, i <= 100, i++,
  TheResult = Eval( Parse( theformula ) );

  If(
   TheResult > TheTarget, High = __value__,
   TheResult < TheTarget, Low = __value__,
   Break()
  );
  If( High == Low, Break() );
  __value__ = Mean( High, Low );
 );
 
 __value__; // Expose the return value
); // End of function gettrans

myFormula = getformula( "PNP3","Johnson Sl Transform PNP3" );

JohnsonMean = Col Mean(dt:Johnson Sl Transform PNP3);

show(JohnsonMean,colMean(:PNP3),Gettrans("PNP3","Johnson Sl Transform PNP3",JohnsonMean,myFormula));

Re: Normal Distributions and Transformations

Thanks for sharing this.

I'm only a scripting beginner, so it's a bit too complicated at the moment and I wasn't sure how to use it!

---

 

 

Re: Normal Distributions and Transformations

Here is a function that I pulled out of a running system that uses Successive Approximations to get the resolved values

/***********************************************************************/
/*                                                                     */
/* The getformula column retrieves the formula from the translation    */
/* column and replaces the Original Column name in the formula with    */
/* string "__value__".  The value of this variable is what will be     */
/* evaluated in the successive approximations done by the script.      */
/*                                                                     */
/***********************************************************************/

getformula = Function( {ColName, FormulaColName},
 {ColName, FormulaColName, TheFormula, coloncolname}, 
 //__value__ = .;
 // Get the transformed data columns formula as a literal string
 TheFormula=Column( FormulaColName ) << Get Formula ;
 
 // Check to see that a formula was found
 If( Is Empty( TheFormula ) == 1 ,
  Dialog(
   "   The column specified as",
   "the Transformed Column does",
   "      not contain a formula. ",
   " ",
   "    Please rerun and select",
   "       the correct column"
  );
  Throw();
 );
 TheFormula=char( Column( FormulaColName ) << Get Formula );

 // Get the actual name of the orignal column since
 // the upper,lower case and spacing is critical in determining
 // where in the formula the column name actually occurs
 ColName = Column( ColName ) << Get Name;
 

 // Determine if the reference to the column name in the 
 // formula is a simple :colname reference or a complex
 // reference :Name(\!"colname\!")
 // If the column name isn't found set the return code to -1
 If(
  Contains( TheFormula, ":" || ColName ), ColonColName = ":" || ColName, // Else
  Contains( TheFormula, ":Name(\!"" || ColName || "\!")" ), ColonColName = ":Name(\!"" || ColName || "\!")", // Else
  rc = -1
 );
  
 // Replace all of the column references in the formula with
 // the string "(__value__)" so that when the formula is 
 // evaluated later, it will take the then value of the memory
 // variable called __value__ and use it in the formula
 If( Contains( TheFormula, ColonColName ) > 0,
  While( Contains( TheFormula, ColonColName ) > 0, TheFormula = Munger( TheFormula, 1, ColonColName, "(__Value__)" ) ),
  Dialog(
   "   The column specified as",
   "the Transformed Column does",
   "     not contain a reference",
   "      to the original column.",
   "           in it's formula.",
   " ",
   "    Please rerun and select",
   "       the correct column"
  );
  Throw();
 );
 TheFormula;
);

.

/***********************************************************************/
/*                                                                     */
/* The gettrans function evaluates the formula in the transformed      */
/* and converts the formula into a generic form for repeated use in    */
/* running of the script.                                              */
/*                                                                     */
/***********************************************************************/

gettrans = Function( {ColName, FormulaColName, TheTarget, Theformula},
 {ColName, FormulaColName, TheFormula, High, Low, TheTarget, TheMax, Themin, __value__}, 

 // The program uses successive approximations to determine the different 
 // parametrics.  The way it works is that it calculates the needed parameter
 // such as Mean, or Standard Deviation, and then by using successive 
 // approximations from the original column's values, and passing those
 // values through the columns formula, when the approximation value matches
 // the calculated value from the transformed column, the retransformed value
 // has been found

 // Set the extreem values
 High = Col Maximum( If( Excluded( Row State( Empty() ) ) == 0, Column( ColName ), . ) );
 Low = Col Minimum( If( Excluded( Row State( Empty() ) ) == 0, Column( ColName ), . ) );
 Highm = 999999999999999999999999;
 Lowm = -999999999999999999999999;
 If( Highm > High,
  High = Highm
 );
 If( Lowm < Low, low = lowm );
 
 // Make a guess at the first value 
 __value__ = Mean( High, Low );

 // Iterate the guessing for up to 100 times, adjusting by 1/2 on each loop
 For( i = 1, i <= 100, i++,
  TheResult = Eval( Parse( theformula ) );
  If(
   TheResult > TheTarget, High = __value__,
   TheResult < TheTarget, Low = __value__,
   Break()
  );
  If( High == Low, Break() );
  __value__ = Mean( High, Low );
 );
 
 __value__; // Expose the return value
); // End of function gettrans

 

 

Re: Normal Distributions and Transformations

Hi Jim,

I have the same issue. I have modelled some Johnson Si transformed data, and got a predicted model. I tried to use the inverse function to transform the predicted data back, but it's not working. Would you be able to provide more guidance on writing a script to do this? I keep coming across these types of distributions when modeling responses from DOE experiments, so it would be really useful to know how to transfer the data back.
Many thanks!

Christel

Re: Normal Distributions and Transformations

I think I missed the point of your question.  If you want to calculate "bands" based on probability then the location of these bands will differ according to the type of distribution you have.  Your numbers 68-95-99.7 are not standard deviations, but are probabilities associated with "bands" based on distances of 1,2,3 standard deviations from the mean based on a normal distribution.  If you don't have a normal distribution, the problem is not with the calculation of the standard deviation, but the conversion to probabilities.  If you want to have +/- 3 standard deviation bands then you are assuming the distribution is normal, or at least symmetric.  Depending on what you want to do, you can either calculate assymetric bands (JMP has probability distributions not only for the normal distributions, but for all distributions), or you have to perform a transformation to normalise the data (and then back-transformations whenever you want to convert back to natural metrics).  My preference would be to use asymetric bands and use the JOHNSON SU function to calculate them.

online help

Re: Normal Distributions and TransformationsThanks for you reply Jim! I will give this a shot.Re: Normal Distributions and TransformationsHere is what I do. To set my limits on my original data, based upon the transformed data values, I take the std from the transformed data, calculate what the values above and below the mean are for 1, 2, 3, etc. stds, and then reverse the transformation back to the original data. In some cases, such as the Johnson SU, there isn't an easy way to transform the values back, What I do then, is to run a little script that passes a value through the original transformation, checks the value of the targeted std, then iterates the value until there is a match. Then you have found the value in the original data that when transformed, results in the transformed values targeted value. Remember, when you do this, the distances above and below the mean in your original data will not be the same.Re: Normal Distributions and Transformations

Oh, I thought it did matter for standard deviation, though.  For example, the 68-95-99.7 (three standard deviations) rule is used to to find the values within a band around the mean in a normal distribution.  However, if my data is not normal, it might not make sense to use this.  For example, if my on resistance of my transistor is not normal, and I want to see what the value is at 3 standard deviations from the mean, I might have a negative value or a very low value that actually doesn't make any sense.

 

Sorry if I am being confusing or misunderstanding something, I am just starting to get back into learning statistics again since university!

Re: Normal Distributions and Transformations

Thank you, I see how it did that.  Now that I see that the data is normal, how can I use this to find the standard deviation?  It says in the summary statistics a value that makes sense based on the transformation, but I would like to know what the standard deviation is for the original data.  Perhaps I don't understand the purpose of transforming data.

 

Re: Normal Distributions and Transformations

Natalie

The formula for variance and standard deviation doesn't make any assumption about the shape of the distribution.  It's just algebra (in the same way that the calculation of an average value doesn't make any assumptions about the type of distribution).

Re: Normal Distributions and Transformations

Natalie,

You should be able to simply save the transform to a new column, and then run the distribution on that column.

 

Jim

Re: Normal Distributions and Transformations

hello David and everyone ,

 

I'm new to JMP and I'm wondering about data transformation.
what are the advantages/disadvantages if we work with data fitting compared to data transformation to obtain a normal distribution?

 

Also after the transformation if I don't have a good p-Value should I consider my transformation?

 

thank you

Re: Normal Distributions and Transformations

Kowa, first welcome to the community.  Your query cannot be answered sufficiently in this forum.  I suggest you start here:

 

https://www.ime.usp.br/~abe/lista/pdfQWaCMboK68.pdf

 

There are 2 primary reasons to do transformation:

1. Meet the quantitative assumptions of normally and independently distributed residuals with a mean of 0 and a consent variance (NID(0, variance).  If these assumptions are not met, then you should question the proposed model.

2. What Dr. Box told me years ago, the only reason to transform, is to simplify the model...in essence make the model more useable.