439254_en-US

liliconcom

Shelf life prediction: Result compare Minitab vs JMP

Dear Colleagues,  

Received customer comcern from a big pharma company regarding the "Shelf life prediction",Result compare Minitab vs JMP using the same set of data

Concern 1:  

after we clarify with customer, one thing they concern about the graph visual display in "Stability Test": Minitab graph shows only one side, while, JMP shows both sided whith no-options to display only one side (even the result is for one side). 

This could be a quite mis-leading interpreation (for their case and in general), especially a big concern for those scientist who need submit approval request 

-> Anyone have different idea on this ?

Concern 2:

Using the same set of data, MiniTab give a lower shelf time as compared to JMP Stability test platform ( same result from Fit Model, inverse prediction), for their case they observe 4 months difference (sorry, they could not share their data with us) 

Lower shelf time means safer for them, however 4 months is a big difference, which software should they trust now become a headache problem 

-> Anyone have done some comparision using both software ? welcome experience sharing on this, Thank you for your valuable input 

@Byron_JMP @Mark_Bailey 

 

Pin 

 

Re: Shelf life prediction: Result compare Minitab vs JMP

Thank you, this is something that Mia, Sam and I can discuss with the R&D team in the context of JMP 18 planning.

Re: Shelf life prediction: Result compare Minitab vs JMP

Sorry, the attachment did not go through. Try again...

Re: Shelf life prediction: Result compare Minitab vs JMP

Hi @peng_liu possible for you to re-share the updated ICH_STAB_JMP data table with the second script that you mentioned? @PIN and I would like to use that to continue the discussion with the customer and provide him with all these good findings. Much appreciate and thanks!

Thanks, @Chandramouli_R as well for the quick validation of the case in Minitab. Very helpful. Agreed that solution for both ICH Q1E, ICH Q1D  will be useful to expand adoption in Pharma.

 

Re: Shelf life prediction: Result compare Minitab vs JMP

@PIN you have mooted an interesting discussion - as stability studies is a vast area of application in pharma. 

With the regulators getting more aggressive on stability data the anxiety of the user is only palpable - a review of FDA's 483 warning letters issued to the industry endorses this fact. There is always a risk of being audited for validity, falsification of data or data integrity and stresses the entire development value chain. 

ICH Q1E is endorsed by all the major regulators and therefore having all the functionalities notified in it in needs to be addressed and having a comprehensive solution would benefit our pharma and biotech users. Also please consider including a solution for the requirements of ICH Q1D as well. 

Re: Shelf life prediction: Result compare Minitab vs JMP

Much appreciate all for your valuable input  @peng_liu @Chandramouli_R @SamGardner @eclaassen @andreacoombs1 

Q1, seems no doubt, we should improve our grpahical display (  maybe + "note" idea from @andreacoombs1 ) to better present the result, reduce confusinon to our users, make their submission work easier 

Q2, according to Peng's investigation result, the Minitab vs JMP result difference comes from the older version of ICH guide that we are using.

more and more this kind of question will come as we are penetrating pharma industry aggressively, customer will adpate to the software based on the NEW ICH calculation guide rather than old (my point of view)   

what else should I do to make this happen if they are good for us JMP? 

Can we have a plan for it or maybe you already have it ? @Daniel_Valente 

Re: Shelf life prediction: Result compare Minitab vs JMP

Minitab's result is based upon assuming equal variance, which is pointing to the most recent ICH guideline.

JMP's result is based upon not-assuming equal variance, which is based upon an older ICH guideline backed up by the SAS macro known as STAB, published by a FDA researcher, who is probably among the authors of ICH guideline(s).

I attach an updated ICH_STAB_JMP data table with the second script in the table to reproduce Minitab's result. The script fits a "Different Intercept and Different Slope" model in Repeated Measure Degradation with equal variance. Run that script, and "Save Crossing Time" on the Inverse Prediction tab, and here is the result.

Here is Minitab result screenshot copy:

 

 

Re: Shelf life prediction: Result compare Minitab vs JMP

Thanks @Chandramouli_R , that is very help, and it confirms what thought about how JMP currently does the shelf-life estimation for the Different|Different model.  

Re: Shelf life prediction: Result compare Minitab vs JMP

Hi @SamGardner @PIN 

Please find the updated documentation of MTB ver. 21; Looks they have covered some ground from the version (ver. 18) you mentioned above.

Overview for Stability Study - Minitab

Example of Stability Study with a fixed batch factor - Minitab

Example of Stability Study with a random batch factor - Minitab

I ran a ICH Q1E stability studies dataset for a topical (proniosomal) gel which I did before; ran the same data in the default settings in both MTB (academic ver. 21.1) and JMP ( ver.16.1) 

Below are the results 

Appended are the MTB report and JMP table

There is a difference and I think can be attributed to the batch being a fixed (50% corresponds to mean response; and CI of 95% and alpha for pooling batches at 0.25) or a random factor ( 90%, CI of 90% and alpha for pooling batches at 0.25). I will try these and update here.

Re: Shelf life prediction: Result compare Minitab vs JMP

You can read the Minitab documentation on how they do stability analysis here:  https://support.minitab.com/en-us/minitab/18/help-and-how-to/modeling-statistics/regression/how-to/stability-study/before-you-start/overview/

Reading through that, it appears Minitab offers much more options on how to do the analysis, including the ability to use a mixed model to allow for random batch effects, and also on how to set the alpha level used to determine the appropriate pooling of batches and/or slopes, and the ability to base the shelf-life determination on a different percentile of the stability prediction model.  

It does appear that if in Minitab they use fixed batch effect and the default alpha and percentile, then the results should match.  However, in some situations JMP does not do the ICH Q1E shelf-life calculations exactly according to the guideline (I think), you can see the JIRA issue here:  https://rndjira.sas.com/browse/JMP-8832  

If we really want to diagnose the issues, we need to see the data, or at least a fully worked Minitab output along with the corresponding data, even if it is an example data set.  Minitab has an example data set https://support.minitab.com/en-us/minitab/18/ShelfLife.MTW , but we would need a copy of Minitab to open it.    

Re: Shelf life prediction: Result compare Minitab vs JMP

Based on the plots, it looks like in Minitab they've fit the line to the "worst performing batch" (presuming the worst performing is the highest in this case) and given the one sided confidence interval for that line, whereas in JMP a common slope/intercept was fit with a two-sided interval. The other batches have lower responses thus moving the line down and extending the shelf-life.

Edit: Sorry for jumping in to the SE discussions. But the shelf-life topic caught my eye. My PhD advisor used to work with a shelf-life workgroup trying to update the ICH standards; another student of his did her dissertation on it, so I had some exposure.

Re: Shelf life prediction: Result compare Minitab vs JMP

I agree with Concern 1.  If you have a one sided specification limit, it would be nice to have a note saying that the shaded interval on the graph is for a 90% confidence interval which gives a one-sided 95% confidence for expiry.  This should be added to the internal wish list.

 

In terms of Concern 2, I am not sure what the difference is in the way Minitab is calculating the expiry.  The graph contains an upper bound and percentile which is unusual.  A quick look at Minitab documentation does not show an example with an upper (or lower) bound and a percentile.  I would expect to see the line of fit and the confidence interval around that fit.  It is not clear what Minitab is calculating here.  One question you can ask is whether Minitab is taking a common slope and common intercept to calculate expiry.