Validation d'un software de lecture par rapport à un l'actuel système

Question

Bonjour,

* dans le cadre de la validation d'un logiciel de mesure de diamètres d'inhibitions nous avons fait des séries de lectures de plusieurs analyses, une avec notre logiciel actuel (ExploraNova) et une avec le nouveau logiciel CirclesMeasure) et avons obtenues des titres en % pour les 2 logiciels

Exemples :

Explora Nova

CirclesMeasure

M11 / Bacillus pumilus

GZ23AA2796-1,1

97,0

%

97,2

%

GZ23AA2796-1,2

96,7

%

95,9

%

GZ23AA2821-1,1

95,5

%

95,0

%

GZ23AA2821-1,2

94,5

%

93,6

%

GZ23AA2829-1,1

97,4

%

98,2

%

GZ23AA2829-1,2

97,1

%

96,2

%

Selon vous quel serait le test statistique le plus pertinent pour pouvoir dire que le nouveau logiciel donne des résultats conformes par rapport à l’actuel logiciel et qu’il peut donc être utilisé.

Avant pour qualifier un logiciel on s’accordait simplement +/- 3% entre les 2 mais j’aurai aimé savoir si un test statistique pouvait répondre à cette question et ainsi présenter en Assurance Qualité lors d’audit un dossier plus complet et abouti.

Merci pour votre aide.

Cordialement

Alain PONTONNIER

Victor_G · Answer

Hello @Alain04,

Welcome in the Community !

I completely agree with @P_Bartell about the size and representativeness of your sample. I suppose the dataset displayed here is just a smaller version of a bigger one, but you should take care about the data collected (and the generation process) to make sure your comparison is reliable and "fair". Start first with visualization before doing tests, the visualization (using box-plots for example) might be sufficient for your needs.

Looking at your topics, there might be several options depending on your objectives :

* As @P_Bartell suggested, you could realize an equivalence test, to analyze if the mean difference stays in a practical equivalence interval or if the two measuring systems are not comparable. However, as the Equivalence test is based on t-tests, there are some assumptions to verify (https://www.jmp.com/en_ch/statistics-knowledge-portal/t-test.html) :* Continuous data
* The sample data have been randomly sampled from a population (so it goes back to the point of the representativeness of your sample).
* There is homogeneity of variance (the variability of the data in each group is similar).
* The distribution is approximately normal (which might not be the case with percentages...).
As your dataset shown here is too small to verify some of the assumptions, I can't help you on this point.
However, if you want to proceed with this analysis, create a formula column with the difference between the two measures, use the Distribution platform to display the distribution of the Difference column, and from here, you can click on the red triangle and launch an equivalence test with the practical difference of 3 considered as equivalence (and hypothesized mean = 0, confidence level = 0,975 if you want the combined result of the two tests to be displayed at confidence level 0,95). Based on your dataset, here is what you could expect :

Remember that this equivalence test only test mean difference, so it won't provide information about a difference of variance between the two measuring systems (which might be very informative to highlight a gain in precision for example).

* Using your datatable in a stacked format, you could have a different analysis option using the platform Fit Y by X. Using your process as X and measure as Y, you can test difference in means (with Student t-test or non-parametric test, depending on the distribution of your data) as well as equality of variance ("unequal variances") between the two measuring processes using options from the red triangle. Here are the type of informations you can collect :
Tests for equality of variance (with Welsh test for difference in means):
T-test for difference in means :

Also the visualization displayed in this platform might be sufficient to understand if your results are comparable, in terms of means and variance :

* You could also use the Measurement Systems Analysis platform to compare the two processes and see how consistents are the results. The visualizations available in this platform are quite helpful to compare the two processes :

Using the Gauge R&R evaluation, you can see (with the small dataset provided) that you may have more variance due to the repeatability than due to a change of measuring process :

These are some options available to analyze your data, but there might be other additional ones. Again, it all depends on your objectives and the data collected.

I join the dataset used to test the three options with the scripts saved so that you can evaluate the different options proposed.

As @P_Bartell, comparing the two processes at a specific time might be informative, but you have to make sure the measuring process is stable and controlled in time.

I hope this response will help you,

Test_JMP-Community.jmp

P_Bartell · Answer

I tried editing my initial reply but had some user interface issues... so here some additional thoughts. I'd be very careful to just use one set of data gathered in one bucket of time for work like this. How stable are both measurement systems? How many 'noises' have they been exposed to such a people, materials, environment, equipment, suppliers, and on and on. So some type of process stability work over time should also be included. Control charts are your friend.

Alain04 · Answer

Hi,

Thank you very much
being more technical than biostatistician I discover JMP.
I will try this solution