sequential testing

Question

Is anyone familiar with sequential testing?

Imagine I am running a reliability test over temperature.  After completing testing of parts at 3 different temperatures, I analyze the data.  My goal is to have a LCB on the MTTF at use conditions that is above some standard (e.g. 1E5 hr.s at 100C).   Unfortunately, after analyzing my data I find that the CB's are too wide and the LCB is too low and below my goal.  I may be tempted to put more parts on test, or test under more temperatures.  If after running more tests I calculate a new LCB and it meets my goal, do I have to take into account that I have done this test sequentially?  Is my significance level still the same for my LCB as it was initially, or do I have to change it since I have added more test levels?

Does JMP take sequential testing like this into account?  Has anyone heard of an add-in that does?

SamGardner · Answer

The testing into compliance issue was a problem for the pharmaceutical industry several decades ago.  What some drug manufacturers would do when an out-of-specification result was generated was to test/measure again, and continue to do so until either the overall average was within specification, or even worse, throw away the old data after they got an in-specification result.   That is no longer allowed, and the FDA created a specific guidance for that matter.  What most companies now do is to have a specific procedure for dealing with OOS results, that included not throwing away previous data unless a specific root cause for why the measurement was erroneous was found, having specific rules for measurement outlier removal, and limited retesting and overall averaging of all the results at the end.

In your case, if the matter is not an issue of legal or regulatory compliance, I'd recommend:

* a well thought out process for how to design and run a study so that it's goals are met* Determining the appropriate number of items to test so that you get a desired accuracy of the final result (i.e. a desired confidence interval width).  I'm not sure how easy that is to do for accelerated time-to-failure studies, though.  
* pre-specified rules/procedures or protocol for dealing with the situation you describe (i.e. the MTTF met its goal, but the confidence interval is too wide)

charlie_whitman · Answer

Sam, thanks for responding.  I think it can easily devolve into a testing into compliance.  After meeting the goal, management might say, "We met the goal!  Stop testing!"

What is the proper way to handle that situation?

SamGardner · Answer

@charlie_whitman interesting question.  For the situation you describe, I don't think you need to do any adjustment, as the goals is to be able to gather enough information to be able to estimate the parameter of interest with a desired uncertainty for that estimate.

But some might view this as "testing into compliance".  Doesn't sound like that is the case you are describing, though.