Request for advice: Sampling validity and aggregation into distributions

Question

Hello,

In our lab, we measure particle sizes (specifically metal particles from single atoms up to ~20nm supported on carbon and oxide powders by STEM for anyone familliar) by aggregating data from various samples at different magnifications, similar to using Google Maps screenshots at different zoom levels to measure building sizes across the US. This approach has two main issues: the selection of sample locations and the zoom levels used.

Typically, we create an empirical distribution by aggrigating the measurements and assume the mean as the population mean. However, I'm concerned that our sampling may not be representative.

I've been using the one-way ANOVA platform in JMP to analyze particle size as a function of image number and zoom level. I want to know if i can validly aggregate the data across images within and between magnifications. High magnifications will have lots of small particles, but miss out on big ones, while low magnifications will have an opposite effect. I think the correct procedure has to do with sampling theory, but most of the sampling theory resources I can find are for social sciences, like for political polling data. However, on the other hand, this is very similar: sampling attributes over a "geographic" area.

So can I do what we've been doing and just throw them together and call it a day, or should I confirm we have representitive sampling from at least the different magnifications? If we do that, I've run some of the tests, like Tukey's that show magnifications have different means, but then I'm not sure if I need the non-paramtric versions.

How would you test the validity of our sampling and assumptions? Am I overthinking this, and should I just take the average as we've been doing? Or should I run a series of tests (e.g. Anova, anom, welch's test, tukey, etc), and which should they be? I'd appreciate any expert advice on this matter.

And sorry that this post is a tad scattershot. I see all of the options in front of me, I just have zero experience choosing between them, and this isn't something people in my field tend to care about.

Thanks!

ehchandlerjr · Answer

These are awesome recommendations. Thank you for putting so much time and thought into it. I'm going to sit down with some people and see how to work in some of what you've recommended.

Unfortunately, we've already gotten some images so we need to analyze those. I think your comment about "...combine[ing] all of the data into one distribution" is what we are limited to right now.

Do you know if JMP has a built in way to do a weighted distribution? Like weighting differently from different levels of a factor (in this case magnification)? I think the technical term is mixture distribution or something. I couldn't find anything, but I may have missed it.

Thanks!

statman · Answer

Let me first suggest a different thought process with an example.  Assume the changes in ambient conditions day-to-day are significant.  Which is the "better" sample size?  10000 samples taken from one day, or 5 samples, one sample taken per day for 5 days.  You see it is not the size of the sample that matters, it is what sources of variation are represented by how the sample was gotten.

Here are some thoughts on things you could try (again, I don't know your process well enough):

Take a sample from the "batch".  Split the sample and prepare each split separately (dissolve in solvent and dry). Process one split through the STEM measurement system (I imagine this is take an image of the split).  Repeat using the same split (so you now have 2 images of the same split). Measure the images at low res and collect all of the data (e.g., diameter, length, width, weight, etc.).  Repeat for high res.  Repeat this entire process for the second split.  This will allow you to estimate the following components of variation:

1. Measurement system precision repeatability (image-to-image within split confounded with within image)

2. Split-to-split (sample preparation process within sample)

3. Res-to Res variation

Or you could correlate the high res with the low res (treat as 2 different Y's) or combine all of the data into one distribution?

You can, of course, design multiple sampling plans each differing in complexity, resources need, which components of variation are confounded, which are estimable and which are not in the study (inference space).

statman · Answer

Unfortunately, I do not understand the specific situation you are in, (e.g., What is the product you are measuring?  Is this a batch process?  DOE on what?) If your main goal is to correlate your "typical" measurement system with STEM, then I'm not sure how "representative" the sample needs to be.  Perhaps use random samples and measure those with both measurement systems?  Or do you want nato ensure the extreme particle sizes measures from the 2 measurement systems correlate?

Here is some general guidance for obtaining a representative sample.  There are 2 different approaches to determine how to get a representative sample (of course, you need to consider representative of what?):

Enumerative:  This is an approach biased to the use of statistical method as a basis for obtaining a representative sample.  The sampling is not based on subject matter knowledge.  In general, guidance is to randomly sample (to reduce bias) and increase sample size.  IMHO, these methods are less effective/efficient at acquiring a representative sample and since the samples are randomly acquired, assigning sources of variation to these samples is challenging if not impossible (see Shewhart).

Analytical: With this approach, the sampling is done with SME hypothesis as a basis for determining how to sample.  Your scientific explanations as to why there would be variation and what factors might affect that variation are used as a basis to determine how to collect the sample and how to measure those samples.  For example, Given the hypothesis that agitation speed in the processing of the materials affects particle size variation, you will need to ensure that samples are gathered from multiple speeds during processing (this may be done via directed sampling based on knowledge of the process or by experimentation).  Another example, you hypothesize that part of sample preparation process (suspending the sample in a solvent) depends on the purity of the solvent so when the suspended sample dries, there are contaminants in the sample.  You will need to sample over varying solvent purities.  Again this may be done with directed sampling (based on knowledge of the inherent variation in solvent purities) or specifically manipulated in an experiment.

The more you understand the process, the better suited (efficiency and effectiveness) you are to develop a representative sample based on rational hypotheses.  The less you understand, the more likely an enumerative approach would be useful.