Background: Waver-level optics
Scenario: Imagine you have a topography (aka camera of a surface with a bunch of pixels). Now you do not have just one but an operator makes many of them. Now he wants to compare pixel by pixel the variation involved of each picture taken to the others. How robust is the measurement? To instrument and to operator.
This is a typical Gage R&R study I thought. They have 100 measurements. If you just would want to compare 100 images that would be easy, but they need to have a pixel by pixel or in other words, position by position over all images variation measure. and this is a daily task for them.
At all 11 Mio rows, 5-6 columns. one categorical column has about 10k categories which forces any Gage R&R to the ground.
Gage R&R is not meant for this kind of size. But what they want to do is or any alternative.
Goal is to understand repeatability and reproducability in a multidimensional scenario
Any ideas or recomendations on how to deal with this kind of scenario?
You might try hierarchical clustering - though it's going to be a little slow depending on the size of the image. A 1024 x 1024 image is 1M pixels - well outside what you'd normally want to throw at HC. I have done it (link) but it wasn't speedy. I also worked with some guys at a MEMS plant on something like this. They were using scanning acoustic microscopy to measure wafer bonding. We used Hierarchical to look at voids between the two wafers.
Just a thought,
M
