Aggregating data rules or best practices?!Hi all,
a customer was about the correct or even allowed way to aggregate datapoints. They have two application, where this question came up:
1. Image Analysis for product quality analyses: They count the number of occasions per picture. A picture has only very few findings, so they want to analyse and control chart packages of pictures e.g. 50 pictures. They found that the number of pictures they aggregate has a significant impact on the results of e.g. control charts.
2. Process Data (Temperature Sensors etc.). They are recorded in seconds, which is too much data for analysis. So the question again, how much can these data points be aggregated (to minutes, hours, etc.)?
They of course try to decide in a way they don't loose information. But they wanted to know whether there is a systematic way to calculate (maybe) the max. aggregation? Or maybe best practices?
In my company we decided on this on a project to project base, so I am not aware of a systematic way or best practice.
Thankful for any help or suggestions!
Best
Florian
Re: Aggregating data rules or best practices?!This is more about what signals they are looking for in the data. Aggregation is fine, and valuable, depending on what they are trying see.
Example 1 has analogs in semiconductor (defects per wafer or defects per die) or automated optical inspection systems (film inspection for photographic films, etc.). If you're looking for reproducible signals that show up in the same regions or tend to form clusters, the aggregation is going to help you see them. If the data has a lot of noise and the signal is fairly weak, then aggregation might drown out the signal.
Example 2 has analogs with spectroscopy or other time series data. The biggest issue with aggregation here is that you'd typically do a rolling aggregation window which can smooth out smaller signals. Chunking time series data (going from second to minute or hour) does happen too, but has additional issues - like what happens when a signal shows up over two chunks (i.e. the signal starts is minute 1 and carries into minute 2 with a max somewhere in-between).
In either case, I'd suggest they consider what the smallest/weakest signal they'd like to see is and then aggregate to that point.
Best,
M
Re: Aggregating data rules or best practices?!Adding to @MikeD_Anderson's reply, this question is about rational sub-grouping. The fundamental question, along with frequency, is about the occurance of common causes and special causes. The sub-group size should be determined so that special causes are very unlikely to occur within a sub-group but very likely to occur between sub-groups.
Hope this guidance helps.
