FDE Question - Why load target before preprocessing?Hi all,
a customer is really happy using FDE, but this one thing bothers them. Why can't they apply the preprocessing steps for the target function? Or, why is it necessary to load the target first (which means the target is not cleaned)?
The reason behind is, the target is not synthetic but real product data, and not preprocessing it makes the prediction break/unrealistic.
The workaround is to preprocess and save, and then relaunch and model the functional data. But it's extra steps making it more complex and they can't track back the preprocessing steps in the final FDE report.
@RyanParker maybe? 
Thanks
Florian
Re: FDE Question - Why load target before preprocessing?Hi @Florian_Vogt , this is probably a bug with specific pre-processing steps. Which pre-processing steps are they having an issue with? For example, you should be able to load target then standardize to have the training + target function standardized.
We need to load the target first so that it can be treated as validation type data in pre-processing steps (so only training data is used then applied to validation type data, for example). In this case, the bug(s) are limiting their ability to have this happen.
The workaround would be to apply all pre-processing steps, call Save Data from the top level RTM, then re-load FDE with all of pre-processed data and then load target. But it would be helpful to know which items in particular he's having trouble with.
Re: FDE Question - Why load target before preprocessing?@RyanParker any filter X or Align option will cause this issue, and I would not wonder any Spectral adjustments as well. Overall as soon you do not use all of the data or transform other than standardize, I guess.
For example, you should be able to load target then standardize to have the training + target function standardized.
This sounds as if it is meant to be that the preprocessing steps are applied to the target function as well. However, I can imagine that one wants to align max having the target function included, cut-off the non-important regions based on/compared to the target.
For having the target function just as a visual control an alternative could be to have a shadowed target function displayed, so you see right away if the preprocessing steps make sense. However, my workflow would rather be to have the preprocessing steps like align adjusted to the reference (aka target).
Re: FDE Question - Why load target before preprocessing?Thank you @martindemel , this is helpful! Outside of target functions not being pre-processed at all, could you provide a list of pre-processing steps where you would like to see some sort of target function reference options? This would be new feature type of work, and I want to make sure we capture it properly.
For example, which Align operation(s) specifically make sense to use the target as a reference? And how might you see that work on the user end?
Re: FDE Question - Why load target before preprocessing?thanks for the reply @RyanParker and @martindemel .
Without having tested every preprocessing option, it seems to be by definition that target cannot be loaded once preprocessing was done:
(my example)
The problem this causes with the customer is, the target is "real" sample data as well:
(customer example)
And if they don't preprocess the target as well, the FDOE profiler button "Optimize Target" is useless:
(customer example)
So, it seems for them it is not preprocessing once target is loaded. These are the steps they did in the example above:
(customer example)
Yes their current workaround is to preprocess all, save data, relaunch FDE and the load target and do the FDOE analysis.
This is what the pain is:
- extra steps increase the hurdle to use it
- they create a new datatable (while the inital one has already some analysis etc.)
- they cannot capture the preprocessing steps done for a given FDOE analysis (reproducibility and documentation).
Hope this makes more sense
Re: FDE Question - Why load target before preprocessing?Edit: it will likely be the "Remove Selected", because the target function disappears once loaded and thus points cannot be selected?
Re: FDE Question - Why load target before preprocessing?Ok yes remove selected does not appear to be properly respecting the target function! Thank you for sending this, we will make sure that gets fixed.
Re: FDE Question - Why load target before preprocessing?I try to be as thorough as possible:
1. If you have a major peak of interest, you might want to use "align max"/"align min". E.g. the major peak might move due to time shifts as the process is slower/faster than the one of the target. Similarly DTW with the option to select the target function would be helpful.
2. Remove selected is for sure of interest
Not sure if transformations will be already applied automatically (I guess so). Same for SVN, ... I guess this is nothing I would want the target function to be transformed as well.
Having the target function overlaid might be useful if:
1. For spectrals, you might want to overlay the target function to see if the baseline correction needs to be done or if the selected baseline correction is appropriate enough
2. For selecting points you do not want to have in the data (remove selected, see 2. above)
