nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

FDE Question - Why load target before preprocessing?

Florian_Vogt

Hi all,

a customer is really happy using FDE, but this one thing bothers them. Why can't they apply the preprocessing steps for the target function? Or, why is it necessary to load the target first (which means the target is not cleaned)?

The reason behind is, the target is not synthetic but real product data, and not preprocessing it makes the prediction break/unrealistic.

The workaround is to preprocess and save, and then relaunch and model the functional data. But it's extra steps making it more complex and they can't track back the preprocessing steps in the final FDE report.

@RyanParker maybe?

Thanks

Florian

Accepted answers

RyanParker

Ok yes remove selected does not appear to be properly respecting the target function! Thank you for sending this, we will make sure that gets fixed.

All comments

RyanParker

Hi @Florian_Vogt , this is probably a bug with specific pre-processing steps. Which pre-processing steps are they having an issue with? For example, you should be able to load target then standardize to have the training + target function standardized.

We need to load the target first so that it can be treated as validation type data in pre-processing steps (so only training data is used then applied to validation type data, for example). In this case, the bug(s) are limiting their ability to have this happen.

The workaround would be to apply all pre-processing steps, call Save Data from the top level RTM, then re-load FDE with all of pre-processed data and then load target. But it would be helpful to know which items in particular he's having trouble with.

martindemel

@RyanParker any filter X or Align option will cause this issue, and I would not wonder any Spectral adjustments as well. Overall as soon you do not use all of the data or transform other than standardize, I guess.

For example, you should be able to load target then standardize to have the training + target function standardized.

This sounds as if it is meant to be that the preprocessing steps are applied to the target function as well. However, I can imagine that one wants to align max having the target function included, cut-off the non-important regions based on/compared to the target.

For having the target function just as a visual control an alternative could be to have a shadowed target function displayed, so you see right away if the preprocessing steps make sense. However, my workflow would rather be to have the preprocessing steps like align adjusted to the reference (aka target).

RyanParker

Thank you @martindemel , this is helpful! Outside of target functions not being pre-processed at all, could you provide a list of pre-processing steps where you would like to see some sort of target function reference options? This would be new feature type of work, and I want to make sure we capture it properly.

For example, which Align operation(s) specifically make sense to use the target as a reference? And how might you see that work on the user end?

Florian_Vogt

thanks for the reply @RyanParker and @martindemel .

Without having tested every preprocessing option, it seems to be by definition that target cannot be loaded once preprocessing was done:

(my example)

The problem this causes with the customer is, the target is "real" sample data as well:

(customer example)

And if they don't preprocess the target as well, the FDOE profiler button "Optimize Target" is useless:

(customer example)

So, it seems for them it is not preprocessing once target is loaded. These are the steps they did in the example above:

(customer example)

Yes their current workaround is to preprocess all, save data, relaunch FDE and the load target and do the FDOE analysis.

This is what the pain is:

- extra steps increase the hurdle to use it

- they create a new datatable (while the inital one has already some analysis etc.)

- they cannot capture the preprocessing steps done for a given FDOE analysis (reproducibility and documentation).

Hope this makes more sense

Florian_Vogt

Edit: it will likely be the "Remove Selected", because the target function disappears once loaded and thus points cannot be selected?

RyanParker

Ok yes remove selected does not appear to be properly respecting the target function! Thank you for sending this, we will make sure that gets fixed.

martindemel

I try to be as thorough as possible:

1. If you have a major peak of interest, you might want to use "align max"/"align min". E.g. the major peak might move due to time shifts as the process is slower/faster than the one of the target. Similarly DTW with the option to select the target function would be helpful.

2. Remove selected is for sure of interest

Not sure if transformations will be already applied automatically (I guess so). Same for SVN, ... I guess this is nothing I would want the target function to be transformed as well.

Having the target function overlaid might be useful if:

1. For spectrals, you might want to overlay the target function to see if the baseline correction needs to be done or if the selected baseline correction is appropriate enough

2. For selecting points you do not want to have in the data (remove selected, see 2. above)