Custom activation function for Neural platform

Question

The Neural Network platform is very handy but the activation function options could be expanded.  It should be very easy to add activation functions in concept -- but I'd like to know if it is possible to do it currently with JSL or other method?  At it's core I would think it amounts to providing a function and it's derivative.  Maybe an initialization strategy as well (but not so important with shallow networks like in JMP).

Dan_Obermiller · Answer

Be sure to use the "JMP Wish List" link above. That is where you place your suggestions.

jwalk · Answer

I would also like to have more activation functions to choose from and/or the option of constructing a custom activation function.

johnpjust · Answer

Yes I have JMP Pro and I've worked with the Nueral platform extensively. Mostly trying to use for regression and I see potential for sure -- also has a very good interface compared to using Python. However...There are only three kernel function options and the linear and Gaussian are not really useful IMO - the Gaussian is taken from SVMs I'm sure and does not seem to offer any advantage over tanh in my experience and the identity function doesn't distort higher order space and is not different than linear regression. The tanh function is good but I think it was intended for classification originally in ANNs. I can see the hard limits/saturations show up in regression actual by predicted plots and I also see undesirable boundary-like conditions in real-time prediction data where the inputs are slowly evolving over time - again an artifact of the tanh function. I would like to try an arcsinh (inverse hyperbolic sin) to alleviate this issue. I hypothesize that it's properties are more desirable for regression and may overcome the shortfalls of a tanh function for regression applications.