Scripting: Upward Compatibility in GeneralAlmost every release (since 7y) I get calls from customers telling their scripts do not work anymore in the new version. Do you get similar calls?
I wonder if we could have a better upwards compatibility check for custom scripts (not the built-in generated ones, where I expect we have this check). To be clear, I do not talk about backwards capability (aka my JMP 15 script does not run in JMP 13) as software evolves naturally. But as a consumer/user I would expect, that a script I write in JMP 14 will run in JMP 15 still.
I know Early adopter is one way to get to know this early and send the issues to the email list. BUT there is a big issue: The users usually do not have the time to do a compatibility check in the early adopter, especially not if there are 7-9 versions of it. This should be done on the software vendor side. That's also what customers most of the time complain about. "It is not their job" an d I must admit I partly agree with it (for most of the cases).
So, do you have similar complains from customers? What is our current process of checking upward compatibility? It would be nice to gather here some cases and suggestions what could be done. (And may also discuss this during a Kick-Off session with the corresponding developers - I remember the Q&A in the plenary where one could address those things, but the last one was I believe 3y ago - may one just for the SEs in 2020?).
Thanks,
Martin
PS: Reason why I make this that prominent, a manufacturing customer with >100 licenses is that upset since years that they think about moving from JMP to other solutions, even open source - and they are really JMP friendly for years).
Re: Scripting: Upward Compatibility in GeneralHi Martin,
FWIW, my experience is that lack of forwards compatibility is almost always related to pulling things from the report to manipulate them further. You say that it's reasonable to expect software to 'evolve naturally', but, like it or not, changes in the report structure are an almost unavoidable consequence of this desire. We should always be at pains to point this out to customers that do indeed make heavy use of scripting.
As you know, for software used in 'production', it's very common, even required, to have staging areas in which upgraded software is thoroughly tested before making it 'live'. This discipline could be seen to be at odds with the fact that JMP is 'just' desktop software, but (as your message illustrates) the risks are indeed real and we should help users to see JMP this way when their usage patterns demand it.
In short, this tension is a side-effect of the fact that JMP is so very functional. I think we can and should do a better job of alerting customers to the risks and consequences of not giving a significant JSL code base the support of good industrial software engineering practices. R&D 'could' provide methods to get all the required values in a report-independent way, but this is not on any roadmap that I'm aware of, and in any case would take away some of the flexibility that JSL programmers seem to value highly.
Like you, I'd be very interested to hear of other examples where we have got into trouble over this . . .
Cheers,
Ian
Re: Scripting: Upward Compatibility in GeneralHi, Martin!
TL;DR:
Point 1: Code breaks between versions are usually because people aren’t following JSL best practice and scripting defensively. Point 2: It’s always the customer’s responsibility to validate any software performance (homegrown or otherwise) between versions. They should treat it just like any other manufacturing tool – which is all vetted before being released to production.
The long version:
Just to riff on the comment @ian_jmp made – For the most part code breaks between versions are because of people not following (or keeping up with) JSL best practice. It’s a lot easier to break code that works but isn’t well thought out. Mark Bailey calls it “defensive scripting.” Ian’s example of people pulling from existing reports is a classic one, particularly if they are subscripting the entire display tree. The best practice for this is to use an outline box name as a local landmark and then subscript (because they don’t change as often).
I’ve also run into a couple of instances where people (myself included) build JSL that relies on a feature that gets an overhaul between versions. For me, the changes in Recode that came out in JMP 14 broke part of an add-in I was maintaining. In that case, I did something to just get the code working and I got burned for it. Another (customer) example is where someone was using non-ODBC compliant names in their database. When we enforced ODBC between versions the scripts broke. Here someone didn’t follow standards and they got burned for it. The take-away is that your customers are less likely to run into issues if they make code that follows our best-practices for writing JSL. If they do that, it’s pretty rare that JSL will break version to version. Most of the stuff I’ve written with a mindset to defensive scripting is pretty robust – but I still sanity check that everything is working. I usually wait until the last couple of early adopter releases to test (changes to JMP are usually pretty stable at that point).
Ian also made a really good point about moving between versions. In the US, we have certain industries that are heavily regulated (Pharma, Biotech, Finance, and some manufacturing with military or national security applications). These industries have validation requirements for any new software or new versions of software that require the user-base to document that they have checked the software works and gives consistent results between versions. My other customers know that they need to sanity check their code between versions. So I think it’s really the problem of maturity and discipline in the company culture at that customer (or at least with the people that generate the JSL there). The testing workflow Ian mentioned (staging, then release to production) is standard practice across every industry I’ve ever worked with (in the US, EU, or Asia) – they should know better than to assume.
So – summing up – if their JSL is mission-critical, they need to treat it like any other production software and validate the JMP version (with the mission-critical JSL) before releasing it to production. I’m betting your complaints come mostly from parts of the engineering community that doesn’t understand how that works. You could ask them if they would release a new piece of equipment to production before thoroughly testing that tool’s performance and matching to the other parts – they would answer “no.” At which point you could point out that software is a tool and should be treated the same.
Follow up Thought
Thinking about this more - if people follow JSL best practice (using functions and expressions, scoping, namespaces, heavily commenting their code, etc.) it's easier to debug when things do break (which they do). So it sounds like the customer in question has some ill-formed brittle JSL that's grown organically over time. It's probably horribly commented and the person that wrote it has moved on or has no idea how to fix it (I call these franken-scripts)... In that case, I'd recommend that they send someone that knows JSL to the SAS Ed JSL courses for a refresher on best practice and then set them loose to modernize the code in question and put some SOP's in place to add the JSL in question to their software validation testing plans.
Re: Scripting: Upward Compatibility in GeneralI agree with all of the comments so far. I see 'production' code all the time that is horrible. I can't believe it works in the current version, much less have a hope in the future. These users have not learned how to script JMP. That is different from general computer programming. It is not just syntax. They just hack. (I use the term 'hacker' with veneration. This activity is different.) They don't know what they are doing.
As for the comment about changes in the JMP UI that thwarts accessing the report layer, many users do not practice safe coding. There are ways to minimize or eliminate these problems, but their way produces brittle code.
Re: Scripting: Upward Compatibility in GeneralThanks so far for your comments. I agree with all of your points regarding testing before going into production and save/defensive coding (though users probably won't like it 
). This is definitively in the responsibility of the user and company using scripts in whatecver language.
However I want to advance the discussion also with a more critical point of view ( - as at the end, I hope we have at best a common message to customers and a more robust automation process to sell to our customers):
Coming from a software house which relies on the programming language the tool is build on, I am surprised how often I get to know JSL is breaking from version to version. In my past life I only got these issues with undocumented fuctions and features, not with settings and options like e.g. color themes and other things in the object trees or methods. Also all functions were documented to the full extent: what inputs and outputs are available, what is the description and definition of this option/input/output and so forth. The scripting guide and index are both not that kind. More a this is the basic call and all other possible options you have to find out yourself.
One could say all not documented part is subject to change - that would be the easy answer. But I believe it is not that easy. Most of the time we need to change at least a bit of the "one level below the built-in features". We do not provide a concrete documentation which states "that's what you can do" and "everything else is unsupported" (aka may works now but is subject to change).
A Programming language should have such a framework - or we say JSL is not such a programming language and be more careful with our current promotion of creating applications to reach out to more users as it's easy to build reproducible code but it might not be as reliable for future versions.
I'd be interesting to hear how others feel about this (I admit the more and with purpose critical point of view, again to foster a divers discussion about this topic), may also the one or other developer hwo do this on their day to day base.
Re: Scripting: Upward Compatibility in GeneralYour last point is the critical one – JSL is incredibly powerful, but at its heart it is not a “programming language” it’s a scripting or macro language. It has structures that resemble Object Oriented Programming, and it can be used in that fashion to some degree, but it it’s built on top of JMP and has always been intended to automate tasks in the GUI or extend the GUI. A lot of people forget that – which also means that they forget when things are changed in the product – things can break in JSL (particularly if they don’t code them well). We shouldn’t be talking about JSL the way we do C++, etc. Strictly speaking, even Python and R are more closely related programming languages than JSL. Maybe some people get confused because the line is blurred with Excel Macros and VBA. However – the point stands that JSL is a scripting language which is different than a programming language.
Best,
M
Re: Scripting: Upward Compatibility in GeneralHi Mike,
Agree, it's a scripting languague and so it should be treaded. I found this dicscussion valuable as it also highlights some additional aspects we always should guide people to think about when starting to automate or script their processes. Do we have - besides JMP JSL Training Course - other JMP related references for "defensive" scripting?
Most of the references found in the web are for other languagues and not JMP related (though you could use them too, of course). The Mastering JMP webcasts do not cover good programming/scripting style. May would be an interesting topic also for a discovery presentation?
Re: Scripting: Upward Compatibility in GeneralThat’s one area that I think we could improve in. There is a lot of information on the community, in the brains of staff, in books, and in the documentation, but there really isn’t a “best practice” document that I’m aware of. There should be. I think a lot of the consternation that you are experiencing in the field can be chalked up to this one point.
@ian_jmp – maybe this would be something the technical community could do as a sidebar during kick off.
Best,
M
Re: Scripting: Upward Compatibility in GeneralAs my reply hinted, I do think this is an area we could do better with, especially since we generally encourage building applications as a component of the 'expand' part of our 'land and expand strategy'.
FWIW, my personal view is that not every SE should be proficient in JSL unless their are so inclined. But I do believe that every SE should know what's possible, and along with this, what good practice might look like and the pitfalls of not adhering to this.
Unfortunately, though the 2020 kick-off is a little compressed, and the technical track is already organized. So we will find another way to handle this. More later.
Re: Scripting: Upward Compatibility in GeneralWe began adding explicit "best practice" suggestions in our scripting courses several editions ago. We continue to extend and refine these suggestions with each revision. We get positive feedback from students. Well, at least from the students who are not in a hurry to hack away.
