Stating the obvious: Not all of us can work in R&D. But we can still learn some of the lingo. Here’s a phrase: JSL unit testing framework. What is it? How did it come to be? And why is it important?
Let’s demystify the JSL unit testing framework.
Thank you to Nicole Jones (@Nicole_Jones), Sr. Manager of Advanced Analytics Testing, for this explanation:
“The term unit refers to the smallest testable component of a software product and the phrase unit testing is the process of validating to ensure that the units operate as expected. The phrase unit testing framework has come to refer to the mechanism used to facilitate automated unit testing.”
Principal Research Statistician Joseph Morgan (@joseph_morgan) introduced these terms to JMP when he started working as a Statistical Tester in 2005. (He now works in the DOE group.) Joseph’s background is in software engineering and computer science, as well as discrete math and statistics.
“He conceived of the whole concept, a central test framework,” says Sr. Manager of Test Engineering Audrey Shull (@Audrey_Shull). “He has lots of big ideas, and this was one of them."
“Once he became familiar with JMP and JSL, he was able to develop the JSL unit testing framework, keeping in mind the statistical nature of the software,” says Nicole.
Joseph Morgan explains (4:38)
‘Drinking our own champagne’
A unit testing framework can be written in other languages, but what better tool than our own JSL?
With the JSL unit testing framework “we are using JMP to test JMP,” explains Audrey. In other words, we are “drinking our own champagne.” And that’s not possible with all tools.
"We can use our software to test our software because we have this versatile JSL language,” Audrey continues. “We stretch the limits of what JSL can do to support the framework. That's a pretty special thing. And the framework itself serves as a huge regression test for JSL backward compatibility.
"If the development staff changes how scripting works, the test group finds it first because we're the first customer of the JSL language.
“All companies try to use automation, but often they use other third-party tools to do so. They won't always have the luxury to use their own software."
An extendable framework
A screenshot of the original dialog.A screenshot of the original dialog.The JSL unit testing framework is used on all JMP platforms. For instance, it can be used to test the drag and drop functionality in Graph Builder. Or the red triangle menus. Or any unit of code in the Distribution platform.
“Not only do we get to use it to test JSL itself, but we get to use the framework to test all the other platforms,” says Audrey.
Nicole elaborates:
“At the time Joseph was developing the framework, his primary focus was to find a way to create a regression test suite for the statistical platforms in JMP. Yet he was able to create a framework that was extendable – the regression test suite includes stat platforms as well as graphing platforms, it works on Macintosh as well as Windows, it tests data table commands as well as report structures.”
Working with ‘a safety net’
The framework can be run manually – testers and developers can launch, point and click, and run. But automation is key to ongoing improvement.
The framework is automated through Perforce (a repository of work), where the developers’ work lives, where code is implemented. JMP testers write test cases and upload them to a dedicated section in Perforce.
Screenshot of the testing dialog as it looks today.Screenshot of the testing dialog as it looks today.
A nightly automated process, run by Principal Systems Developer Jim Borek, grabs all tests uploaded that day, grabs all code pushed that day, runs the test cases on the updated code and notifies the team when anything fails. Jim is central to making this process automated and maintaining the machines.
Audrey described the enormity of the testing that takes place: 2.4 million tests are executed each night. And that’s happening on every operating system (e.g., Windows, Catalina, BigSur) for both JMP 17 and JMP 16.2. The process to run the tests on a single machine takes 6-10 hours, depending on how fast the machines go.
“That's the regression test suite – without this framework we'd have to test these things every single day to see if they break or not,” says Audrey. “It can't be manual. The suite of tests is automated so that the system alerts us that failure happened.
"We sometimes like to call it a safety net. I know it worked last time I tested it, I wrote a test case, and if it breaks the system lets me know. I can move forward to other things. We're trying to test what's new and coming and being added to the software.”
The JSL Test Framework Workflow
Finding an Apple bug
Joseph shares an early anecdote that cemented the value of the unit testing framework.
“Shortly after the initial implementation of the JSL unit testing framework was up and running, its value became dramatically apparent in a rather unusual way. Interestingly, it wasn’t a regression of JMP that it detected but instead, it was key in isolating a rather serious bug in MacOS.
“In early 2006, Apple started transitioning the Macintosh line from PowerPC CPU’s to Intel CPU’s and, in order to aid this transition, it bundled an emulator with MacOS known as Rosetta. This emulator allowed PowerPC applications to run on Intel Macs and so was viewed as a key piece of software by Apple.
“As it turns out, there was a bug in the Rosetta emulator that led to several unusual anomalies in JMP’s behavior. With the unit testing framework, I was able to isolate the problem to a bug in the implementation of the square root function in Rosetta (see Defect S0351925).
“Michael Hecht and I had several conference calls with Apple engineers to explain the problem, but the engineers were adamant that it was a JMP bug. It wasn’t until Michael suggested that I write a very simple C++ app that essentially mimicked one of the unit tests that showcased the bug that Apple was convinced that the issue was a Rosetta bug. A year later, in March 2007, Apple released a fix for the Rosetta bug in Mac OS X 10.4.8.”
Who benefits?
“[The framework] was developed with testers in mind but has proven to be useful to developers as well,” says Nicole. “I appreciate Joseph’s ability to create a framework that can address the needs of so many of us.”
Testers located in Cary and Beijing divvy up functions and monitor certain areas of JMP. But no matter what part of JMP they are responsible for testing, everyone is essentially testing the scripting language too.
Audrey says, “The more we use JMP and JSL we think of 'nice-to-haves.' Even though we aren't the ones implementing things, we are sometimes the ones that will request ‘can you please add this or that?’”
And the constant improvement, of course, benefits customers.
“Our power JSL users may want to do some of the things we're doing. It prompts us to think how we can script in a more robust way.”
Lots of bells and whistles that have been added were crowdsourced from development, testing and the release engineering team.
Kudos to key players who have added functionality over the years:
Joseph Morgan @joseph_morgan
Melanie Drake @Melanie_J_Drake
Jim Borek
Xan Gregg @XanGregg
Michael Hecht @hecht_jmp
Justin Chilton @Justin_Chilton
Evan McCorkle @EvanMcCorkle
Nick Holmes @nickholmes13
This is just one of many examples of the JMP division’s ingenuity, collaboration and commitment to continuous improvement. We hope to highlight more stories like this one. Reach out to Jessica Marquardt with ideas.
Want more information?
JMP Testing Dashboard
This China R&D Newsletter article by Test Engineering Manager Nala Sun (@Nala) provided inspiration for this post: Automation Framework of JSL Unit Tests
Here’s how JSL unit test results are output, processed and utilized for the nightly unit test runs: JSL Framework Data Pipeline
An article by Joseph Morgan: JSL Unit: Automated JSL Testing
Orso, Alessandro, and Gregg Rothermel. “Software testing: a research travelogue (2000–2014).” Future of Software Engineering Proceedings. 2014. 117-132.
Alex Orso's professional page: https://www.cc.gatech.edu/home/orso/
And thanks to @shannon_conners for providing background and guidance.
