COMPANY CONFIDENTIAL. DO NOT DISCLOSE.
Removing the risk from text data
Sandia National Laboratories marshals JMP Text Explorer to process and evaluate unstructured risk management data
_____________________________________________________________________________________________
Challenge: After eight years using a risk-and-issues management tool that captured records as freeform text, Sandia’s leadership realized that due to the cumbersome format of the data, little analysis had actually been done. The organization’s Contractor Assurance Group was tasked with developing a more effective tool.
Solution: Analysts relied heavily on JMP Text Explorer to process historical text data using latent semantic analysis with singular value decomposition.
Results: Within three months, the results of this text analysis provided a clear path forward to configure a new, more accurate risk management tool.
_____________________________________________________________________________________________
Few organizations approach data analytics with more rigor than Sandia National Laboratories. One of three research and development bodies for the U.S. National Nuclear Security Administration (NNSA), Sandia’s mandate is to maintain the reliability of nuclear weapons systems, develop nonproliferation technologies and explore methods for the disposal of nuclear waste.
With a high-stakes mission and budget to match, it’s no surprise that risk mitigation is a core tenet. To that end, Sandia developed a risk-and-issues management tool that, over the course of eight years, empowered users across the organization to create detailed records of arisings. The format of these records was unstructured text; a medium that, while comprehensive in theory, was leading to problems.
For example, not all content entered in in the tool’s comment box met the severity threshold to be classified as a risk or issue at all. Some fields were left blank. Spelling and grammatical errors interfered with a correct interpretation of users’ meaning. And there were just too many comments to distill into patterns or trends that could point at meaningful insight.
By 2019, it was clear that despite the richness of the data archive, little of its content was being analyzed – and even less was being acted on. So Sandia’s leadership announced an overhaul of the system.
This mammoth task was assigned to Sandia’s Contractor Assurance Group where compliance analyst Yvonne Petrova and computer scientist Scarlett Marklin took it on full-time. The stakes were high; as the existing tool no longer complied with a new organizational mandate, Sandia was at risk of losing project funding. “The Department of Energy introduced new data requirements,” Marklin explains. “If we didn’t revise the tool immediately, we’d end up losing budget.”
The trouble with text
The pair first set about understanding the flaws of the existing tool so as to identify a set of clarifications and guidelines that could be used to configure a new approach. “[We were seeing that] users might write five words, or they might write War and Peace [in the tool’s comment field]!” jokes Petrova. “They might refer to documents, with no indication of where those documents resided. It wasn’t good.”
“No one ever analyzed the text part of the data because it was so varied and messy,” Marklin adds. “People aren’t going to look through 100,000 rows of text in Excel. We were capturing the information, but we weren’t doing anything with it.” To uncover actionable information from this morass, they would need to find a software that could enable them to apply a set of systematic, analytical principles to unstructured data.
That’s when one of their colleagues showed them JMP. “I was like holy cow! I could do so much with it because of the visualization and interactivity,” Marklin recalls. “I hadn’t used JMP before, but I figured better now than never.” And so the pair began to pre-process the reams of text archived by the risk management tool, using JMP to help bring structure to the unstructured data.
“Pre-processing text is [usually] messy, ugly and difficult,” Marklin explains. “You have to know what you’re looking for early on. And without being able to see the data, it’s really, really hard to anticipate the questions that may come out of it. But with JMP, pre-processing was easy.”
That simplification is by design; the JMP Text Explorer platform allows users to combine similar words, recode misspecified terms, perform latent semantic analysis and use singular value decomposition (SVD) to group like documents into topics or clusters. In a massive and varied text-based data set, these tools were crucial. Says Marklin: “JMP has a lot of different tools built into the Text Explorer platform that allow you to go in depth with the data in ways that even R can’t.”
A JMP vs. R face-off is proof of concept
Both Petrova and Marklin are long-time users of open-source programming languages R and Python. Despite their proficiency with these platforms, however, the pair saw value in evaluating a new tool like JMP that could reduce analysis time through in-built functionalities. “JMP allowed us to learn more about the data much faster,” Marklin explains, citing the ability to sift through 8,000 rows of data to identify three key indicators that were critical for the team to incorporate into a new risk-and-issues tool. This discovery, which the pair made using JMP, would have been overlooked had they only used R. Now, she says, these indicators are having a big impact.
In fact, the two estimate a heavy emphasis on the use of regular expressions in JMP cut data pre-processing time by half – and improved the quality of results overall. By using regular expressions as part of sentiment analysis, they could see where no corrective actions were taken, where steps were taken but made no difference, and where actions drove improvements.
With input from subject matter experts, Petrova and Marklin were able to judge the accuracy of their categorizations, showing gaps and inconsistencies with data captured by the original tool. Furthermore, the results of their study – which start to finish, took less than three months – provided an essential blueprint for a better data capture system in the future. In the end, Sandia’s leadership not only incorporated all of Petrova and Marklin’s recommendations, they used the issues revealed by the team’s text analysis to configure a new system.
An added bonus, Marklin explains, was that the whole project became a proof of concept in evaluating the need for JMP licenses at Sandia, where many have traditionally used open-source alternatives. But they found that JMP reduced analysis time and added real value, particularly in its ability to quickly process unstructured data. That being said, Petrova explains, there will always be value in maintaining expertise in R and Python as a quality verification measure within the department. “When we're doing something in JMP, we also have a check. Can my colleague do this in R? Then let's make sure we get the same answer.”
In the meantime, though, the two will continue using JMP. Adds Petrova, “since we have JMP licenses, we use it for almost all of our projects. JMP is touching them all one way or another.”
‘A real-time, on-the-fly fluid continuum’
Beyond their role assessing internal systems, Petrova and Marklin, provide analytical support to colleagues at Sandia – some of whom are not as versed in statistical methods. But, says Marklin, “JMP lets us show people how consistent and valuable the analysis can be, and how they can visualize the results. When you see the data in JMP, it’s real-time, it’s on-the-fly, it’s a fluid continuum. And it’s super-efficient.” Plus, the visualizations in JMP, Petrova adds, are “really strong, and they make it easy for people to understand what they’re looking at.”
Petrova and Marklin themselves don’t have to look too hard to find reasons to deploy a statistical approach. In one case, Marklin explains that a colleague’s text-analytics project had run for three years without meaningful results. The team had reached the end of its funding but, she adds, “they gave it to us, and with JMP, in less than three months we had the results we needed.”
