425286_en-US

liliconcom

Importing .ch file

I have a customer interested in analyzing gas chromatograph data; the default data format is a .ch file. Anyone have experience bringing this data format directly into JMP? Example data file is here.

Re: Importing .ch file

@Bill_Worley , correct. The customer is analyzing  downhole fluids for the purpose of understanding  corrosion of oil and gas components. They are trying to understand the relationship between the frac fluid mixture prior to injection, the ability for the fluid to induce production of hydrocarbons from a reservoir, and most importantly to their task, how in-situ fluids alter the chemistry of the mixture as quantified through the analysis of recovered fluid with in an effort to predict corrosion of downhole components with time. It is a major challenge in oil and gas in part because the in-situ fluids vary from well to well and geologic basin to basin--What was true 'here' may not be true 'over there.' This is a multi-million dollar problem. The challenge for us (JMP) is they are as tight-lipped as it comes because this is all very valuable IP.

My best guess based on the limited information they are giving me is: Practically, they are cooking the injected fluid to quantify presence and concentration of hydrocarbons that would either be produced or had been incorporated into the fluid pre injection, as a producing agent of the fluid, in addition to quantifying the presence of some sort of corrosion signal.

Re: Importing .ch file

Hi Phil,

I was playing around with this data set early on and I was under the impression this was some sort of "test" mixture of short and long chain hydrocarbons.  That guess is based on  @Peter_Polito  mentioning the company deals in  oil, water and frac fluids.  

Bill

Re: Importing .ch file

That is a very messy chromatogram. About as challenging as it gets. But it depends what the customer is trying to find out - it would be interesting to know.

Re: Importing .ch file

I was able to download and install OpenChrom, import the .ch file, export it as a .xy file, and then import it into JMP.  The process of figuring out the install and how to import with OpenChrom wasn't as straightforward as one would like, but once you know how to do it is easy.

As the name says, OpenChrom is open source. We could study how it's .ch file importer works. There were also references to Matlab .m files that are also out there. The source code for these may point to a location where the file layout is described and we could put together an importer add-in if this format is common (as I think it is). Some other examples with more than one chromatogram packaged together somehow would be useful, since for JMP the -gram is only a single observation.

I did some basic testing with the data once I loaded it into FDE. P-Splines with 200 knots are pretty inadequate, but I wasn't thrilled with the Wavelet model fit in JMP 17 EA3 either. 

Re: Importing .ch file

I do like the line of the Open Babel link that @briancorcoran shared:

"Chemists are a very imaginative group. They keep thinking of new file formats."

I think it is more about the instrument manufacturers than the chemists that are imaginative. By having their own file format, they too can keep their product "sticky" with their customers.

Anybody know why there are so many different file formats other than instrument manufactures want to keep customers and competitors out?

Doesn't hurt to put this on the wish list, external and internal.

Re: Importing .ch file

I appreciate your candor @briancorcoran and I agree we can't do it all. 

Re: Importing .ch file

If you look at a tool like Open Babel, which supports importing 84 chemical formats, you can see that it becomes impractical for JMP to do all of these (http://openbabel.org/docs/2.3.0/FileFormats/Overview.html). And, BTW, Open Babel does not even support .ch files.  There is a tool out there that does, https://lablicate.com/platform/openchrom, and that is really experimental.  

Much as I don't like suggesting alternative products and tools, there is just no way for us to address all of the scientific tool formats out there in a meaningful way without hiring lots of people to do that.  Right now, that is not an option.  So, you might want to suggest something like Lablicate. 

Almost all forums end up advising users to dump out .CSV files in the end.  I know your customer doesn't like this option, but it is often a requirement to use the data in more mainstream tools like JMP.

 

Brian Corcoran

Re: Importing .ch file

@JeremyAshJMP the only information the customer would offer up was that they would prefer directly loading .ch files. Given the type of work these folks do (analyzing produced water, oil, and frac fluids) they are probably a high volume operation and they would like to not have to take the iterative step of exporting every file to a .csv. @Phil_Kay  and @ian_jmp I don't know explicitly why they want to open .ch files directly into JMP rather than a post processed data file (is it possible to have a post-processed .ch file? I don't know) but I've always felt that the more we can get our customers working in the JMP environment they more sticky JMP will be. I would not be surprised if they use a proprietary method to process their raw data anyway, so the post-processing tools in ChemStation may be moot. Regardless, I will continue trying to pump them for information. 

Regardless, they are trying to solve a problem that plagues every production company  and the solution they seek will vary by geography and geology so this problem and the process they are using to solve for it is reproducible across nearly all companies in the industry.   

Any savvy scripters want to collaborate on an Add-in?  I'll get the first drink/bite/coffee the next time we're all together.

Re: Importing .ch file

Hi Peter.  Is it not possible to output the chromatogram as csv? This is possible if they are using Agilent's OpenLab software: https://community.agilent.com/technical/software/m/videos/722?CommentId=c1e4cb78-1831-43c9-8cb5-84e28934148b

Re: Importing .ch file

Regarding "JMP does not have great tools for the kind of analysis that you would want to do with raw chromatograms" then it's worth saying there have been discussions to perhaps fix this. So that's even more reason to know more about the use cases involved.

Re: Importing .ch file

It looks like .ch is a data format from Chemstation - the software provided with HP/Agilent chromatography hardware. Chemstation is very well known in the world of chromatography. I would be interested in why they want to bring the chromatography data directly into JMP. As it stands, JMP does not have great tools for the kind of analysis that you would want to do with raw chromatograms. People tend to use Chemstation to do the raw data processing (baseline subtraction, peak picking, peak integration), which would be difficult in JMP.

Re: Importing .ch file

Might be worth taking a look here. Also worth drawing this to the attention of Product Management since it might be within the scope of our 'data connector' efforts. Not sure how much extra business we would drive if we could actually do this, but it looks like an add-in might be possible for someone with the time and inclination.

Re: Importing .ch file

Looks like it's binary. If you have the data dictionary, it could be scripted. That said, it wouldn't be unusual if quite a bit of flexibility was built into the data dictionary and scripting it would be a chore. A good example of this is with LabView and STDF files. I know there has been talk about adding connectors to JMP so more binary files could be opened natively. Could this be a good candidate?