Compiling large scoring algorithm generated in C

dmmike

I have a large machine learning data model in JMP.  I generated code in JavaScript and created a node.js app to score the data, but it takes almost 2 minutes just to score 50 records.  I thought I might try generating code in C and using that to score the data, as I figured it would be faster.  The code file that is generated is 67MB.  When I try to compile it (I'm using MS Visual Studio), the compiler keeps running out of heap space.  I've also tried compiling using Clang and had similar problems.

 

Does anybody have any experience compiling C applications using very large scoring code from JMP?  If so, what compiler/options did you use?

nascif_jmp

Hi @dmmike,

Two suggestions:
1) In JavaScript, take advantage of the parallel nature of scoring and the node.js utlilities to execute asynchronous code, like Promises. You might be able to achieve a higher throughput by scoring multiple rows in parallel.
2) in C, this MSDN article has a few compiler flag combinations that might help: https://msdn.microsoft.com/en-us/library/yz7kx3y2.aspx
3) how about a Python application?

I would be very interested in learning more about your issue and figure out if there are ways to generate code in the future that will be more resilient to this kind of problem.
If you can, please DM me with more details so that I can try to reproduce and learn from it.

Thanks,
Nascif