Hello, I'm trying to replace all missing values in a data table with 0. The below script doesn't work. I use JMP 13. Is there any other way to do this? dt = Current Data Table ( ) ; nc = dt get column names ( Numeric ) ; For ( i = 1 , i N Items ( nc ) , i ++ , nc [ i ] [ dt get rows where ( Is Missing ( nc [ i ] [ ] ) ) ] = 0 ) ; Thanks!

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Replace missing values with 0

nshivan

Hello,

I'm trying to replace all missing values in a data table with 0. The below script doesn't work. I use JMP 13. Is there any other way to do this?

dt <SPAN class="token operator">=</SPAN> <SPAN class="token function">Current Data Table</SPAN><SPAN class="token punctuation">(</SPAN><SPAN class="token punctuation">)</SPAN><SPAN class="token punctuation">;</SPAN>
nc <SPAN class="token operator">=</SPAN> dt <SPAN class="token operator"><</SPAN><SPAN class="token operator"><</SPAN><SPAN class="token messages"> get column names</SPAN><SPAN class="token punctuation">(</SPAN> Numeric <SPAN class="token punctuation">)</SPAN><SPAN class="token punctuation">;</SPAN>
<SPAN class="token function">For</SPAN><SPAN class="token punctuation">(</SPAN> i <SPAN class="token operator">=</SPAN> <SPAN class="token number">1</SPAN><SPAN class="token punctuation">,</SPAN> i <SPAN class="token operator"><=</SPAN> <SPAN class="token function">N Items</SPAN><SPAN class="token punctuation">(</SPAN> nc <SPAN class="token punctuation">)</SPAN><SPAN class="token punctuation">,</SPAN> i<SPAN class="token operator">++</SPAN><SPAN class="token punctuation">,</SPAN>
  nc<SPAN class="token punctuation">[</SPAN>i<SPAN class="token punctuation">]</SPAN><SPAN class="token punctuation">[</SPAN>dt <SPAN class="token operator"><</SPAN><SPAN class="token operator"><</SPAN><SPAN class="token messages"> get rows where</SPAN><SPAN class="token punctuation">(</SPAN> <SPAN class="token function">Is Missing</SPAN><SPAN class="token punctuation">(</SPAN> nc<SPAN class="token punctuation">[</SPAN>i<SPAN class="token punctuation">]</SPAN><SPAN class="token punctuation">[</SPAN><SPAN class="token punctuation">]</SPAN> <SPAN class="token punctuation">)</SPAN> <SPAN class="token punctuation">)</SPAN><SPAN class="token punctuation">]</SPAN> <SPAN class="token operator">=</SPAN> <SPAN class="token number">0</SPAN>
<SPAN class="token punctuation">)</SPAN><SPAN class="token punctuation">;</SPAN>

Thanks!

Accepted answers

txnelson

Here is the correct syntax for what you are trying to do

dt = Current Data Table();
nc = dt << get column names( Numeric );
For( i = 1, i <= N Items( nc ), i++,
	Column( nc[i] )[dt << get rows where( Is Missing( As Column( nc[i] ) ) )] = 0
);

All comments

txnelson

Here is the correct syntax for what you are trying to do

dt = Current Data Table();
nc = dt << get column names( Numeric );
For( i = 1, i <= N Items( nc ), i++,
	Column( nc[i] )[dt << get rows where( Is Missing( As Column( nc[i] ) ) )] = 0
);

nshivan

Hi Jim,

I tried the above, but getting the folliowing error.

could not find column in access or evaluation of 'Column' , Column/*###*/(nc[i])

Any idea what's going on? I use JMP 13, I'm not sure if the syntax isn't working for this version.

Thanks,

cwillden

Hi @nshivan,

I ran Jim's code in versions 13 and 14 and got no errors. Maybe restart JMP and run it again?

nshivan

Hi Cameron,

I restarted JMP and the script works fine on the specific data table. The missing values get replaced with 0s. But when I integrate to the rest of my script, it gives me the below error:

could not find column in access or evaluation of 'Column' , Column/*###*/(nc[i])

Prior to this piece of script, I have an original data table which I subset based on selected columns, stack the data, summarize anf then tabulate into the final data table. I'm trying to replace missing values in the final data table with 0s.

I'm fairly new to JMP scripts, not sure why the integration is giving problems. Any idea why this is happening?

Evan_Morris

I use this script constantly. Can't thank you enough for providing it.

geoffrey_mann

Hi all,

I have been working on this exact problem with a hundreds of columns with hundreds of missing rows. I wasn't happy with the performance and tried many approaches. It seems that Recode solves this problem about 150 times faster than the current solution. Sharing the code with anyone else that might care about speed. Please let me know if you find something faster. Every millisecond counts!

</CODE></P> <PRE><CODE class=" language-jsl">replaceMissingStart = HP Time(); // replace missings with zeros nc = dmdt << get column names( Numeric ); For( i = 1, i <= N Items( nc ), i++, col = Column(dt,nc[i]); colname = col << getname(); Eval(Parse(EvalInsert("\[ dmdt << Recode Column( :Name("^colname^"), {Map Value( _rcOrig, {., 0}, Unmatched( _rcNow ) )}, Update Properties( 1 ), Target Column( :Name("^colname^") ) ); ]\"))); //Code from the Community //Column( nc[i] )[dmdt << get rows where( Is Missing( As Column( nc[i] ) ) )] = 0; // standardize column attributes, change all to nominal col << Set Modeling Type( "Nominal" ) ); replaceMissingStop = HP Time(); totalReplaceTime = replaceMissingStop-replaceMissingStart; show(totalReplaceTime/1000000);

Thanks to Rebecca Lyzinski for the idea!!!

ian_jmp

FWIW, this sort of approach might be better. For 10,000 by 10,000 it took about 10 seconds on my laptop (including making the starting table):

NamesDefaultToHere(1);

// Size the problem . . .
nr = 10000;		// Number of rows
nc = 10000;		// Number of columns
nm = 0.1;		// Proportion of missing values

// Make a table and inject some missing values
mat = J(nr*nc, 1, RandomNormal());
mat[RandomIndex(nr*nc, nr*nc*nm)] = .;
dt = AsTable(Shape(mat, nr, nc));
dt << setName("Data with some missingness");

// Start here: Get values into a matrix (assumes contiguous numeric columns)
mat = dt << getAsMatrix;
// Reshape as a vector
mat = Shape(mat, nr*nc, 1);
// Replace missing
mat[Loc(mat, .)] = 0;
// Reshape back to a matrix
mat = Shape(mat, nc, nc);
// Make a new table
dt2 = AsTable(Shape(mat, nr, nc));
dt2 << setName("Data with missing set to zero values");

Craige_Hales

Combining solutions, with data table subscripting...

nr = 10000;		// Number of rows
nc = 10000;		// Number of columns
nm = 0.9;		// Proportion of missing values

// Make a table and inject some missing values
mat = J(nr*nc, 1, RandomNormal());
mat[RandomIndex(nr*nc, nr*nc*nm)] = .;
dt = AsTable(Shape(mat, nr, nc));
dt << setName("Data with some missingness");


replaceMissingStart = HP Time();

nc = dt << get column names( Numeric );
mat = dt[0,nc];
mat[Loc(mat, .)] = 0;
dt[0,nc] = mat;


replaceMissingStop = HP Time();
totalReplaceTime = replaceMissingStop-replaceMissingStart;
show(totalReplaceTime/1000000);

About 10 seconds.

geoffrey_mann

Dear Ian and Craige,

Thank you so much for your submissions. I ran some preliminary tests and Ian's solution is more than 4 times faster than Recode for this and Craige's solution is almost 10 times faster. If anyone would like to compare all 3 solutions, try the following:

//Recode solution

nr = 10000;		// Number of rows
nc = 10000;		// Number of columns
nm = 0.9;		// Proportion of missing values

// Make a table and inject some missing values
mat = J(nr*nc, 1, RandomNormal());
mat[RandomIndex(nr*nc, nr*nc*nm)] = .;
dt = AsTable(Shape(mat, nr, nc));
dt << setName("Data with some missingness");

replaceMissingStart = HP Time();

// replace missings with zeros
nc = dt << get column names( Numeric );

For( i = 1, i <= N Items( nc ), i++,
	col = Column(dt,nc[i]);
	colname = col << getname();

   	Eval(Parse(EvalInsert("\[
		dt << Recode Column(
			:Name("^colname^"),
			{Map Value( _rcOrig, {., 0}, Unmatched( _rcNow ) )},
			Update Properties( 1 ),
			Target Column( :Name("^colname^") )
		);
	]\")));
	
	//Code from the Community
	//Column( nc[i] )[dmdt << get rows where( Is Missing( As Column( nc[i] ) ) )] = 0;

	// standardize column attributes, change all to nominal
	//col << Set Modeling Type( "Nominal" ) 
);

replaceMissingStop = HP Time();
totalReplaceTime = replaceMissingStop-replaceMissingStart;
show(totalReplaceTime/1000000);

Russ Wolfinger had mentioned he thought matrices would solve the problem more quickly, I guess that was what you two were thinking as well!

Goblin_King

in Can I replace missing values in Numeric column with "N/A" string?

mikedriscoll

>> Can I replace missing values in Numeric column with "N/A" string?

Use value labels to do that. Right click on column header, go to column properties, select Value Labels. Then put a period in the value and N/A in the label field as shown below. Then click add, and then ok.

To script it, follow the log recorder of above steps...

Data Table( "DT name" ):Column 904 << Value Labels( {"." = "N/A"} ) << Use Value Labels( 1 );