658869_en-US

liliconcom

Returning an expression from a function differs when namespace is used.

Hello everyone,

 

I ran into some strange behavior when returning an expression from a function, and could not find any discussions about this already. I am in the habit of using namespaces for pretty much everything. I found that when a function returns an expression, the expression is not evaluated for an unscoped function, but is evaluated if a namespace is used. Does anyone know what's going on here?

 

This script should show that the variable x is not incremented when the first function is called, but is incremented on the second function call. I am using JMP v17.0.0, so this has possibly been fixed, but I am not able to update right now.

 

NewNamespace("varNamespace");
varNamespace:x = 0;<BR />
myFunc = Function({},
 {DEFAULTLOCAL},
 returnVal = Expr(varNamespace:x++);
 NameExpr(returnVal);
);

myFunc();
Show(varNamespace:x);

NewNamespace("functionTest");
functionTest:myFunc = Function({},
 {DEFAULTLOCAL},
 returnVal = Expr(varNamespace:x++);
 NameExpr(returnVal);
);

functionTest:myFunc();
show(varNamespace:x);

Automation and ScriptingRe: Returning an expression from a function differs when namespace is used.

I have updated to 17.2.0, and can confirm this is still issue still occurs.

 

Here is a simplified example that should hopefully be easier to read

x = 0;
myFunc = Function({}, {DEFAULTLOCAL},
 Expr(::x++);
);

myFunc(); // Returns unevaluated expression: Expr(::x++)
Show(::x); // ::x=0;

NewNamespace("funcNS");
funcNS:myFunc = Function({},{DEFAULTLOCAL},
 Expr(::x++);
);

funcNS:myFunc(); // Returns evaluated expression;
show(::x); // ::x=1;

Re: Returning an expression from a function differs when namespace is used.

Pretty sure you've discovered a bug. Here's a even more simplified demo of two ways to use functions in namespaces:

funcNS = New Namespace(
 "funcNS",
 f1 = Function( {}, 
  expr(1+2)
 )
);

funcNS:f2 = Function( {}, 
 expr(1+2)
);
 
Show( Type( funcNS:f1() ), Type( funcNS:f2() ) );

Type(funcNS:f1()) = "Expression";
Type(funcNS:f2()) = "Number";

 

f1 is returning the correct answer; f2 is incorrectly evaluating the answer. I prefer the f2 way (except for this bug) because it is faster:

funcNS = New Namespace(
 "funcNS",
 f1 = Function( {}, 
  1
 )
);

funcNS:f2 = Function( {}, 
 1
);
 
nf1 = 0;
For( endtime = HP Time() + 1e6, endtime > HP Time(), nf1++,
 funcNS:f1()
);
Show( nf1 ); // 180K calls
nf2 = 0;
For( endtime = HP Time() + 1e6, endtime > HP Time(), nf2++,
 funcNS:f2()
);
Show( nf2 ); // 260K calls

nf1 = 182239;
nf2 = 258929;

 

The two issues are probably related. Items specified inside the namespace function call are not constructed until they are used (they are kept as expressions.) That means f1 is not compiled when the namespace is created, but f1 is compiled each time it is used, thus slower.  So there is probably some specialized C++ code to recompile f1 and then evaluate it. My guess is that same code is triggered for f2, but doesn't recompile because f2 is already compiled, and then incorrectly evaluates the result of f2, which most of the time is a primitive value and no one notices. The expr() value that is returned does evaluate into a simpler value, causing you to notice.

 

View more...

@julian  - pretty sure this is a bug. I don't think there will be many namespace users depending on the wrong behavior.

Re: Returning an expression from a function differs when namespace is used.

Has there been any update to this?
As far as i can tell this behavior still exists in jmp 19.

 

It definitely seems like the return value gets evaluated twice in the ns:f2 definition:

ns = New Namespace("fns",
	f1 = Function({},
		x = Expr(1+3);
		return(Name Expr(x));
	);
);

f1 = Function({},
	x = Expr(1+3);
	return(Name Expr(x));
);

ns:f2 = Function({},
	x = Expr(1+3);
	return(Name Expr(x));
);

ns:f3 = Function({},
	x = Expr(1+3);
	return(Name Expr(Name Expr(x)));
);

Show(f1(), ns:f1(), ns:f2(), ns:f3());

f1() = 1 + 3;
ns:f1() = 1 + 3;
ns:f2() = 4;
ns:f3() = 1 + 3;

f1() = 1 + 3;
ns:f1() = 1 + 3;
ns:f2() = 4;
ns:f3() = x;

 

If anyone else encounters this problem the quick workaround is as the example suggest; Wrap the return value in "Name Expr" until the desired result 

This does not work. I had a x defined in the global/here scope from previous runs. There seem to be no way to return an expression from a function defined like ns:f2

 

This definitely seem like a bug, i dont think anyone would expect a ns:f2 to return a different value compared to ns:f1 or f1; the function code is exactly the same.

Re: Returning an expression from a function differs when namespace is used.

I'll make a report through the support team. 

 

TS-00235798

Re: Returning an expression from a function differs when namespace is used.

I also made a report: TS-00235515

 

The response from support was that this was not considered a bug, even if the behavior was unexpected.

Root cause was as you described; Basically the ns:f2 function gets evaluated twice.

 

This seem to also affect everything defined outside the initial namespace creation, As can be seen in the updated example below.

The proposed solution is to wrap everything in an expression - ns:f2 = Expr(Function(....))

Names Default to Here(1);
delete symbols();

ns = New Namespace("fns",
	{
	f1 = Function({},{Default Local},
		x = Expr(1+3);
		return(Name Expr(x));
	),
	e1 = Name Expr(expr(1 + a))
	}
);

f1 = Function({},{Default Local},
	x = Expr(1+3);
	return(Name Expr(x));
);

ns:f2 = Function({},{Default Local},
	x = Expr(1+3);
	return(Name Expr(x));
);

ns:f3 = Expr(Function({},{Default Local},
	x = Expr(1+3);
	return(Name Expr(x));
));

ns:e2 = Name Expr(expr(1 + a));
ns:e3 = Expr(Name Expr(expr(1 + a)));

show(ns:e1);
show(try(ns:e2, exception_msg));
show(ns:e3);

Show(f1(), ns:f1(), ns:f2(), ns:f3());

ns:e1 = 1 + a;
Try(ns:e2, exception_msg) = {"Name Unresolved: a"(1, 2, "a", a /*###*/)};
ns:e3 = 1 + a;
f1() = 1 + 3;
ns:f1() = 1 + 3;
ns:f2() = 4;
ns:f3() = 1 + 3;

 

Re: Returning an expression from a function differs when namespace is used.

That workaround is...unfortunate. Fortunately, most users are not trying to return expressions and don't need it.

 

Re: Returning an expression from a function differs when namespace is used.

Also, most of the unfortunate aspect can be alleviated by returning 

expr(expr(1+1))

rather than making the entire function be

expr(function(...

funcNS = New Namespace(
	"funcNS",
	f1 = Function( {}, 
		expr(1+2)
	)
);

funcNS:f2 = Function( {}, 
	expr(expr(1+2)) // <<<< extra expr wrapper
);
	
Show( Type( funcNS:f1() ), Type( funcNS:f2() ) ); // expr, expr

Re: Returning an expression from a function differs when namespace is used.

That solution is indeed more elegant, however it seem to only work when you can directly write the expression to return.

If the expression is stored in a variable and we wrap that in an Expr() the value returned is an expression with the variable name

 

In this case it seems the only solution seems to be to wrap the return value in a Name Expr() and the entire function in an expr

funcNS:f5 = Expr(Function( {}, {Default Local},
	x = Expr(1+2);
	Name Expr(x)
));

Show( funcNs:f4() ); // 1+2

One big danger with this us that the second evaluation seem to happen outside the local scope of the function.

A global variable might shadow the return value, which would not give an error.

 

Names Default to Here(1);
Delete Symbols(); // Only use if you dont have variables you need

funcNS = New Namespace("funcNS");

funcNS:f1 = Function( {}, {Default Local},
	x = Expr(1+2);
	Expr(x);
);

funcNS:f2 = Function( {}, {Default Local},
	x = Expr(1+2);
	Expr(Expr(x));
);

funcNS:f3 = Expr(Function( {}, {Default Local},
	x = Expr(1+2);
	x;
));

funcNS:f4 = Expr(Function( {}, {Default Local},
	x = Expr(1+2);
	Expr(x);
));

funcNS:f5 = Expr(Function( {}, {Default Local},
	x = Expr(1+2);
	Name Expr(x)
));
	
Show( Try(funcNs:f1(), exception_msg) ); // Name Unresolved
Show(funcNs:f2()); // X
Show(funcNs:f3()); // 3
Show(funcNs:f4()); // x
Show(funcNs:f5()); // 1+2

// Second Evaluation happens outside Function scope
x = Expr(Expr(1+10));
Show(funcNs:f1()); // 1+10

 

Re: Returning an expression from a function differs when namespace is used.

wow!

added to Tiny Traps in Jmp and JSL 

Re: Returning an expression from a function differs when namespace is used.

 Names Default to Here(1);

x= Expr("Hello");

funcNS = New Namespace("funcNS");
funcNS:f = Function( {}, {Default Local},
	x = Expr(1+2);
	Substitute(Expr(Expr(ex)), Expr(ex), (Name Expr(x)))
);
	
Show( funcNS:f() , Type( funcNS:f() ) ); // expr, expr

• go give the function a chance to look up x inside the function:   Name Expr() instead of Expr().
• to prevent the namespace from evaluating the return value "twice", the return value has to be protected with Expr()
• this makes the life difficult for the Name Expr() 
  Expr(Name Expr()) will stop Name Expr() from doing it's job
• In 99% of such cases, one can use Eval Expr().
  But here it's difficult, because Expr() is used in JMP for 2 purposes:
  -  "it is here()" inside of Eval Expr()
  - I protect you()  - as a solution to the namespace issue
  unfortunately, Eval Expr() will be happy with the first Expr() and evaluate it.
  so, Eval Expr(Expr(Expr(Name Expr(x)))) will not work in the spirit of 
  Eval Expr ( I protect you ( it is here(Name Expr(x))))
  instead it  interpretes it as
  Eval Expr( it is here(I protect you(Name Expr(x))))
  and therefore it will return 
  Expr(Name Expr(x))
  ... and then JMP will search for x outside of the scope of the function 
  
  
• The ultimate solution to all Expression issues is Substitute()
  Re: Expression Handling in JMP: Tipps and Trapdoors 
  
• is there an easier solution than 
  Substitute(Expr(Expr(ex)), Expr(ex), (Name Expr(x)))