mlua and lua_setfenv integration

[bobbens]

Hello, I'm still working on moving a large C codebase to mlua, and have hit a snag using environments in C with lua_setfenv using mlua::Tables. In particular:

1. I create the table with mlua
2. I register the table with lua.create_registry_value()
3. I later push the value to C with RegistryValue::id()
4. and set it with lua_setfenv before calling a Lua function XXX
5. However, if I try to use lua.globals() rust side from functions being called from Lua function XXX, they don't seem to see the globals from the table created in 1.

I am using luajit. I assume I'm misunderstanding how the globals work in mlua vs lua5.1 C API? Is it possible to get them to play well together?

[bobbens]

I've made a minimal test case on the issue:

fn main() -> mlua::Result<()> {
   #[allow(non_snake_case)]
   let L = unsafe{ mlua::ffi::luaL_newstate() };
   let lua = unsafe { mlua::Lua::init_from_ptr( L ) };

   unsafe {
      mlua::ffi::luaopen_base( L );
   }

   let t = lua.create_table()?;

   let m = lua.create_table()?;
   m.set("__index", lua.globals())?;
   t.set_metatable(Some(m));

   t.set( "testfunc", lua.create_function( |lua, ()| {
       let name: mlua::Value = lua.globals().get("__name")?;
       dbg!(name);
       Ok(())
    })?)?;

   t.set("__name", "foo" )?;
   let rk = lua.create_registry_value( t )?;

   lua.load("
function main ( str )
   print( __name, str )
   testfunc()
end
   ").exec()?;

   unsafe {
      mlua::ffi::lua_getglobal( L, c"main".as_ptr() );
      mlua::ffi::lua_pushstring( L, c"hello world".as_ptr() );
      mlua::ffi::lua_rawgeti( L, mlua::ffi::LUA_REGISTRYINDEX, rk.id().into() );
      mlua::ffi::lua_setfenv( L, -3 );
      if mlua::ffi::lua_pcall( L, 1, 0, 0 ) != 0{
          dbg!( std::ffi::CStr::from_ptr( mlua::ffi::luaL_tolstring( L, -1, std::ptr::null_mut() ) ) );
      }
   }

   Ok(())
}

Which prints:

 cargo run
   Compiling mluatest v0.1.0 (/home/ess/naev/mluatest)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/mluatest`
foo     hello world
[src/main.rs:18:8] name = nil

I would expect the last print to be the value of __name, but it is not.

EDIT: Created repo at https://github.com/bobbens/mlua_setfenv_globals for easy testing.

[khvzak]

testfunc is a C function and lua.globals() will always be the thread (main) globals regardless of what you do via setfenv.
It's equivalent to

lua_pushvalue(L, LUA_GLOBALSINDEX);

Another note, when you call a Lua function inside another Lua function with modified environment, inner function has own environment and does not inherit from caller (only from creator).

See:

function a()
    print("a", global_var)
    b()
end

function b()
    print("b", global_var)
end

global_var = 10

setfenv(a, setmetatable({global_var = 20}, {__index = _G}))
a()

prints:

a	20
b	10

[bobbens]

Thanks for the quick response. That's weird, because the same set up is working without Rust, but with C. Maybe something else weird is going on.

How would it be possible to handle the above case? As in, set the globals before calling a function so that it extends to all subfunctions? Would the solution be to have the global table use use metatable operators __index and __newindex to handle this sort of virtual environment? Or is there some more elegant solution?

[khvzak]

That's weird, because the same set up is working without Rust, but with C. Maybe something else weird is going on.

Would be helpful to understand how exactly this works in your (original) code. C functions in Lua 5.1 have globals (LUA_GLOBALSINDEX) and environment (LUA_ENVIRONINDEX). The last one was removed from Lua 5.2+ and mlua does not use at all LUA_ENVIRONINDEX.

[bobbens]

That's weird, because the same set up is working without Rust, but with C. Maybe something else weird is going on.

Would be helpful to understand how exactly this works in your (original) code. C functions in Lua 5.1 have globals (LUA_GLOBALSINDEX) and environment (LUA_ENVIRONINDEX). The last one was removed from Lua 5.2+ and mlua does not use at all LUA_ENVIRONINDEX.

The original code uses lua_upvalueindex( 1 )to handle the "global" variables, which I don't think is accessible via the Rust mlua API.

It's structured so that there are a ton of separate environments, but they share a single lua_State. Each script is run in its own environment, that has its own globals and requires (effectively like a lua_State). The reason for sharing a lua_State is so they can send messages to each other directly through some special API.

Looking around, it seems like potentially https://stackoverflow.com/a/45808326 is a solution, although I'm not sure how well it does performance-wise.

Also, Lua 5.2+'s _ENV seems to be more like what I want, but it doesn't seem like it's possible to use with luajit without using debug.sethook...

[bobbens]

I think I've been able to solve what I wanted to do with: https://github.com/bobbens/mlua_setfenv_globals

I'm not entirely happy with the implementation because of the following seems a bit suboptimal, but it works.

        wrapped_mt.set( "__index", lua.create_function( |lua, (t, k): (mlua::Table, mlua::Value)| -> mlua::Result<mlua::Value> {
            let env_str = lua.create_string(ENV)?;
            match k == mlua::Value::String(env_str) {
                true => t.raw_get( k ),
                false => {
                    let e: mlua::Table = t.raw_get(ENV)?;
                    e.get(k)
                },
            }
        } )?)?;

The Lua version for reference is below.

local globals = setmetatable( _G, {
   __newindex = function()
      error("globals are read only")
   end
} )
local function env_new ()
   return setmetatable( {}, { __index=globals } )
end
local myG = {}
local function env_set( env )
   rawset( myG, "_ENV", env )
end
setfenv( 1, setmetatable( myG, {
   __index = function( t, k )
      if k=="_ENV" then
         return rawget( t, k )
      end
      return rawget( t, "_ENV" )[k]
   end,
   __newindex = function( t, k, v )
      rawget( t, "_ENV" )[k] = v
   end,
} ) )

local ENV1 = env_new()
local ENV2 = env_new()

env_set( ENV1 )
assert( cat==nil )
cat = 5
assert( cat==5 )
cat = 7
assert( cat==7 )

env_set( ENV2 )
assert( cat==nil )
cat = 9
assert( cat==9 )

env_set( ENV1 )
assert( cat==7 )