Assigning a value to a table with a metatable with __newindex function causes the table reference to become nil

[Charanor]

Version: 0.4.2 AND 0.5-dev (cloned branch)
Issue: Assigning a value to a table with a metatable containing a __newindex function causes the table reference to become nil. If __newindex is a table or nil then there are no issues.

Runnable C# MVCE:

try
{
	var luaState = LuaState.Create();
	luaState.OpenBasicLibrary();
	await luaState.DoStringAsync("""
	                             local tbl = setmetatable({}, { __newindex = function() end })
	                             print(tbl)
	                             tbl.value = 0
	                             print(tbl)
	                             """);
}
catch (LuaRuntimeException e)
{
	// ...
}

Reproducible Lua code:

local tbl = setmetatable({}, { __newindex = function() end })
print(tbl) -- table: <hash>
tbl.value = 0
print(tbl) -- nil

Without __newindex there are no issues:

local tbl = {}
print(tbl) -- table: <hash>
tbl.value = 0
print(tbl) -- table: <hash>

Also no issues if __newindex is a table:

local tbl = setmetatable({}, { __newindex = {} })
print(tbl) -- table: <hash>
tbl.value = 0
print(tbl) -- table: <hash>

Maybe it's related to #72?

[Charanor]

After much trial and error... this change fixes it (LuaVirtualMachine.cs @ SetTableValueSlowPath @ line 1509):

Function:
if (table.TryReadFunction(out var function))
{
    context.PostOperation = PostOperationType.Nop;
    var callResult = CallSetTableFunc(targetTable, function, key, value, ref context, out doRestart);
    if (callResult && !doRestart)
    {
        var tableIndex = context.Instruction.A + context.FrameBase;
        context.Stack.Get(tableIndex) = targetTable;
    }
    return callResult;
}

I would open a PR but I don't know enough about the inner workings of the VM so I can't guarantee this doesn't break something else. If a maintainer can have a look at the above code I would be most grateful.

[Charanor]

Actually unfortunately this only fixes it for the simple MVCE I posted above. I still have some other code that does not work :/ Here is a MVCE that does not work with my above fix:

public class Mvce2
{
	private static readonly LuaFunction LuaCreateObject = new(async (context, memory, _) =>
	{
		var typename = context.GetArgument<string>(0);
		var state = context.State;
		var baseType = state.Environment[typename].Read<LuaTable>();
		var luaObject = await state.Environment["CreateInstance"].Read<LuaFunction>().InvokeAsync(state, [baseType]);
		var obj = new Mvce2(state, luaObject[0].Read<LuaTable>());
		await obj.OnCreated();
		memory.Span[0] = obj.luaObject;
		return 1;
	});

	private readonly LuaState luaState;
	private readonly LuaTable luaObject;

	private Mvce2(LuaState luaState, LuaTable luaObject)
	{
		this.luaState = luaState;
		this.luaObject = luaObject;
	}

	private async Task OnCreated()
	{
		var onCreated = await Index(luaObject, luaState, nameof(OnCreated));
		if (onCreated.TryRead(out LuaFunction onCreatedFn))
		{
			await onCreatedFn.InvokeAsync(luaState, [luaObject]);
		}
	}

	public static async Task Test()
	{
		try
		{
			var luaState = LuaState.Create();
			luaState.OpenBasicLibrary();
			luaState.Environment["CreateObject"] = LuaCreateObject;
			await luaState.DoStringAsync("""
			                             object_spawner = {}
			                             function object_spawner:OnCreated() 
			                                 print(self) 
			                                 self.value = 0 
			                                 print(self) 
			                             end
			                             function Test() 
			                                 CreateObject("object_spawner") 
			                             end 
			                             function CreateInstance(luaType)
			                                 return setmetatable({}, {
			                                     __index = luaType,
			                                     __newindex = function(tbl, key, value) end
			                                 })
			                             end
			                             """);
			var testFn = luaState.Environment["Test"].Read<LuaFunction>();
			await testFn.InvokeAsync(luaState, []);
		}
		catch (LuaRuntimeException e)
		{
			AnsiConsole.WriteException(e);
		}
	}

	public static async ValueTask<LuaValue> Index(LuaTable table, LuaState state, LuaValue key)
	{
		if (table.TryGetValue(key, out var value))
		{
			return value;
		}

		if (!(table.Metatable?.ContainsKey(Metamethods.Index) ?? false))
		{
			return LuaValue.Nil;
		}

		const string k = "__temp_table_value";
		var lastValue = state.Environment[k];
		state.Environment[k] = table;
		var indexResult = (await state.DoStringAsync($"return {k}['{key}']"))[0];
		state.Environment[k] = lastValue;
		return indexResult;
	}
}

[akeit0]

I couldn't reproduce this on 0.5-dev.

local tbl = setmetatable({}, { __newindex = function() end })
print(tbl) -- table: <hash>
tbl.value = 0
print(tbl) -- nil