Client channel overflow caused by too many e2 functions with additional e2-cores

[Linus045]

Describe the bug
It seems that too many e2 functions can cause a client buffer overflow issue on reload/restart e.g. "Disconnect: Client 0 overflowed reliable channel".

I've noticed this issue myself when working on my vgui core addon in the past.
My e2_vgui_core addon adds a lot of new e2 functions.
The problem only occurs when reloading/restarting the map (in singleplayer) with the addon. I am unable to test multiplayer.
The issue is described here as well: Linus045/e2_vgui_core#32.

I've done some testing and it appears that the "wire_expression2_sendfunctions_think" hook that triggers on "Think" to send the e2 function signatures to the client causes problems, see https://github.com/wiremod/wire/blob/master/lua/entities/gmod_wire_expression2/core/init.lua#L367-L391.

As mentioned in the Gmod Wiki here:
https://wiki.facepunch.com/gmod/Networking_Usage#netlimits

The net library has a 64kb (65535 bytes - 3 bytes used internally) limit per message. The library has an internal buffer that has roughly a 256kb(2048kbit) limit before it overflows, and if it overflows, it will cause clients that receive the net message to disconnect. So do NOT fill the buffer.
It also has clienside a reliable stream which will overflow if too many net messages are queued and the total size of all net messages is over 256kb(2048bit).

I think the issue seems to be that the internal buffer of 256kb(2048kbit) overflows causing the client to disconnect.
I've added a "proof-of-concept" solution using a timer.Simple to add a longer delay between each net-message and that seems to solve the problem, however this is not a good/viable solution.
See here: master...Linus045:wire:net_message_client_overflow

I'm not familiar with wire's codebase so I don't know the best approach to fix this.

To Reproduce
Reproducing this issue with wiremod alone is not easily possible.
However I was able to reproduce the bug with just having wiremod and my e2_vgui_core installed.
Steps then are pretty simple:

1. Start Gmod
2. Start singleplayer world
3. Restart singleplayer world (e.g. Esc -> New Game or just 'restart' in console)
4. "Disconnect: Client 0 overflowed reliable channel" error appears

Expected behavior
The client should not receive a "Disconnect: Client 0 overflowed reliable channel" message and the net-messages should get spread over a bigger time frame to prevent internal buffer overflows.

Screenshots

Setting readyToSend = true
Setting readyToSend = false
Bytes after calling .Start: 	3	24
Bytes written before sending: 	64078	512620
Setting readyToSend = true
Setting readyToSend = false
Bytes after calling .Start: 	3	24
Bytes written before sending: 	64040	512317
Setting readyToSend = true
Setting readyToSend = false
Bytes after calling .Start: 	3	24
Bytes written before sending: 	64118	512937
Setting readyToSend = true
Setting readyToSend = false
Bytes after calling .Start: 	3	24
Bytes written before sending: 	64002	512014
Setting readyToSend = true
Setting readyToSend = false
Bytes after calling .Start: 	3	24
Bytes written before sending: 	9629	77027
Setting readyToSend = true
Setting readyToSend = false
Setting readyToSend = true
Setting readyToSend = false

This is the console log my print statements generate.
If we add up all the bytes that are sent we get: 64078 + 64040 + 64118 + 64002 + 9629 = 265867 which exceeds the internal buffer limit mentioned in the wiki of roughly 256kb (see above).

The big net-messages can also be seen when using 'net_graph 4' in the console or activating "Show network graph with extra info" in the console's "Extra" menu at the top.

[thegrb93]

Can probably be fixed using the new WireLib.NetQueue

[thegrb93]

Will commit this later.

local E2FunctionQueue = WireLib.NetQueue("e2_functiondata")
local E2FUNC_SENDMISC, E2FUNC_SENDFUNC, E2FUNC_DONE = 0, 1, 2
if SERVER then
	-- Serverside files are loaded in extloader
	include("extloader.lua")

	-- -- Transfer E2 function info to the client for validation and syntax highlighting purposes -- --

	do
		local miscdata = {} -- Will contain {E2 types info, constants}, this whole table is under 1kb
		local functiondata = {} -- Will contain {functionname = {returntype, cost, argnames, extension}, this will be between 50-100kb

		-- Fills out the above two tables
		function wire_expression2_prepare_functiondata()
			-- Sanitize events so 'listening' e2's aren't networked
			local events_sanitized = {}
			for evt, data in pairs(E2Lib.Env.Events) do
				events_sanitized[evt] = {
					name = data.name,
					args = data.args
				}
			end

			local types = {}
			for typename, v in pairs(wire_expression_types) do
				types[typename] = v[1] -- typeid (s)
			end

			miscdata = { types, wire_expression2_constants, events_sanitized }
			functiondata = {}

			for signature, v in pairs(wire_expression2_funcs) do
				functiondata[signature] = { v[2], v[4], v.argnames, v.extension, v.attributes } -- ret (s), cost (n), argnames (t), extension (s), attributes (t)
			end
		end

		wire_expression2_prepare_functiondata()

		-- Send everything
		local function sendData(ply)
			if not (IsValid(ply) and ply:IsPlayer()) then return end

			local queue = E2FunctionQueue.plyqueues[ply]
			queue:add(function()
				net.WriteUInt(E2FUNC_SENDMISC, 8)
				net.WriteTable(miscdata[1])
				net.WriteTable(miscdata[2])
				net.WriteTable(miscdata[3])
			end)
			for signature, tab in pairs(functiondata) do
				queue:add(function()
					net.WriteUInt(E2FUNC_SENDFUNC, 8)
					net.WriteString(signature) -- The function signature ["holoAlpha(nn)"]
					net.WriteString(tab[1]) -- The function's return type ["s"]
					net.WriteUInt(tab[2] or 0, 16) -- The function's cost [5]
					net.WriteTable(tab[3] or {}) -- The function's argnames table (if a table isn't set, it'll just send a 1 byte blank table)
					net.WriteString(tab[4] or "unknown")
					net.WriteTable(tab[5] or {}) -- Attributes
				end)
			end
			queue:add(function()
				net.WriteUInt(E2FUNC_DONE, 8)
			end)
			E2FunctionQueue:flushQueue(ply, queue)
		end

		local antispam = WireLib.RegisterPlayerTable()
		function wire_expression2_sendfunctions(ply, isconcmd)
			if isconcmd and not game.SinglePlayer() then
				if not antispam[ply] then antispam[ply] = 0 end
				if antispam[ply] > CurTime() then
					ply:PrintMessage(HUD_PRINTCONSOLE, "This command has a 60 second anti spam protection. Try again in " .. math.Round(antispam[ply] - CurTime()) .. " seconds.")
					return
				end
				antispam[ply] = CurTime() + 60
			end
			sendData(ply)
		end

		-- add a console command the user can use to re-request the function info, in case of errors or updates
		concommand.Add("wire_expression2_sendfunctions", wire_expression2_sendfunctions)

		if game.SinglePlayer() then
			-- If single player, send everything immediately
			hook.Add("PlayerInitialSpawn", "wire_expression2_sendfunctions", sendData)
		end
	end

elseif CLIENT then

	e2_function_data_received = nil
	-- -- Receive E2 function info from the server for validation and syntax highlighting purposes -- --

	wire_expression2_reset_extensions()

	local function insertData(signature, ret, cost, argnames, extension, attributes)
		local fname = signature:match("^([^(:]+)%(")
		if fname then
			wire_expression2_funclist[fname] = true
			wire_expression2_funclist_lowercase[fname:lower()] = fname
		end
		if not next(argnames) then argnames = nil end -- If the function has no argnames table, the server will just send a blank table
		wire_expression2_funcs[signature] = { signature, ret, false, cost, argnames = argnames, extension = extension, attributes = attributes }
	end

	local function doneInsertingData()
		e2_function_data_received = true

		if wire_expression2_editor then
			wire_expression2_editor:Validate(false)

			-- Update highlighting on all tabs
			for i = 1, wire_expression2_editor:GetNumTabs() do
				wire_expression2_editor:GetEditor(i).PaintRows = {}
			end
		end
	end

	---@param events table<string, {name: string, args: { placeholder: string, type: string }[]}>
	local function insertMiscData(types, constants, events)
		wire_expression2_reset_extensions()

		-- types
		for typename, typeid in pairs(types) do
			wire_expression_types[typename] = { typeid }
			wire_expression_types2[typeid] = { typename }
		end

		-- constants
		wire_expression2_constants = constants
		E2Lib.Env.Events = events
	end

	function E2FunctionQueue.receivecb()
		local state = net.ReadUInt(8)
		if state==E2FUNC_SENDFUNC then
			insertData(net.ReadString(), net.ReadString(), net.ReadUInt(16), net.ReadTable(), net.ReadString(), net.ReadTable())
		elseif state==E2FUNC_SENDMISC then
			insertMiscData(net.ReadTable(), net.ReadTable(), net.ReadTable())
		elseif state==E2FUNC_DONE then
			doneInsertingData()
		end
	end
end