Incident Report: Apr 3, 2023

[dyc3]

Server crashed with this error

2023-04-03 13:14:15.869 [opentogethertube] web.1 /app/node_modules/lodash/lodash.js:653
2023-04-03 13:14:15.869 [opentogethertube] web.1       result[index] = iteratee(array[index], index, array);
2023-04-03 13:14:15.869 [opentogethertube] web.1                       ^
2023-04-03 13:14:15.870 [opentogethertube] web.1 TypeError: Cannot read properties of null (reading 'service')
2023-04-03 13:14:15.870 [opentogethertube] web.1     at /app/server/room.ts:609:42
2023-04-03 13:14:15.870 [opentogethertube] web.1     at /app/node_modules/lodash/lodash.js:3782:18
2023-04-03 13:14:15.870 [opentogethertube] web.1     at arrayMap (/app/node_modules/lodash/lodash.js:653:23)
2023-04-03 13:14:15.870 [opentogethertube] web.1     at /app/node_modules/lodash/lodash.js:3781:24
2023-04-03 13:14:15.870 [opentogethertube] web.1     at /app/node_modules/lodash/lodash.js:3585:27
2023-04-03 13:14:15.870 [opentogethertube] web.1     at /app/node_modules/lodash/lodash.js:4943:15
2023-04-03 13:14:15.870 [opentogethertube] web.1     at baseMap (/app/node_modules/lodash/lodash.js:3584:7)
2023-04-03 13:14:15.870 [opentogethertube] web.1     at baseOrderBy (/app/node_modules/lodash/lodash.js:3780:20)
2023-04-03 13:14:15.870 [opentogethertube] web.1     at Function.orderBy (/app/node_modules/lodash/lodash.js:9665:14)
2023-04-03 13:14:15.870 [opentogethertube] web.1     at /app/server/videoqueue.ts:120:20

server/room.ts:609:42 is in the following block:

// sort queue according to queue mode
if (this.queueMode === QueueMode.Vote) {
	await this.queue.orderBy(
		[
			video => {
				const votes = this.votes.get(video.service + video.id); // line 609
				return votes ? votes.size : 0;
			},
		],
		["desc"]
	);
}

The crash is the result of video being null for some reason, which should be impossible.

Alerted at 8:00 am
Response started at 9:30 am

• Restarting the server does not fix it, because its saved into a room's state in redis, which is loaded on startup
• --- is the room with the bad state
• manually removed --- from redis

Service restored at 9:47 am

Follow up items

• when a room's update() fails, it should not crash the whole server
  • needs to log which room failed