Error: http panic after upgrading from Druid 25.0.0 to Druid 29.0.0

[frankgrimes97]

We're running opstree/druid-exporter:v0.11 and recently noticed the following errors after upgrading Druid (from 25.0.0 to 29.0.0):

2024/03/06 19:17:12 http: panic serving <IP_REDACED>:<PORT_REDACTED>: interface conversion: interface {} is nil, not string
goroutine 1949 [running]:
net/http.(*conn).serve.func1(0xc0006183c0)
	/usr/local/go/src/net/http/server.go:1801 +0x147
panic(0x8c85a0, 0xc00071d380)
	/usr/local/go/src/runtime/panic.go:975 +0x47a
druid-exporter/listener.DruidHTTPEndpoint.func1(0x9e10c0, 0xc00088e000, 0xc000db2200)
	/go/src/druid-exporter/listener/druid_endpoint.go:140 +0x2578
net/http.HandlerFunc.ServeHTTP(0xc00018fc80, 0x9e10c0, 0xc00088e000, 0xc000db2200)
	/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0001fe000, 0x9e10c0, 0xc00088e000, 0xc000db2000)
	/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0xd3
net/http.serverHandler.ServeHTTP(0xc00021c000, 0x9e10c0, 0xc00088e000, 0xc000db2000)
	/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0006183c0, 0x9e2940, 0xc0006f2040)
	/usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2969 +0x36c

This seems similar to what was previously reported here (and presumably fixed): #70

It seems some metrics still get scraped successfully though. e.g. we also see these in the logs:

{"level":"info","msg":"Successfully collected data from druid emitter, druid/indexer","time":"2024-03-06T19:17:13Z"}
{"level":"info","msg":"Successfully collected data from druid emitter, druid/indexer","time":"2024-03-06T19:17:14Z"}

[pchang388]

Seeing the same issue after upgrading from 26.0.0 to 27.0.0

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rmadamson]

We're also experiencing this on 31.0.1. I took a look at the change notes for Druid 27.0.0 and noticed this PR: apache/druid#14443. In it, it seems new metrics were added for the overlord and coordinator nodes which may be the root cause of this issue.