Reduce telemetry noise

[tomkerkhove]

Proposal

Reduce telemetry noise by:

• Reducing official Microsoft telemetry to warning only
• Only log HTTP requests for /api/v1/health when it fails

Component

Resource Discovery, Scraper

Contact Details

No response

[locmai]

Hi again! We are having a logging noise issue as well with both scraper and resource discovery even when we set the log level to error.

[18:40:15 INF] Request finished in 0.5297ms 200 text/plain; version=0.0.4
[18:40:17 INF] Request starting HTTP/1.1 GET http://x.y.z.w/api/v1/health?includeDependencies=false

Is this the same issue?

And could it possibly because of this webHostBuilder with no log level configuration?

promitor/src/Promitor.Agents.Core/AgentProgram.cs

Line 33 in d5647fa

.UseSerilog()

[tomkerkhove]

I will add this to the checklist but the whole Serilog setup has to change for sure.

[geertn]

We have the same issue. Is there any known workaround? This is currently our biggest source of logging adding up to our costs.

│ [09:18:12 INF] Request starting HTTP/1.1 GET http://10.11.10.181:8888/api/v1/health?includeDependencies=false - -                                                                                           │
│ [09:18:12 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(Boolean) on  │
│ [09:18:12 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.HealthController.Get (Promitor.Agents.ResourceDiscovery) in 0.2218ms                                                            │
│ [09:18:12 WRN] {"RequestMethod": "GET", "RequestHost": "http://10.11.10.181:8888", "RequestUri": "/api/v1/health", "ResponseStatusCode": 200, "RequestDuration": "00:00:00.0004844", "RequestTime": "2022-0 │
│ [09:18:12 INF] Request finished HTTP/1.1 GET http://10.11.10.181:8888/api/v1/health?includeDependencies=false - - - 200 60 application/json;+charset=utf-8 0.6816ms                                         │

[tomkerkhove]

No but this will be fixed in the next version, sorry about that.

[locmai]

We have turned off the log on all environments for a while now to save the logging storage cost. We only turn it on QA env for debugging.

[cktwo]

Any news?

[tomkerkhove]

Nope but open to contributions for it though!

[jkutes]

Any update on this?
Any workarounds?
Would there be an easy way to disable logging via a helm value or env var?

Perhaps easiest solution (not the prettiest :)) would be to change "Logger.LogInformation" to "Logger.LogDebug"?
... in Scraper.cs & ResourcesScrapingJob.cs.

Thanks

[tomkerkhove]

Waiting for somebody to contribute it :)