Remove Fleet Agentcfg Fetcher (which implements an md5 call)

[maxgio92]

Hello,
it would be nice to replace this call to md5 with a call to a FIPS 140 allowed algorithm like sha256 to calculate the etag for the fleet agent configuration.

What do you think?

Thank you

[inge4pres]

hello @maxgio92 👋🏼

i don't know how the eTag is used, but it doesn't seem like sensitive information that needs strong hashing.

Why do you think we should update it?
What benefit would we gain compared to the current usage of md5?

[maxgio92]

Hi @inge4pres, I see and understand the point.
I think anyway that the advantage is to be able to meet compliance with FIPS-140 that doesn't allow usage of md5 in general, at a pretty low cost.
Thank you

[xnox]

@inge4pres it is not security sensitive, but many FIPS go toolchains block crypto/md5 usage such that it fails at runtime. This triggered go coverage to switch away from md5 to a non-security hash FNV-1a instead (which is available in go standard library, and faster than md5) see https://go-review.googlesource.com/c/go/+/617360

Another option, which webpack chose is to switch to xxhash (which is even faster than above two) for eTag like purposes.

md5 often raises suspicion, because whilst today it almost almost used as a non-security relevant hash; it used to be classed as a secure one, and today it is a very slow choice for non-security one.

Does this make sense?

[axw]

+1 on switching away from MD5, but we'll need to be mindful of whether this is a breaking change. Similar issue: elastic/apm-data#146

It's might be possible that we can remove the linked code altogether, as we have a new mechanism for agent configuration these days. I think we kept that around for migration, but that's nearly 2 years old now.

[xnox]

+1 on switching away from MD5, but we'll need to be mindful of whether this is a breaking change. Similar issue: elastic/apm-data#146

It's might be possible that we can remove the linked code altogether, as we have a new mechanism for agent configuration these days. I think we kept that around for migration, but that's nearly 2 years old now.

would you be open compiling it out with a go build tag? that is sensitive to a custom one or any of the fips-toolchain ones? //go:build requirefips || goexperiment.systemcrypto || goexperiment.boringcrypto will ensure that a custom buildtag (popular in other projects) or a fips-oriented toolchain result in an MD5-less behaviour. And people who use such build tags or toolchains, likely to compile full set of clients and server, and deploy only a consistent set.

[kruskall]

that setup call is only used for the fallback method, it's not called if agentcfg is configured to use ES and should probably be removed in 9.0.0

IMO there's no need for adding a build tag

[xnox]

@maxgio92 it also means we can likely use go-msft toolchain as is; and let md5 fail at runtime.

[simitt]

The Fleet Agentcfg Fetcher code path should already be a dead path. With the changes we implemented over time, this is currently how agentcfg feature is implemented:

• (default) APM Server fetches config from Elasticsearch. This is the newest and default implementation. However, it requires that an Elasticsearch endpoint is configured. This is by default the case, as ES is the common method for the APM Server to send events to. If another output such as logstash or Kafka is configured, users would need to manually configure an ES endpoint for the agent configuration. If they don't do this, the below described fallback mechanism comes into play.
• (Fallback)
  • APM Server is Fleet managed and Fleet injects agent configuration into the APM Server in regular intervals. This is the problematic version we aim to remove. Fleet managed APM Server only supports Elasticsearch as output. Hence, there should be no scenario where an Elasticsearch output is not available but APM Server is Fleet managed.
  • APM Server fetches config from Kibana: this still needs to be supported in case a non-ES output is configured and no dedicated ES endpoint is configured for the agent config.

The proposed path forward is the following:

• for the remaining 8.x versions, a warning should be issued in the logs and telemetry data should be collected if the Fleet Agentcfg Fetcher code path is executed. Given the reasoning above we do not expect this code path to be executed, but telemetry will confirm.
• for >=9.0 the Agentcfg Fetcher code path should be removed.

@kruskall please reach out if there are any questions when implementing this.