[Stack 9.0.0-SNAPSHOT] failing tests in packages due to file size is too small for ingestion (no documents found)

[mrodm]

There are some packages that are failing in the daily CI Jobs for the Elastic stack 9.0.0-SNAPSHOT.

Example of build failing: https://buildkite.com/elastic/integrations/builds/21038

These packages are failing due to errors in the elastic-agent like this one and no documents are ingested into Elasticsearch:

• apache_tomcat: @elastic/obs-infraobs-integrations

cannot start ingesting from file "/tmp/service_logs/localhost.log": filesize of "/tmp/service_logs/localhost.log" is 319 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion

• coredns @elastic/obs-infraobs-integrations

cannot start ingesting from file "/tmp/service_logs/coredns.log": filesize of "/tmp/service_logs/coredns.log" is 383 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion

• gitlab @elastic/security-service-integrations

cannot start ingesting from file "/tmp/service_logs/test-gitlab-audit.log": filesize of "/tmp/service_logs/test-gitlab-audit.log" is 781 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion

• microsoft_exchange_server @elastic/sec-windows-platform

cannot start ingesting from file "/tmp/service_logs/imappop_Imap4.log": filesize of "/tmp/service_logs/imappop_Imap4.log" is 246 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":394,"file.name

• ping_federate @elastic/security-service-integrations

cannot start ingesting from file "/tmp/service_logs/test-admin.log": filesize of "/tmp/service_logs/test-admin.log" is 189 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion

• tychon @elastic/security-service-integrations

cannot start ingesting from file "/tmp/service_logs/test-base.ndjson": filesize of "/tmp/service_logs/test-base.ndjson" is 99 bytes, expected at least 1024 bytes for fingerprinting: file size is too small for ingestion

[mrodm]

Probably related to these issues (and could be closed if this issue is solved):

• [Stack 9.0.0-SNAPSHOT] [ping_federate] Failing test daily: system test: logfile in ping_federate.admin #12321
• [Stack 9.0.0-SNAPSHOT] [ping_federate] Failing test daily: system test: logfile in ping_federate.audit #12322
• [Stack 9.0.0-SNAPSHOT] [tychon] Failing test daily: system test: default in tychon.cmrs #12327
• [Stack 9.0.0-SNAPSHOT] [microsoft_exchange_server] Failing test daily: system test: default in microsoft_exchange_server.smtp #12320
• [Stack 9.0.0-SNAPSHOT] [microsoft_exchange_server] Failing test daily: system test: default in microsoft_exchange_server.imap4_pop3 #12318
• [Stack 9.0.0-SNAPSHOT] [gitlab] Failing test daily: system test: filestream in gitlab.sidekiq #12308
• [Stack 9.0.0-SNAPSHOT] [gitlab] Failing test daily: system test: default in gitlab.pages #12306
• [Stack 9.0.0-SNAPSHOT] [coredns] Failing test daily: system test: filestream (variant: v1) in coredns.log #12303

Due to some other error there is no issue created (yet) for apache_tomcat.

EDIT:

• [Stack 9.0.0-SNAPSHOT] [apache_tomcat] Failing test daily: system test: default (variant: v10.1.5) in apache_tomcat.localhost #12522

[mrodm]

Looking into the logs, it can be checked how fingerprint mode is enabled:

{"log.level":"debug","@timestamp":"2025-01-28T01:49:24.339Z","message":"fingerprint mode enabled: offset 0, length 1024","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"scanner","log.origin":{"file.line":311,"file.name":"filestream/fswatch.go","function":"github.com/elastic/beats/v7/filebeat/input/filestream.newFileScanner"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-01-28T01:49:24.339Z","message":"recursive glob enabled","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"scanner","log.origin":{"file.line":334,"file.name":"filestream/fswatch.go","function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).resolveRecursiveGlobs"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-01-28T01:49:24.339Z","message":"file identity is set to fingerprint","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"input.filestream","log.origin":{"file.line":57,"file.name":"filestream/prospector_creator.go","function":"github.com/elastic/beats/v7/filebeat/input/filestream.newProspector"},"service.name":"filebeat","filestream_id":"filestream-coredns.log-48963670-6129-4d62-b9ce-adf88595cf96","ecs.version":"1.6.0"}

In this PR elastic/beats#41762 , the default file_identity is changed from native to fingerprint.

There were some packages that added options to disable this file_identity mode: #12349 (issue #12325)

Another option would be to update the corresponding test docs to have more than 1024 bytes.

[efd6]

Another possible option is to expose the prospector.scanner.fingerprint.length in the integrations so that they can be set lower in tests (or alternatively just set them lower, though this may be unwise given the small size of some of the test cases).

[efd6]

Addressing the SSI integrations:

• gitlab gitlab,ping_federate: make system tests pass on 9.0.0 #12510
• ping_federate gitlab,ping_federate: make system tests pass on 9.0.0 #12510

[mrodm]

Thanks @efd6!

Should the related issues to gitlab and ping_federate mentioned in #12499 (comment) be closed too ?

EDIT:
I see that tychon is also related to SSI integrations, could tychon package be updated similar to gitlab or ping_federate?

[chemamartinez]

@mrodm I am already taking a look at the Tychon failure.

We can wait until confirm in the daily build that every issue affecting these packages has been solved. It runs daily at 1:35am UTC: https://buildkite.com/elastic/integrations/builds?query=Check+integrations+local+stacks+-+Stack+Version+v9.0

[efd6]

@mrodm Closed those. Apologies for missing that.

[mrodm]

Thanks for solving the issues related to the SSI packages @efd6 @chemamartinez !

@elastic/obs-infraobs-integrations @elastic/sec-windows-platform could you please take a look to the packages related to your teams (apache_tomcat, coredns and microsoft_exchange_server)? Once solved, this would help to run all the tests in those packages successfully for 9.0.0-SNAPSHOT.

Thanks in advance!

[ishleenk17]

Yes @mrodm , we are taking care of these as part of this ticket.