[qualys_vmdr] Update data collection to get unique identifiers for each interval of ingestion

[brijesh-elastic]

Proposed commit message

qualys_vmdr: use unique identifiers for each asset_host_detection collection interval

This will help to debug if there is any issue in data ingestion.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/qualys_vmdr directory.
• Run the following command to run tests.

elastic-package test

Related issues

• Closes [Qualys VMDR]: Generate unique identifiers for each interval ingestion #13167

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package qualys_vmdr 👍(2) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
user_activity	11494.25	9259.26	-2234.99 (-19.44%)	💔

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

Suggested commit message:

qualys_vmdr: use unique identifiers for each asset_host_detection collection interval

This will help to debug if there is any issue in data ingestion.

[efd6]

It seems to me that these should be always generated if they do not exist and they should not be included in the returned state in the cases that there is an error or want_more is unset or false.

[brijesh-elastic]

We should generate a UUID for the initial call when want_more is set to false. For the remaining calls within that interval, where want_more is true, we should not generate a new UUID but instead preserve the existing one.

[efd6]

Yes, but what you are doing is using want_more as a proxy for the reality. The behaviour that you describe can also be achieved by generating the UUID and storing it in state if it is not present, and removing it from state in the case that want_more is set to false or is remove from state. The difference is that the latter implementation is easier to reason about and follow in logs.

[brijesh-elastic]

I understand your point and will update the PR. However, this will add a considerable amount of code since there are many instances where we set want_more: false

[efd6]

I'm not convinced that it does. If these two fields are put into a subobject, say interval, then at the bottom of the program you can have

.as(state, state.with(state.want_more ? {"interval": {}} : {}))

or (probably nicer)

.as(state, state.want_more ? state : state.drop("interval"))

and rename the fields to interval.id and interval.start throughout.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
89.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b0fe51b

History

• 💚 Build #26567 succeeded b73f8cf
• 💚 Build #25984 succeeded 5084944
• 💚 Build #25914 succeeded 57874e4
• 💚 Build #25861 succeeded 2c16c01
• 💚 Build #25727 succeeded 306c6a5

cc @brijesh-elastic

[elastic-vault-github-plugin-prod]

Package qualys_vmdr - 6.7.0 containing this change is available at https://epr.elastic.co/package/qualys_vmdr/6.7.0/