crowdstrike.fdr: Parse additional notification payloads using parsing script.

[kcreddy]

Proposed commit message

crowdstrike.fdr: Update script to parse all notification payloads.

Current parsing script works well when SQS notification payload 
matches FDR queue notification format. AWS S3 input supports 
several other ingestion modes such as via generic SQS, SNS, and 
EventBridge. While current script can still lead to successful 
ingestion in some cases such as via generic SQS and via SNS, 
it fails via EventBridge. It also depends on "Is FDR queue: false" 
to succeed in those few cases. 

This PR removes the dependency on users choosing "Is FDR queue" 
option. It removes the option and updates the parsing script to check
various notification formats and infer the ingestion mode.The logic is 
similar to beats AWS S3 input[1].

[1]: https://github.com/elastic/beats/blob/main/x-pack/filebeat/input/awss3/sqs_s3_event.go#L308-L334

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Tested documented in #13065 (comment)

Related issues

• Closes [Crowdstrike FDR]Handle All SQS Notification Payloads #13065

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

Can the semantic equivalent of this in the context of the net code be added to the documentation for fdr_parsing_script?

[kcreddy]

Updated in 2ae8e7c. Let me know if this updated description of parsing script makes sense.

[kcreddy]

Also, do you think removing this option is a breaking-change for the users? it has required: true before, but our script is capable of parsing custom notifications too, although I haven't tested due to lack of FDR queue setup.

[efd6]

What circumstances could break users here? If they are following the documented behaviour and using this to indicate that the data is coming from an FDR queue, and our parsing script is able to process this without having this exist — which is the case. The issue arises when they are saying that it is not an FDR source and somehow the data is acceptable without any processing, being broken when there is some mutation applied to the data. I can't think of any case where that would be what is going on, but I guess it might.

[kcreddy]

What circumstances could break users here?

Yeah I don't think so too, but just wanted to confirm as well. We have handled all the cases that AWS S3 input can along with FDR format.

[efd6]

Maybe we could leave the door open to a work-around by making this not required and have the inclusion in the template dependent on it existing, with a note in the documentation that it should only be left empty when the user knows what they are doing. Maybe wait for @andrewkroh to chime in?

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 56cd3cf

History

• 💚 Build #25909 succeeded 91587aa
• 💔 Build #25906 failed 2392773
• 💚 Build #25814 succeeded 2150b43
• 💚 Build #25813 succeeded a3e6e65