[Crowdstrike FDR]Handle All SQS Notification Payloads

[narph]

The FDR data stream expects SQS notifications in the custom format used by Crowdstrike, as described in this example. This is different from the standard notification format used by AWS for S3 ObjectCreated notifications, as documented in this guide.

In this deployment, the user was managing Crowdstrike data in their own bucket and had configured S3 object notifications to be sent to SNS. They then tee'd these notifications to two separate SQS queues to feed both their production and proof-of-concept test environments with Elastic. As a result, they couldn't directly use the Crowdstrike integration because it couldn't accept the SQS notification format.

To work around this limitation, they used the Custom AWS Logs integration and routed the data into the FDR data stream and pipeline.

Ideas

• Make the FDR integration automatically accept all notification formats (S3 to SQS, S3 via SNS, EventBridge, Crowdstrike).

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[kcreddy]

I'm working on S3 -> SQS case first.

Setup:

1. Created S3, SQS with permissions.
2. Enabled S3 ObjectCreated event notifications to SQS.
3. Upload 3 files from system tests into S3 bucket i.e., 1 aidmaster file, 1 userinfo file, and 1 sample logs file.
4. Verify SQS messages.

As seen in above screenshot, 3 SQS messages were received, one each for an ObjectCreated event.
Following are the messages itself (redacted):

09fcec14-425a-41a6-b450-8a937697c760

{
    "Records": [
        {
            "eventVersion": "2.1",
            "eventSource": "aws:s3",
            "awsRegion": "us-east-1",
            "eventTime": "2025-03-26T19:02:38.014Z",
            "eventName": "ObjectCreated:Put",
            "userIdentity": {
                "principalId": "AWS:AAAAAAAAAAAA"
            },
            "requestParameters": {
                "sourceIPAddress": "1.1.1.1"
            },
            "responseElements": {
                "x-amz-request-id": "ASASASASQSQSQSQS",
                "x-amz-id-2": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
            },
            "s3": {
                "s3SchemaVersion": "1.0",
                "configurationId": "crowdstrike-test-notif",
                "bucket": {
                    "name": "aaaaaaaaaaa-test-s3",
                    "ownerIdentity": {
                        "principalId": "AAAAAAAAAAAA"
                    },
                    "arn": "arn:aws:s3:::aaaaaaaaaaa-test-s3"
                },
                "object": {
                    "key": "data/fdr-0_aidmaster.log",
                    "size": 4414,
                    "eTag": "aaaaaaaaaaa",
                    "sequencer": "1111111111111"
                }
            }
        }
    ]
}

8a35e77c-2b92-4d99-9085-04b33e5d6475

{
    "Records": [
        {
            "eventVersion": "2.1",
            "eventSource": "aws:s3",
            "awsRegion": "us-east-1",
            "eventTime": "2025-03-26T19:02:42.169Z",
            "eventName": "ObjectCreated:Put",
            "userIdentity": {
                "principalId": "AWS:AAAAAAAAAAAA"
            },
            "requestParameters": {
                "sourceIPAddress": "11.1.1.1"
            },
            "responseElements": {
                "x-amz-request-id": "aaaaaaaaaa",
                "x-amz-id-2": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
            },
            "s3": {
                "s3SchemaVersion": "1.0",
                "configurationId": "crowdstrike-test-notif",
                "bucket": {
                    "name": "aaaaaaaaatest-s3",
                    "ownerIdentity": {
                        "principalId": "AAAAAAAAAAAA"
                    },
                    "arn": "arn:aws:s3:::aaaaaaaaatest-s3"
                },
                "object": {
                    "key": "data/fdr-sample.log",
                    "size": 114782,
                    "eTag": "aaaaaaa",
                    "sequencer": "11111111"
                }
            }
        }
    ]
}

06397535-d8fe-414d-b657-cd90564b547c

{
    "Records": [
        {
            "eventVersion": "2.1",
            "eventSource": "aws:s3",
            "awsRegion": "us-east-1",
            "eventTime": "2025-03-26T19:02:38.799Z",
            "eventName": "ObjectCreated:Put",
            "userIdentity": {
                "principalId": "AWS:AAAAAAAAAAAA"
            },
            "requestParameters": {
                "sourceIPAddress": "1.1.1.1"
            },
            "responseElements": {
                "x-amz-request-id": "aaaaaaaaa",
                "x-amz-id-2": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
            },
            "s3": {
                "s3SchemaVersion": "1.0",
                "configurationId": "crowdstrike-test-notif",
                "bucket": {
                    "name": "aaaaaaaaa-test-s3",
                    "ownerIdentity": {
                        "principalId": "AAAAAAAAAAAA"
                    },
                    "arn": "arn:aws:s3:::aaaaaaaaa-test-s3"
                },
                "object": {
                    "key": "data/fdr-0_userinfo.log",
                    "size": 489,
                    "eTag": "aaaaaaaaa",
                    "sequencer": "1111111"
                }
            }
        }
    ]
}

The process created 3 separate SQS messages (one each for aidmaster, userinfo metadata, and sample log), and each SQS message itself has a single S3 event notification containing single-element Records array. The FDR parsing script is meant to sort a particular S3 notification event when it contains multiple Records-like array and aims to prioritise reading aidmaster and userinfo metadata files present inside same notification. But as can be seen in this test, we get them in separate S3 notifications containing single-element Records array.

I believe if we really want to prioritise reading aidmaster and userinfo files first, what we should be sorting is not a S3 notification, but the ordering of SQS messages itself.
I discussed with @efd6 on this and AWS S3 input doesn't seem to provide a way to order the SQS messages itself and there is no way for us to read aidmaster and userinfo files first.

@andrewkroh, may I know your thoughts on this and if we can still achieve the requirement of this issue?

[andrewkroh]

The existing FDR parsing script handles the notification format delivered by CS directly to SQS. This format does "batch" multiple changes into a single SQS message.

But for notifications generated by S3 directly, which aren’t guaranteed¹ to arrive in the same order that the events occurred, there nothing reasonable we can do reorder these IMO. We could sort the .Records[] values in case a notification contained more than one, but currently S3 only ever sends one record per notification² so that would add nothing. So it's possible that you have slightly stale metadata cache when processing some events.

Despite the ordering limitations of s3 object notifications, the main goal is to have the integration be able to automatically process the notification format that it gets from SQS whether it be the CS format or one of the native AWS formats, thus removing something that the user needs to be aware of in order to deploy the integration properly and increasing their chances of success.

Footnotes

1. https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#event-ordering-and-duplicate-events ↩

2. https://repost.aws/questions/QUzbHHiTa4TF2gpTJD8I0vdQ/do-s3-objectcreated-put-event-notidfications-always-contain-a-single-record ↩

[kcreddy]

Make the FDR integration automatically accept all notification formats (S3 to SQS, S3 via SNS, EventBridge, Crowdstrike).

@andrewkroh, I've tested various scenarios using our existing system test samples by manually them to S3. Following are my observations:

Current Behaviour:

1. S3 to SQS:
   • When the option Is FDR queue: false, all the sample events are successfully ingested.
   • When the option Is FDR queue: true, no events are ingested. This is going to be same for rest of the cases as well whenever Is FDR queue: true. This is because current implementation returns empty events when it doesn't match FDR queue notification format.
   • S3-SQS-current.zip has logs for both cases.
2. S3 via SNS
   • When the option Is FDR queue: false, all the sample events are successfully ingested.
   • S3-SNS-SQS-current.zip has logs.
3. S3 via EventBridge
   • When the option Is FDR queue: false, no events ingested. It fails with error Invalid SQS message body: missing Records field. This is surprising because when I copy the sqs_message_body from the agent log and run it past the beats unit test, the test passes. Not sure why the integration couldn't read this EventBridge format but beats input can.
   • S3-EventBridge-SQS-current.zip has logs.

In #13875, I worked on updating the parsing script by removing the dependency on Is FDR queue option. Irrespective of what the users set as their source (S3 to SQS, S3 via SNS, EventBridge, Crowdstrike FDR queue), it should lead to successful ingestion.

Updated Behaviour: (from PR)

1. S3 to SQS
   • All the sample events are successfully ingested.
   • S3-SQS-updated.zip has logs.
2. S3 via SNS
   • All the sample events are successfully ingested.
   • S3-SNS-SQS-updated.zip has logs.
3. S3 via EventBridge
   • All the sample events are successfully ingested.
   • S3-Eventbridge-SQS-updated.zip has logs.