{o365.audit},{m365_defender.alert}: Enhance field mappings #12888

kcreddy · 2025-02-25T12:57:40Z

Proposed commit message

Extract `email.*` fields from `m365_defender.alert` data-stream.
Also derive `source.ip` from `evidence.senderIp`

Extract `email.*`, `related.*`, `user.*` fields from `Data` and `Data.Entities` fields inside 
`o365.audit` data-stream.
Other enhancements to `o365.audit`:
- Make `AttachmentData` and `AuthDetails` as `nested` to make independent queries.
- Add several top-level `o365.audit` fields based on sample events.
- Add drop-null processor.
- Handle error.message at end of the pipeline.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Pipeline tests passes containing new fields:
1. M365 Defender

--- Test results for package: m365_defender - START ---
╭───────────────┬─────────────┬───────────┬────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM │ TEST TYPE │ TEST NAME                                                      │ RESULT │ TIME ELAPSED │
├───────────────┼─────────────┼───────────┼────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ m365_defender │ alert       │ pipeline  │ (ingest pipeline warnings test-alert.log)                      │ PASS   │  348.92925ms │
│ m365_defender │ alert       │ pipeline  │ test-alert.log                                                 │ PASS   │  99.644209ms │
│ m365_defender │ event       │ pipeline  │ (ingest pipeline warnings test-alert.log)                      │ PASS   │ 376.436666ms │
│ m365_defender │ event       │ pipeline  │ (ingest pipeline warnings test-app-and-identity.log)           │ PASS   │ 342.705083ms │
│ m365_defender │ event       │ pipeline  │ (ingest pipeline warnings test-device.log)                     │ PASS   │ 364.856291ms │
│ m365_defender │ event       │ pipeline  │ (ingest pipeline warnings test-email.log)                      │ PASS   │ 327.063792ms │
│ m365_defender │ event       │ pipeline  │ test-alert.log                                                 │ PASS   │ 139.360208ms │
│ m365_defender │ event       │ pipeline  │ test-app-and-identity.log                                      │ PASS   │    132.109ms │
│ m365_defender │ event       │ pipeline  │ test-device.log                                                │ PASS   │  1.29156225s │
│ m365_defender │ event       │ pipeline  │ test-email.log                                                 │ PASS   │   95.93325ms │
│ m365_defender │ incident    │ pipeline  │ (ingest pipeline warnings test-incident.log)                   │ PASS   │ 357.501791ms │
│ m365_defender │ incident    │ pipeline  │ test-incident.log                                              │ PASS   │ 274.046417ms │
│ m365_defender │ log         │ pipeline  │ (ingest pipeline warnings test-m365-defender-empty-ndjson.log) │ PASS   │  322.74475ms │
│ m365_defender │ log         │ pipeline  │ (ingest pipeline warnings test-m365-defender-ndjson.log)       │ PASS   │  371.32925ms │
│ m365_defender │ log         │ pipeline  │ test-m365-defender-empty-ndjson.log                            │ PASS   │  40.019459ms │
│ m365_defender │ log         │ pipeline  │ test-m365-defender-ndjson.log                                  │ PASS   │ 159.439917ms │
╰───────────────┴─────────────┴───────────┴────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: m365_defender - END   ---
Done

2. O365
New files added containing new fields:

test-wl-airinvestigation.json
test-wl-securitycompliancecenter.json
test-wl-threatintelligence.json

--- Test results for package: o365 - START ---
╭─────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                             │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-azuread-events.json)                   │ PASS   │ 358.723375ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-azuread-sts-logon-events.json)         │ PASS   │ 384.870416ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-bad-ips.json)                          │ PASS   │ 385.783834ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-data-duplicated-querytime-events.json) │ PASS   │ 324.737417ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-data-insights-api-events.json)         │ PASS   │ 405.747208ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-dlp-exchange-events.json)              │ PASS   │ 387.674084ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-dlp-sharepoint-events.json)            │ PASS   │ 381.929792ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-exchange-admin-events.json)            │ PASS   │ 381.202584ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-exchange-item-events.json)             │ PASS   │ 398.749625ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-ip-formats-events.json)                │ PASS   │ 357.165834ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-modified-properites.json)              │ PASS   │ 356.582209ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-ms-teams-events.json)                  │ PASS   │ 356.578584ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-parameter-string.json)                 │ PASS   │    456.804ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-platform-attribute-events.json)        │ PASS   │   361.8045ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-sec-comp-alerts-events.json)           │ PASS   │  370.36875ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-sharepoint-events.json)                │ PASS   │ 367.199917ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-sharepointfileop-events.json)          │ PASS   │ 372.218375ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-sp-sharing-op-events.json)             │ PASS   │ 383.053708ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-stringly-json-events.json)             │ PASS   │ 413.074083ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-wl-airinvestigation.json)              │ PASS   │ 351.694333ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-wl-securitycompliancecenter.json)      │ PASS   │ 368.423084ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-wl-threatintelligence.json)            │ PASS   │ 356.965541ms │
│ o365    │ audit       │ pipeline  │ (ingest pipeline warnings test-yammer-events.json)                    │ PASS   │ 342.123791ms │
│ o365    │ audit       │ pipeline  │ test-azuread-events.json                                              │ PASS   │ 3.071293167s │
│ o365    │ audit       │ pipeline  │ test-azuread-sts-logon-events.json                                    │ PASS   │  1.98057625s │
│ o365    │ audit       │ pipeline  │ test-bad-ips.json                                                     │ PASS   │  77.807208ms │
│ o365    │ audit       │ pipeline  │ test-data-duplicated-querytime-events.json                            │ PASS   │  51.384292ms │
│ o365    │ audit       │ pipeline  │ test-data-insights-api-events.json                                    │ PASS   │  99.803625ms │
│ o365    │ audit       │ pipeline  │ test-dlp-exchange-events.json                                         │ PASS   │ 124.631375ms │
│ o365    │ audit       │ pipeline  │ test-dlp-sharepoint-events.json                                       │ PASS   │ 130.957167ms │
│ o365    │ audit       │ pipeline  │ test-exchange-admin-events.json                                       │ PASS   │    1.117815s │
│ o365    │ audit       │ pipeline  │ test-exchange-item-events.json                                        │ PASS   │  159.23625ms │
│ o365    │ audit       │ pipeline  │ test-ip-formats-events.json                                           │ PASS   │  118.68825ms │
│ o365    │ audit       │ pipeline  │ test-modified-properites.json                                         │ PASS   │  59.363083ms │
│ o365    │ audit       │ pipeline  │ test-ms-teams-events.json                                             │ PASS   │  77.816958ms │
│ o365    │ audit       │ pipeline  │ test-parameter-string.json                                            │ PASS   │    71.5595ms │
│ o365    │ audit       │ pipeline  │ test-platform-attribute-events.json                                   │ PASS   │   45.77825ms │
│ o365    │ audit       │ pipeline  │ test-sec-comp-alerts-events.json                                      │ PASS   │  75.670084ms │
│ o365    │ audit       │ pipeline  │ test-sharepoint-events.json                                           │ PASS   │ 133.682083ms │
│ o365    │ audit       │ pipeline  │ test-sharepointfileop-events.json                                     │ PASS   │ 342.594334ms │
│ o365    │ audit       │ pipeline  │ test-sp-sharing-op-events.json                                        │ PASS   │ 232.921417ms │
│ o365    │ audit       │ pipeline  │ test-stringly-json-events.json                                        │ PASS   │  96.670833ms │
│ o365    │ audit       │ pipeline  │ test-wl-airinvestigation.json                                         │ PASS   │  90.202166ms │
│ o365    │ audit       │ pipeline  │ test-wl-securitycompliancecenter.json                                 │ PASS   │ 124.492833ms │
│ o365    │ audit       │ pipeline  │ test-wl-threatintelligence.json                                       │ PASS   │  94.889416ms │
│ o365    │ audit       │ pipeline  │ test-yammer-events.json                                               │ PASS   │  68.542625ms │
╰─────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: o365 - END   ---
Done

Related issues

Closes [m365_defender] [o365] Add new fields to o365 and M365 Defender integrations #12519

…365-m365_defender-new-fields

elastic-vault-github-plugin-prod · 2025-02-25T13:22:15Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6 · 2025-03-03T04:01:29Z

packages/o365/changelog.yml

+  changes:
+    - description: Extrac ECS fields from Data and AttachmentData.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/1


Suggested change

link: https://github.com/elastic/integrations/pull/1

link: https://github.com/elastic/integrations/pull/12888

This keeps happening, I will follow your epb alias from now.

efd6 · 2025-03-03T04:02:12Z

packages/o365/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.10.0"
+  changes:
+    - description: Extrac ECS fields from Data and AttachmentData.


Suggested change

- description: Extrac ECS fields from Data and AttachmentData.

- description: Extract ECS fields from Data and AttachmentData.

efd6 · 2025-03-03T04:09:22Z

packages/o365/data_stream/audit/fields/fields.yml

+    - name: Description
+      type: keyword


match_only_text?

Sadly this is not possible in the m365 case where the text is much longer.

packages/o365/data_stream/audit/_dev/test/pipeline/test-wl-airinvestigation.json

packages/o365/data_stream/audit/_dev/test/pipeline/test-wl-airinvestigation.json-expected.json

efd6 · 2025-03-03T04:24:13Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

+      field: o365audit.Parameters._raw
+      if: ctx.o365audit?.NetworkMessageId == null || ctx.o365audit.NetworkMessageId == ''
+      patterns: 
+        - '^(-)?Identity\s(\")?%{DATA:o365audit.NetworkMessageId}(\")?$'


Suggested change

- '^(-)?Identity\s(\")?%{DATA:o365audit.NetworkMessageId}(\")?$'

- '^-?Identity\s"?%{DATA:o365audit.NetworkMessageId}"?$'

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

efd6

LGTM after the doc vet passes.

packages/o365/data_stream/audit/fields/fields.yml

elasticmachine · 2025-03-03T06:18:29Z

💚 Build Succeeded

Buildkite Build
Commit: 6962d89

History

💔 Build #22888 failed d8da7c7
💚 Build #22842 succeeded 954d095
💚 Build #22724 succeeded 0a5b594

cc @kcreddy

elastic-sonarqube · 2025-03-03T06:18:34Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-03-03T06:35:26Z

Package m365_defender - 2.23.0 containing this change is available at https://epr.elastic.co/package/m365_defender/2.23.0/

elastic-vault-github-plugin-prod · 2025-03-03T06:35:27Z

Package o365 - 2.10.0 containing this change is available at https://epr.elastic.co/package/o365/2.10.0/

kcreddy added 3 commits February 25, 2025 18:25

Add email ECS fields to alert data stream.

576fe64

Merge branch 'main' of https://github.com/elastic/integrations into o…

584243d

…365-m365_defender-new-fields

update pr number

0a5b594

kcreddy self-assigned this Feb 25, 2025

kcreddy added Integration:o365 Microsoft Office 365 Integration:m365_defender Microsoft M365 Defender enhancement New feature or request labels Feb 25, 2025

kcreddy added 2 commits February 27, 2025 21:37

Add fields for o365

cd69a57

Add remaining fields and handle lists

954d095

kcreddy marked this pull request as ready for review February 28, 2025 11:39

kcreddy requested a review from a team as a code owner February 28, 2025 11:39

kcreddy marked this pull request as draft February 28, 2025 11:40

kcreddy marked this pull request as ready for review February 28, 2025 11:45

efd6 reviewed Mar 3, 2025

View reviewed changes

Address PR comments

d8da7c7

kcreddy requested a review from efd6 March 3, 2025 05:42

efd6 approved these changes Mar 3, 2025

View reviewed changes

packages/o365/data_stream/audit/fields/fields.yml Show resolved Hide resolved

update readme

6962d89

kcreddy merged commit acecb4d into elastic:main Mar 3, 2025
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{o365.audit},{m365_defender.alert}: Enhance field mappings #12888

{o365.audit},{m365_defender.alert}: Enhance field mappings #12888

kcreddy commented Feb 25, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Feb 25, 2025 •

edited

Loading

efd6 Mar 3, 2025

kcreddy Mar 3, 2025

efd6 Mar 3, 2025

efd6 Mar 3, 2025

efd6 Mar 3, 2025

efd6 left a comment

elasticmachine commented Mar 3, 2025

elastic-sonarqube bot commented Mar 3, 2025

elastic-vault-github-plugin-prod bot commented Mar 3, 2025

elastic-vault-github-plugin-prod bot commented Mar 3, 2025

	link: https://github.com/elastic/integrations/pull/1
	link: https://github.com/elastic/integrations/pull/12888

	- description: Extrac ECS fields from Data and AttachmentData.
	- description: Extract ECS fields from Data and AttachmentData.

	- '^(-)?Identity\s(\")?%{DATA:o365audit.NetworkMessageId}(\")?$'
	- '^-?Identity\s"?%{DATA:o365audit.NetworkMessageId}"?$'

{o365.audit},{m365_defender.alert}: Enhance field mappings #12888

{o365.audit},{m365_defender.alert}: Enhance field mappings #12888

Conversation

kcreddy commented Feb 25, 2025 • edited Loading

Proposed commit message

Checklist

How to test this PR locally

Related issues

elastic-vault-github-plugin-prod bot commented Feb 25, 2025 • edited Loading

🚀 Benchmarks report

efd6 Mar 3, 2025

Choose a reason for hiding this comment

kcreddy Mar 3, 2025

Choose a reason for hiding this comment

efd6 Mar 3, 2025

Choose a reason for hiding this comment

efd6 Mar 3, 2025

Choose a reason for hiding this comment

efd6 Mar 3, 2025

Choose a reason for hiding this comment

efd6 left a comment

Choose a reason for hiding this comment

elasticmachine commented Mar 3, 2025

💚 Build Succeeded

History

elastic-sonarqube bot commented Mar 3, 2025

Quality Gate passed

elastic-vault-github-plugin-prod bot commented Mar 3, 2025

elastic-vault-github-plugin-prod bot commented Mar 3, 2025

kcreddy commented Feb 25, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Feb 25, 2025 •

edited

Loading