zscaler_zia: add strict field template mode for tcp and http_endpoint input data streams #13904

efd6 · 2025-05-14T04:53:25Z

Proposed commit message

zscaler_zia: add strict field template mode for tcp and http_endpoint input data streams

This checks that the template version matches the current expected
template version if the template specifies it, falling back to an
assessment of the fields that are present in the message. The check is
only performed when enabled, which is not the case by default.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [zscale_zia] Validate response format versions #10743

Screenshots

elastic-vault-github-plugin-prod · 2025-05-14T05:37:19Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2025-05-14T06:08:26Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

… input data streams This checks that the template version matches the current expected template version if the template specifies it, falling back to an assessment of the fields that are present in the message. The check is only performed when enabled, which is not the case by default.

packages/zscaler_zia/data_stream/web/elasticsearch/ingest_pipeline/default.yml

Needs syntax to link directly to the noted version.

kcreddy · 2025-05-21T08:13:10Z

packages/zscaler_zia/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml

@@ -43,6 +43,32 @@ processors:
      tag: rename_resp_event
      target_field: json
      ignore_missing: true
+  - script:
+      params:
+        pkg_version: 3.12.0


Looks like we are not using this pkg_version param anywhere. Can w remove it?

Waiting for feedback from docs for how I can link directly to the correct version; it will be used then, that's why this commit is a wip. Without a direct link to the correct version, I'd rather not link at all.

Got it. My bad, I didn't expand the commit.

I have confirmation that this is not possible. The new code links to the markdown. It's not friendly, but it is at least correct. I would prefer correctness over friendliness, but happy to hear your position.

Got it. Yeah correctness makes much more sense. I am okay with this.
Its probably going to be slightly inconvenient to maintain this params.pkg_version flag as it needs to be bumped every PR on zscaler_zia.
Do you think we should document such exceptions into our wiki?

I think at least adding a comment above the manifest.yml version field. Would that do for you?

Yeah that works 👍🏼

packages/zscaler_zia/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

Still wip; the links are to raw MD and are not direct to the affected data stream.

chrisberkhout

I think this is good.

I noted an idea about getting the package version for the error message.

I would prefer strict fields or something like it to always be on. It doesn't fail all the processing, it just adds an error message. That may cause some level of disruption, but if it's an advanced option that's off by default, I think we'll almost never get this information when it matters.
My preference would be to set the error message by default, and have an option to skip it or downgrade it to a warning (although I'm not sure we have a great ECS field for warnings).

packages/zscaler_zia/_dev/build/docs/README.md

packages/zscaler_zia/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

elasticmachine · 2025-05-23T00:30:46Z

💚 Build Succeeded

Buildkite Build
Commit: 192b7a6

History

💔 Build #26272 failed a40b8d8
💚 Build #26234 succeeded d004c9a
💚 Build #26227 succeeded 451811b
💚 Build #26221 succeeded 4a48264
💚 Build #26215 succeeded 17f9c0d

cc @efd6

elastic-sonarqube · 2025-05-23T00:30:48Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

chrisberkhout

Thanks!

elastic-vault-github-plugin-prod · 2025-05-25T21:36:58Z

Package zscaler_zia - 3.12.0 containing this change is available at https://epr.elastic.co/package/zscaler_zia/3.12.0/

* main: (42 commits) [jamf_pro] Fix `flattened` field types for non-object values (elastic#13985) [Netskope Alerts] Add text multi-field to netskope.alerts.breach.description field (elastic#13977) zscaler_zia: add strict field template mode for tcp and http_endpoint input data streams (elastic#13904) apm: Add config for tail-based sampling discard on write (elastic#13950) [CI] Add dev/coverage into backport script (elastic#13987) Update configuration updatecli for 8.x snapshot (elastic#13981) [Prometheus] Add username, password, and SSL related fields for query dataset (elastic#13969) o365: Ignore failures in rename processors for organization fields (elastic#13983) aws.firewall: Document ingested log types of AWS Network Firewall (elastic#13978) mimecast: resolve field data type conflicts between data streams (elastic#13825) [Infoblox NIOS] Handle the parsing of IPv6 address (elastic#13947) [Cribl] Fix handling of metric event type (elastic#13930) zscaler_zpa: fix handling of multiple remote IPs, and event categorisation (elastic#13755) Adding agentless deployment to the sublime security integration (elastic#13963) [integration/system] add use_performance_counters in system integration (elastic#13150) crowdstrike,m365_defender,microsoft_defender_{cloud,endpoint},sentinel_one: normalise severity handling (elastic#13955) [forgerock] Map `forgerock.response.elapsedTime` as a long not a date (elastic#13959) github: squelch errors from pagination ends (elastic#13965) cisco_secure_endpoint: squelch errors from pagination ends (elastic#13964) [Cloud Security] Cloud Asset Inventory: fixed cloud formation URL (elastic#13971) ...

* feature/use-google-secrets: (43 commits) use -ci account [jamf_pro] Fix `flattened` field types for non-object values (#13985) [Netskope Alerts] Add text multi-field to netskope.alerts.breach.description field (#13977) zscaler_zia: add strict field template mode for tcp and http_endpoint input data streams (#13904) apm: Add config for tail-based sampling discard on write (#13950) [CI] Add dev/coverage into backport script (#13987) Update configuration updatecli for 8.x snapshot (#13981) [Prometheus] Add username, password, and SSL related fields for query dataset (#13969) o365: Ignore failures in rename processors for organization fields (#13983) aws.firewall: Document ingested log types of AWS Network Firewall (#13978) mimecast: resolve field data type conflicts between data streams (#13825) [Infoblox NIOS] Handle the parsing of IPv6 address (#13947) [Cribl] Fix handling of metric event type (#13930) zscaler_zpa: fix handling of multiple remote IPs, and event categorisation (#13755) Adding agentless deployment to the sublime security integration (#13963) [integration/system] add use_performance_counters in system integration (#13150) crowdstrike,m365_defender,microsoft_defender_{cloud,endpoint},sentinel_one: normalise severity handling (#13955) [forgerock] Map `forgerock.response.elapsedTime` as a long not a date (#13959) github: squelch errors from pagination ends (#13965) cisco_secure_endpoint: squelch errors from pagination ends (#13964) ...

efd6 self-assigned this May 14, 2025

efd6 added enhancement New feature or request Integration:zscaler_zia Zscaler Internet Access Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels May 14, 2025

efd6 force-pushed the 10743-zscaler_zia branch 2 times, most recently from eef8f0c to 8b4f86f Compare May 14, 2025 04:58

efd6 marked this pull request as ready for review May 14, 2025 06:08

efd6 requested a review from a team as a code owner May 14, 2025 06:08

efd6 force-pushed the 10743-zscaler_zia branch from 8b4f86f to 2125a6e Compare May 14, 2025 22:43

kcreddy reviewed May 20, 2025

View reviewed changes

packages/zscaler_zia/data_stream/web/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

packages/zscaler_zia/data_stream/web/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

efd6 added 2 commits May 21, 2025 08:07

address pr comment - wip

05c57c7

Needs syntax to link directly to the noted version.

update expectations

9e9b376

efd6 force-pushed the 10743-zscaler_zia branch from e78fee3 to 9e9b376 Compare May 20, 2025 22:38

kcreddy reviewed May 21, 2025

View reviewed changes

efd6 force-pushed the 10743-zscaler_zia branch from d83f6e0 to 17f9c0d Compare May 22, 2025 02:10

efd6 commented May 22, 2025

View reviewed changes

packages/zscaler_zia/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

add versioned links

4a48264

Still wip; the links are to raw MD and are not direct to the affected data stream.

efd6 force-pushed the 10743-zscaler_zia branch from 17f9c0d to 4a48264 Compare May 22, 2025 04:30

add protective comment/tags and use data_stream param

451811b

efd6 requested a review from kcreddy May 22, 2025 05:35

kcreddy approved these changes May 22, 2025

View reviewed changes

efd6 requested a review from chrisberkhout May 22, 2025 05:36

chrisberkhout requested changes May 22, 2025

View reviewed changes

packages/zscaler_zia/_dev/build/docs/README.md Outdated Show resolved Hide resolved

packages/zscaler_zia/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

efd6 requested a review from chrisberkhout May 22, 2025 08:51

address pr comment

d004c9a

efd6 force-pushed the 10743-zscaler_zia branch from ed8bdba to d004c9a Compare May 22, 2025 08:51

make strict fields on by default

a40b8d8

disable strict fields for system tests

192b7a6

chrisberkhout approved these changes May 23, 2025

View reviewed changes

efd6 merged commit e9063eb into elastic:main May 25, 2025
8 checks passed

zscaler_zia: add strict field template mode for tcp and http_endpoint input data streams #13904

zscaler_zia: add strict field template mode for tcp and http_endpoint input data streams #13904

Uh oh!

Conversation

efd6 commented May 14, 2025

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elastic-vault-github-plugin-prod bot commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

elasticmachine commented May 14, 2025

Uh oh!

Uh oh!

Uh oh!

kcreddy May 21, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 May 21, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy May 21, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 May 22, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy May 22, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 May 22, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy May 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chrisberkhout left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented May 23, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented May 23, 2025

Quality Gate passed

Uh oh!

chrisberkhout left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented May 25, 2025

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented May 14, 2025 •

edited

Loading