[New Rule] Forbidden Request from Unusual User Agent in Kubernetes

[Aegrah]

Summary

This rule detects when a forbidden request is made from an unusual user agent in a Kubernetes environment. Adversary tooling may use non-standard or unexpected user agents to interact with the Kubernetes API, which can indicate an attempt to evade detection or blend in with legitimate traffic. In combination with a forbidden request, this behavior can suggest an adversary is attempting to exploit vulnerabilities or misconfigurations in the Kubernetes cluster.

Telemetry

Other than TPs related to leveraging the Python requests module to interact with the k8s API, 0 hits in telemetry over all k8s rules last 90d. In my testing stack, only 1 hit, related to a custom script.

I will write another rule related to suspicious user agents in a white-list manner later.

[github-actions]

Rule: New - Guidelines

These guidelines serve as a reminder set of considerations when proposing a new rule.

Documentation and Context

• Detailed description of the rule.
• List any new fields required in ECS/data sources.
• Link related issues or PRs.
• Include references.

Rule Metadata Checks

• creation_date matches the date of creation PR initially merged.
• min_stack_version should support the widest stack versions.
• name and description should be descriptive and not include typos.
• query should be inclusive, not overly exclusive, considering performance for diverse environments. Non ecs fields should be added to non-ecs-schema.json if not available in an integration.
• min_stack_comments and min_stack_version should be included if the rule is only compatible starting from a specific stack version.
• index pattern should be neither too specific nor too vague, ensuring it accurately matches the relevant data stream (e.g., use logs-endpoint.process-* for process data).
• integration should align with the index. If the integration is newly introduced, ensure the manifest, schemas, and new_rule.yaml template are updated.
• setup should include the necessary steps to configure the integration.
• note should include any additional information (e.g. Triage and analysis investigation guides, timeline templates).
• tags should be relevant to the threat and align/added to the EXPECTED_RULE_TAGS in the definitions.py file.
• threat, techniques, and subtechniques should map to ATT&CK always if possible.

New BBR Rules

• building_block_type should be included if the rule is a building block and the rule should be located in the rules_building_block folder.
• bypass_bbr_timing should be included if adding custom lookback timing to the rule.

Testing and Validation

• Provide evidence of testing and detecting the expected threat.
• Check for existence of coverage to prevent duplication.

[imays11]

Do you think this rule could get too noisy on public facing K8s deployments or too easy to bypass with the user agent exclusions? Wouldn't it be a bit more suspicious for one of those known user agents to suddenly be making "forbidden" requests? As they typically would only be trying to do approved actions they were designed to do.

[terrancedejesus]

Can we scope this down to logs-kubernetes.audit_logs-*?

[imays11]

Agree here for all the K8s rules, unless you're using something other than audit logs for these?

[Aegrah]

Yep good call!

[Aegrah]

Additionally, I sharpened the indices for all existing k8s rules in 4822!

[terrancedejesus]

If not sequencing and identifying rare user agents, should we consider New Terms for this rule on user agent?

[Aegrah]

I was considering this, but now that we know that new_terms is relatively expensive in terms of compute, I wanted to go the EQL route, using currently available telemetry to tune out any non-usual ones over the last 90d.

In case this one were to remain noisy, I will move to new_terms.

[terrancedejesus]

Couple questions, but looks good!

[Aegrah]

@imays11

Do you think this rule could get too noisy on public facing K8s deployments or too easy to bypass with the user agent exclusions?

I wrote this rule based on the telemetry we have available for all k8s deployments over the last 90+ days, and with these exclusions, the only user agent that remains is the Python requests module. So I think it won't be too noisy, and if it is, I will turn it into a new_terms rule.

Wouldn't it be a bit more suspicious for one of those known user agents to suddenly be making "forbidden" requests? As they typically would only be trying to do approved actions they were designed to do.

That would be, however, you already wrote a rule that covers the default service account in Kubernetes Denied Service Account Request. One option could be to monitor all user agents, and have a new_terms hit on the decision field changing, which is something I could look into. The only issue with that rule is that forbidden requests often relate to server/permission issues as well, but I like the idea. Something like:

event.dataset:"kubernetes.audit_logs" and host.os.type:"linux" and kubernetes.audit.stage:"ResponseComplete" and user_agent.original:*

• new_terms on kubernetes.audit.annotations.authorization_k8s_io/decision, kubernetes.audit.user.username

I'm also exploring ES|QL for a large number of 403's by user agent, and unusual user agents based on statistical analysis over an environment.

Finally, I will add one additional rule to this list related to a whitelist of known bad user agents, based on pre-built reconnaissance tooling. I think we should then be good from a user agent perspective.

[imays11]

Thanks for explaining, nice work! Those are some good follow-up rule ideas