Skip to content

systemd: build: fix service startup failure when using IMDSv2 #10221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

systemd: build: fix service startup failure when using IMDSv2 #10221

wants to merge 1 commit into from

Conversation

charltonstanley
Copy link

@charltonstanley charltonstanley commented Apr 16, 2025 •
edited
Loading

PR #9845 added a service dependency sethostname.service for Amazon Linux 2, which uses Amazon's Instance Metadata Service (IMDS) to query and set the hostname. This works for instances using the legacy IMDSv1, however it fails when admins have enforced the usage of IMDSv2. This patch is a modification of @cpandya-we's comment/work on #10186. Closes #10186

Note: I used modified versions of the code samples @cosmo0920 submitted in #9845. Specifically, note the differences in the packer file, where IMDSv2 support was enabled.

Testing

  • Example configuration file for the change
[SERVICE]
    flush        1
    daemon       Off
    log_level    info
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020
    storage.metrics on

[INPUT]
    name dummy
    tag  dummy.local

    # Read interval (sec) Default: 1
    interval_sec 10

[FILTER]
    Name  record_modifier
    Match *
    Record hostname ${HOSTNAME}
    Record filter_type record_modifiler

[OUTPUT]
    name  stdout
    match *
  • [] Debug log output from testing the change

Log output confirming both fluent-bit.service and sethostname.service service start successfully.

[ec2-user@ip-172-31-10-223 ~]$ systemctl status fluent-bit -l
● fluent-bit.service - Fluent Bit
   Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2025-04-16 18:14:35 UTC; 52s ago
     Docs: https://docs.fluentbit.io/manual/
 Main PID: 3229 (fluent-bit)
   CGroup: /system.slice/fluent-bit.service
           └─3229 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [ctraces ] version=0.6.2
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [input:dummy:dummy.0] initializing
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [sp] stream processor started
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [2025/04/16 18:14:35] [ info] [output:stdout:stdout.0] worker #0 started
Apr 16 18:14:45 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827284.897177091, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:14:55 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827294.896876191, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:15:05 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827304.897044922, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:15:15 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827314.896878894, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
Apr 16 18:15:25 ip-172-31-10-223.ec2.internal fluent-bit[3229]: [0] dummy.local: [[1744827324.896980969, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-10-223.ec2.internal", "filter_type"=>"record_modifiler"}]
[ec2-user@ip-172-31-10-223 ~]$ systemctl status sethostname -l
● sethostname.service - Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support
   Loaded: loaded (/usr/lib/systemd/system/sethostname.service; disabled; vendor preset: disabled)
   Active: active (exited) since Wed 2025-04-16 18:14:35 UTC; 1min 21s ago
  Process: 3218 ExecStartPost=/bin/bash -c rm -f /run/imds_token /run/ec2_hostname (code=exited, status=0/SUCCESS)
  Process: 3212 ExecStart=/bin/bash -c /usr/bin/hostnamectl set-hostname "$(cat /run/ec2_hostname)" (code=exited, status=0/SUCCESS)
  Process: 3204 ExecStartPre=/bin/bash -c curl -s "http://169.254.169.254/latest/meta-data/hostname"    -H "X-aws-ec2-metadata-token: $(cat /run/imds_token)" > /run/ec2_hostname (code=exited, status=0/SUCCESS)
  Process: 3174 ExecStartPre=/bin/bash -c curl -sX PUT "http://169.254.169.254/latest/api/token"    -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" > /run/imds_token (code=exited, status=0/SUCCESS)
 Main PID: 3212 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/sethostname.service

Apr 16 18:14:35 ip-172-31-10-223.ec2.internal systemd[1]: Starting Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support...
Apr 16 18:14:35 ip-172-31-10-223.ec2.internal systemd[1]: Started Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support.

Here is the packer template for creating AMI. Note the addition of the the imds_support property, as well as the additional metadata_options.

packer {
  required_plugins {
    amazon = {
      version = ">= 1.2.8"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

source "amazon-ebs" "amazonlinux2" {
  ami_name      = "fbit-amazon-linux2-amd64"
  instance_type = "t3.micro"
  region        = "us-east-1"
  source_ami_filter {
    filters = {
      name                = "amzn2-ami-kernel-5.10-hvm-2.*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
      architecture        = "x86_64"
    }
    most_recent = true
    owners      = ["amazon"]
  }
  ssh_username = "ec2-user"

### Added IMDSv2 support ###
  imds_support = "v2.0"
  metadata_options {
    http_endpoint = "enabled"
    http_tokens = "required"
    http_put_response_hop_limit = 2
  }
}

build {
  name = "az2-fbit-packer"
  sources = [
    "source.amazon-ebs.amazonlinux2"
  ]

  provisioner "file" {
    source      = "/path/to/github/fluent-bit/packaging/packages/amazonlinux/2/2025-04-16-10_05_01/fluent-bit-4.0.1-1.x86_64.rpm"
    destination = "/home/ec2-user/fluent-bit.rpm"
  }

  provisioner "file" {
    source      = "path/to/fluent-bit.conf"
    destination = "/home/ec2-user/fluent-bit.conf"
  }

  provisioner "shell" {
    inline = [
      "sudo yum install -y /home/ec2-user/fluent-bit.rpm",
      "sudo mv /home/ec2-user/fluent-bit.conf /etc/fluent-bit/fluent-bit.conf",
      "sudo systemctl enable fluent-bit"
    ]
  }
}
  • [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [N/A] Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@charltonstanley charltonstanley force-pushed the charltonstanley-systemd-fix-sethostname-on-amazonlinux2 branch 2 times, most recently from cc450b4 to 08fdd4a Compare April 16, 2025 19:00
@charltonstanley
Copy link
Author

charltonstanley commented Apr 21, 2025 •
edited
Loading

Hi there, any chance this could be merged in soon? We have paused updates on our servers because we don't want to have to write an automation script to disable the fluent-bit repository on each instance. (The related bug affects ~125 instance within my org.) Thanks!

cosmo0920
cosmo0920 reviewed Apr 30, 2025
View reviewed changes
Copy link
Contributor

@cosmo0920 cosmo0920 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd confirmed that this patch works under IMDSv2.
Could you rebase off the current master?
This is because at the moment of sending this PR, our CI was broken.
So, we wanted to do CI at least once.

@cosmo0920
Copy link
Contributor

I'd confirmed that the same logs on my created EC2 instance:

[ec2-user@ip-172-31-22-3 ~]$ systemctl status fluent-bit -l
● fluent-bit.service - Fluent Bit
   Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; enabled; vendor preset: disabled)
   Active: active (running) since 水 2025-04-30 06:17:46 UTC; 1min 17s ago
     Docs: https://docs.fluentbit.io/manual/
 Main PID: 3215 (fluent-bit)
   CGroup: /system.slice/fluent-bit.service
           └─3215 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

 4月 30 06:17:46 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [2025/04/30 06:17:46] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
 4月 30 06:17:46 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [2025/04/30 06:17:46] [ info] [sp] stream processor started
 4月 30 06:17:46 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [2025/04/30 06:17:46] [ info] [output:stdout:stdout.0] worker #0 started
 4月 30 06:17:58 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993877.020040385, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:08 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993887.020059688, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:18 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993897.020056403, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:28 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993907.020051693, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:38 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993917.020051760, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:48 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993927.020056001, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
 4月 30 06:18:58 ip-172-31-22-3.ec2.internal fluent-bit[3215]: [0] dummy.local: [[1745993937.020064700, {}], {"message"=>"dummy", "hostname"=>"ip-172-31-22-3.ec2.internal", "filter_type"=>"record_modifiler"}]
[ec2-user@ip-172-31-22-3 ~]$ systemctl status sethostname -l
● sethostname.service - Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support
   Loaded: loaded (/usr/lib/systemd/system/sethostname.service; disabled; vendor preset: disabled)
   Active: active (exited) since 水 2025-04-30 06:17:46 UTC; 5min ago
  Process: 3201 ExecStartPost=/bin/sh -c rm -f /run/imds_token /run/ec2_hostname (code=exited, status=0/SUCCESS)
  Process: 3192 ExecStart=/bin/sh -c /usr/bin/hostnamectl set-hostname "$(cat /run/ec2_hostname)" (code=exited, status=0/SUCCESS)
  Process: 3181 ExecStartPre=/bin/sh -c curl -s "http://169.254.169.254/latest/meta-data/hostname"    -H "X-aws-ec2-metadata-token: $(cat /run/imds_token)" > /run/ec2_hostname (code=exited, status=0/SUCCESS)
  Process: 3153 ExecStartPre=/bin/sh -c curl -sX PUT "http://169.254.169.254/latest/api/token"    -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" > /run/imds_token (code=exited, status=0/SUCCESS)
 Main PID: 3192 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/sethostname.service

 4月 30 06:17:46 ip-172-31-22-3.ec2.internal systemd[1]: Starting Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support...
 4月 30 06:17:46 ip-172-31-22-3.ec2.internal systemd[1]: Started Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support.

@charltonstanley charltonstanley force-pushed the charltonstanley-systemd-fix-sethostname-on-amazonlinux2 branch from d11d346 to 08fdd4a Compare April 30, 2025 14:17
@charltonstanley
systemd: build: fix service startup failure when using IMDSv2
5d83094 
PR fluent#9845 added a service dependency "sethostname.service" for Amazon Linux 2,
which uses Amazon's Instance Metadata Service (IMDS) to query and set the
hostname. This works for instances using the legacy IMDSv1, however it fails
when admins have enforced the usage of IMDSv2. This patch is a modification of
@cpandya-we comment/work on fluent#10186. Fixes fluent#10186

Signed-off-by: Charlton Stanley <[email protected]>
@charltonstanley charltonstanley force-pushed the charltonstanley-systemd-fix-sethostname-on-amazonlinux2 branch from 08fdd4a to 5d83094 Compare April 30, 2025 14:24
@charltonstanley
Copy link
Author

charltonstanley commented Apr 30, 2025 •
edited
Loading

I'd confirmed that this patch works under IMDSv2.
Could you rebase off the current master?
This is because at the moment of sending this PR, our CI was broken.
So, we wanted to do CI at least once.

@cosmo0920 Thanks for verifying my patch. Rebase has been done!

@cosmo0920 cosmo0920 added this to the Fluent Bit v4.0.2 milestone May 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cosmo0920 cosmo0920 cosmo0920 left review comments

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich leonardo-albertovich is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
docs-required
Projects
None yet
Milestone
Fluent Bit v4.0.2
Development

Successfully merging this pull request may close these issues.

Amazon Linux 2 - fluent-bit.service fails to start with error "Dependency failed for Fluent Bit."
2 participants
@charltonstanley @cosmo0920