Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add opt-in metrics to CloudWatch agent #988

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Add opt-in default metrics to CloudWatch agent
Report on memory and disk usage
  • Loading branch information
ouranos committed Sep 29, 2022
commit a1e83f64e9f8be65cbe3a1756ad75b94019974cf
24 changes: 24 additions & 0 deletions packer/linux/conf/bin/bk-configure-cloudwatch-agent.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/bin/bash
# shellcheck disable=SC2094
set -euo pipefail

if [[ "${CLOUDWATCH_ENABLE_METRICS:-false}" == "true" ]]; then
cw_config="/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json"
cat <<<"$(jq \
--arg queue "$BUILDKITE_QUEUE" \
'. + {
metrics: {
metrics_collected: {
mem: {measurement: ["mem_used_percent"], append_dimensions: {BuildkiteQueue: $queue}},
disk: {measurement: ["used_percent"], resources: ["*"], append_dimensions: {BuildkiteQueue: $queue}}
Comment on lines +12 to +13
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we're adding the queue as dimension for memory and disk, should we do it for CPU too?

},
append_dimensions: {
AutoScalingGroupName: "${aws:AutoScalingGroupName}"
}
}
}' $cw_config)" >$cw_config
Comment on lines +6 to +19
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it actually rewrites it. It's the same pattern used to configure docker, albeit more complicated.

  • The jq command will read the $cw_config file, append the metrics configuration block and output it (jq [options] <jq filter> [file...]). Where the jq filter is:
    • . identity filter
    • {}: object construction
    • +: addition operator between two objects. So takes the original configuration and merge the new object created:

      Objects are added by merging, that is, inserting all the key-value pairs from both objects into a single combined object.

  • This output is passed to the standard input cat with a here-string (<<<) using command substitution ($(..)). So the jq command runs in a subshell.
  • cat is then overwriting the original config file

It's equivalent to:

new_config=$(jq <filter> $cw_config)
echo $new_config | cat > $cw_config

From what I understand, the combination of here-string and command substitution will use a temp file or memory to avoid any race condition while modifying the file in place.

fi

# Enable and start amazon-cloudwatch-agent
systemctl enable amazon-cloudwatch-agent
systemctl start amazon-cloudwatch-agent
4 changes: 0 additions & 4 deletions packer/linux/scripts/install-cloudwatch-agent.sh
Original file line number Diff line number Diff line change
@@ -2,16 +2,12 @@

set -eu -o pipefail


echo "Installing cloudwatch agent..."
sudo yum install -y amazon-cloudwatch-agent

echo "Adding amazon-cloudwatch-agent config..."
sudo cp /tmp/conf/cloudwatch-agent/config.json /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

echo "Configuring amazon-cloudwatch-agent to start at boot"
sudo systemctl enable amazon-cloudwatch-agent

# These will send some systemctl service logs (like the buildkite agent and docker) to logfiles
echo "Adding rsyslogd configs..."
sudo cp /tmp/conf/cloudwatch-agent/rsyslog.d/* /etc/rsyslog.d/
14 changes: 14 additions & 0 deletions templates/aws-stack.yml
Original file line number Diff line number Diff line change
@@ -81,6 +81,7 @@ Metadata:
- InstanceRolePermissionsBoundaryARN
- IMDSv2Tokens
- EnableDetailedMonitoring
- EnableCloudWatchMetrics

- Label:
default: Auto-scaling Configuration
@@ -437,6 +438,14 @@ Parameters:
- "false"
Default: "false"

EnableCloudWatchMetrics:
Type: String
Description: Enable CloudWatch agent metrics
AllowedValues:
- "true"
- "false"
Default: "false"

EnableCostAllocationTags:
Type: String
Description: Enables AWS Cost Allocation tags for all resources in the stack. See https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html
@@ -1050,6 +1059,11 @@ Resources:
BUILDKITE_ENABLE_INSTANCE_STORAGE="${EnableInstanceStorage}" \
/usr/local/bin/bk-configure-docker.sh
--==BOUNDARY==
Content-Type: text/cloud-boothook; charset="us-ascii"
CLOUDWATCH_ENABLE_METRICS="${EnableCloudWatchMetrics}" \
BUILDKITE_QUEUE="${BuildkiteQueue}" \
/usr/local/bin/bk-configure-cloudwatch-agent.sh
--==BOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash -v
BUILDKITE_STACK_NAME="${AWS::StackName}" \