Skip to content

Increase resources for SPS core components #359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Mar 19, 2025
Merged

Conversation

LucaCinquini
Copy link
Collaborator

@LucaCinquini LucaCinquini commented Mar 18, 2025

Purpose

  • This PR increases the EC2 resources (memory and CPU) available to the Airflow core components (web server, triggerer, dag-processor, scheduler).
  • It also adds new EC2 instance types to the list of available machines that a user can select
  • Finally, it changes the configuration of the Karpenter node pools to be compatible with all the selectable EC2 types

Proposed Changes

  • [ADD] New EC2 types to the selectable list
  • [CHANGE] Allocation of the Airflow core components pods onto the "airflow-core-components" Karpenter node pool
  • [FIX] Lower and upper limits for some Karpenter node pool configuration
  • [CHANGE] Increase the default values for total number of CPUs and memory for all jobs running concurrently

Issues

Testing

  • Successfully deployed SPS with new configuration.
  • Successfully executed EMIT workflows on EC2s of type "m5ad" and the older types.

@LucaCinquini LucaCinquini marked this pull request as draft March 18, 2025 14:23
@LucaCinquini LucaCinquini marked this pull request as ready for review March 18, 2025 21:45
Copy link
Collaborator

@nikki-t nikki-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me!

@LucaCinquini LucaCinquini merged commit d7b9f3a into develop Mar 19, 2025
4 checks passed
@LucaCinquini LucaCinquini deleted the scaling-up-sps-core branch March 19, 2025 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants