[New Feature]: Experiment with using EC2 types with attached storage for better performance

[LucaCinquini]

The ASIPS team has identified a bottleneck in DAG execution related to downloading the Docker images to the Pod. It is possible that performance would nbe improved if we use EC2 instances with attached SSD storage.

o Try to use m5ad.xlarge as worker node. Must make sure that the Pod uses the attached storage for all I/O operations

o Also, probably less important, try to use m5ad.xlarge to host the Pods for all Airflow services ("airflow-core-components" and "celery-workers").

[rtapella]

Wouldn’t this be fixed if we have a DAG per App Pack/Algorithm? So it’s just all pre-built and ready to go?

[LucaCinquini]

I don't think so... The Docker container that encapsulates the algorithm is totally separate from the EC2 node on which it will run.

[nikki-t]

I did some investigating and think I have a better understanding of NVMe SSD instance store volumes for EC2.

This link contains some helpful general information: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html

I launched a m5ad.xlarge instance which has 1 SSD attached to it and it looks like the NVMe device is already mounted to the root volume on the EC2 instance, so there is no need to mount it:

[ec2-user@ip-xx-xx-xx-xx ~]$ lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme1n1       259:0    0 139.7G  0 disk 
nvme0n1       259:1    0    30G  0 disk 
├─nvme0n1p1   259:2    0    30G  0 part /
└─nvme0n1p128 259:3    0     1M  0 part 

I think this means that I can just try launching the m5ad.xlarge instance and running the CWL DAG and it will take advantage of the attached SSD. But I do want to note that Karpenter has the ability to request an instance with NVMe disks. See this link: https://karpenter.sh/docs/concepts/scheduling/#advanced-scheduling-techniques

Here is another example: https://github.com/aws/karpenter-provider-aws/blob/main/examples/v1/instance-store-ephemeral-storage.yaml

It does look like this may require the NVMe devices be configured as a RAID0 but I am not sure when this is necessary. Maybe when an instance is requested that has more than 1 attached SSD?

For now I am going to test running the DAG on the m5ad.xlarge instance and see how far I get.

[LucaCinquini]

Thanks @nikki-t , let's try your last suggestion first. Once the m5ad worker is up and running, maybe you can ssh into it and verify that the "/data" partition is created onto the "nvmen1" blockstore.

[nikki-t]

I am able to run this CWL which runs the lsblk and df commands to view mounts inside of the Pod. (I am not sure how to SSH into the Pod; my attempts were not successful.)

Here are the logs from a run:

[2025-03-03, 20:00:07 UTC] {pod_manager.py:471} INFO - [base] List block devices
[2025-03-03, 20:00:07 UTC] {pod_manager.py:471} INFO - [base] NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
[2025-03-03, 20:00:07 UTC] {pod_manager.py:471} INFO - [base] nvme1n1       259:0    0 139.7G  0 disk
[2025-03-03, 20:00:07 UTC] {pod_manager.py:471} INFO - [base] nvme0n1       259:1    0    30G  0 disk
[2025-03-03, 20:00:07 UTC] {pod_manager.py:471} INFO - [base] |-nvme0n1p1   259:2    0    30G  0 part /etc/hosts
[2025-03-03, 20:00:07 UTC] {pod_manager.py:471} INFO - [base] |                                       /etc/hostname
[2025-03-03, 20:00:07 UTC] {pod_manager.py:471} INFO - [base] |                                       /etc/resolv.conf
[2025-03-03, 20:00:07 UTC] {pod_manager.py:471} INFO - [base] `-nvme0n1p128 259:3    0     1M  0 part
[2025-03-03, 20:00:07 UTC] {pod_manager.py:471} INFO - [base] List mounted drives
[2025-03-03, 20:00:07 UTC] {pod_manager.py:471} INFO - [base] Filesystem     Type     Size  Used Avail Use% Mounted on
[2025-03-03, 20:00:07 UTC] {pod_manager.py:471} INFO - [base] overlay        overlay   30G  5.7G   25G  19% /

It looks like nvme1n1 is the SSD drive mounted to the underlying EC2 per this documentation. I did some further investigation on an manually launched EC2 instance and saw that the nvme0n1 is an EBS while nvme1n1 is the EC2 NVMe Instance Storage.

sudo nvme list-subsys
nvme-subsys0 - NQN=nqn.2014.08.org.nvmexpress:1d0f1d0fvol0715b6caa297bbed5Amazon Elastic Block Store              
\
 +- nvme0 pcie 0000:00:04.0 live 
nvme-subsys1 - NQN=nqn.2014.08.org.nvmexpress:1d0f0000AWS22813CFDD6743CD6AAmazon EC2 NVMe Instance Storage        
\
 +- nvme1 pcie 0000:00:1f.0 live 

So I need to mount nvme1n1 on /data which is the working directory for the DAG entrypoint script so I tested the Karpenter NodePool Requirements:

{
	"key": "karpenter.k8s.aws/instance-local-nvme",
	"operator": "Gt",
	"values": ["99"]
},

The logs from running with this requirement are the same as above. It seems to be best practice to run a node with NVME attached storage and then mount that to the container using Kubernetes Volumes.

I don't know if there is a way to run a "user-data" script on the EKS pods when they launch as it seems like this functionality is documented for nodes.

Next steps might be:

1. Run a node as m5ad.xlarge
2. Mount NVME to /data
3. Modify Airflow Terraform to define a volume and volume mount for /data
4. Modify DAG KPO to include volume and volume mount

[mike-gangl]

would it make sense to add a param to the instance types (selectable) that specify if NVME is included in the type? and then we can adjust the KPO accordingly?

[rtapella]

I don't think so... The Docker container that encapsulates the algorithm is totally separate from the EC2 node on which it will run.

I guess I imagined that when you "deploy" an app to Airflow, all the parts of it get copied "into" Airflow somehow as a local copy. Then when you execute it wouldn't need to pull the container (etc) from the public repository. This isn't very easy with the single-DAG approach we have now, but if we have a DAG per algorithm I could imagine storing all the algorithm parts (App Package) in "local" storage for Airflow to pull quickly into a new job.

[mike-gangl]

@rtapella you're right- this is a different issue though. I'll create a ticket for this- it's a large piece of work but removes external docker registries as dependencies.

[mike-gangl]

@rtapella , I created a feature here: unity-sds/unity-project-management#244 for what you're describing (and more).