[ami,gce]: reduce snapshot size to minimum allowed #491

yaronkaikov · 2023-12-05T08:54:59Z

Every AMI we create today holds a 30Gb EBS snapshot, if we take into account the fact that we copy those images to other regions and have multiple images (dev, debug, releases) it is adding up to our cost.

This commit reduces the size of the snapshot to 8Gb (it's the minimum value allowed for a snapshot)

In addition, it's reducing also the time for the build process from 30 minutes to ~12 minutes (which is also good)

Refs: https://github.com/scylladb/scylla-pkg/issues/3712

yaronkaikov · 2023-12-05T09:00:05Z

fixed typos in commit message :-)

benipeled · 2023-12-05T09:34:41Z

This change might cause disk-space issues due to grow-log-files etc. on the OS level,
Please make sure this change is fully verified with QA tests (both tier1 and tier2)
Other than that, lgtm

yaronkaikov · 2023-12-05T10:13:37Z

This change might cause disk-space issues due to grow-log-files etc. on the OS level, Please make sure this change is fully verified with QA tests (both tier1 and tier2) Other than that, lgtm

Running https://jenkins.scylladb.com/job/scylla-master/job/releng-testing/job/longevity/job/longevity-100gb-4h-test/32/console, the log shows that the instance is with 30Gb root disk

12:11:00  root_disk_size_db: 30

So i assume it's good, will wait for @fruch to review as well to make sure

yaronkaikov · 2023-12-05T10:37:05Z

Verified with https://jenkins.scylladb.com/job/scylla-master/job/releng-testing/job/ami/282/ and https://jenkins.scylladb.com/job/scylla-master/job/releng-testing/job/ami/283/ which also trigger https://jenkins.scylladb.com/job/scylla-master/job/releng-testing/job/longevity/job/longevity-100gb-4h-test/32/

fruch · 2023-12-05T10:45:07Z

what about the swap ? isn't that create on build time ? or it's not anymore ?

fruch · 2023-12-05T10:46:47Z

Also, I think SCT would be o.k. with that, but places like cloudformation and such, might have usability issue and should extend to pick bigger root disks.

fruch · 2023-12-05T10:47:46Z

packer/scylla.json

@@ -4,40 +4,6 @@
      "name": "aws",
      "type": "amazon-ebs",
      "access_key": "{{user `access_key`}}",
-      "ami_block_device_mappings": [


how this is related to the snapshot size ?

It's not been used. so I removed it. I can update the commit message or split it to different commit

fruch · 2023-12-05T10:49:21Z

Also what about GCP/AZure ? there we have smaller disks ?

yaronkaikov · 2023-12-05T11:19:57Z

what about the swap ? isn't that create on build time ? or it's not anymore ?

Swap creation including the calculation is been handled in https://github.com/scylladb/scylla-machine-image/blob/next/packer/scylla_install_image

yaronkaikov · 2023-12-05T11:20:59Z

Also what about GCP/AZure ? there we have smaller disks ?

No, they still using 30Gb. I will adjust it in a follow up patch

yaronkaikov · 2023-12-05T11:23:37Z

Also, I think SCT would be o.k. with that, but places like cloudformation and such, might have usability issue and should extend to pick bigger root disks.

Not sure if anyone is using the cloudformation, and even if someone is using it, shouldn't it be for POC or testing/evaluating Scylla?

yaronkaikov · 2023-12-05T11:40:22Z

@fruch Added another commit with the clenaup

yaronkaikov · 2023-12-05T12:02:09Z

Also what about GCP/AZure ? there we have smaller disks ?

No, they still using 30Gb. I will adjust it in a follow up patch

Only GCE has this configuration, trying to do it in the PR to see if it's working

yaronkaikov · 2023-12-05T16:11:33Z

@fruch Added also GCE with minimal disk size (10Gb) - verified with https://jenkins.scylladb.com/job/scylla-master/job/releng-testing/job/gce-image/95/ and also with https://jenkins.scylladb.com/job/scylla-master/job/releng-testing/job/longevity/job/longevity-10gb-3h-gce-test/22/

fruch · 2023-12-05T17:20:46Z

what about the swap ? isn't that create on build time ? or it's not anymore ?

Swap creation including the calculation is been handled in https://github.com/scylladb/scylla-machine-image/blob/next/packer/scylla_install_image

that was the reason for enlarging the root disk in the fist place, so we can create the swap upfront.
if we don't do that anymore, should be an issue to resize it back to minimum size

syuu1228 · 2023-12-06T12:04:21Z

Every AMI we create today holds a 30Gb EBS snapshot, if we take into account the fact that we copy those images to other regions and have multiple images (dev, debug, releases) it is adding up to our cost.

This commit reduces the size of the snapshot to 8Gb (it's the minimum value allowed for a snapshot)

Why we incleased rootfs size from 10GB to 30GB was because there was not enough disk space for the swapfile (c22807f).
Reducing image size may work, but our setup script automatically shrink swap size to 'half of diskfree'.

(Recommended swap size for Scylla is either total_mem/3 or 16GB - lower of the two, so 10GB of rootfs is almost always not enough)

I think we probably should move swapfile to /var/lib/scylla (data volume) instead of rootfs, then we will have enough space without enlarge rootfs .
We already doing similar thing in Azure, Azure has "Resouce Disk" which is the disk just for swapfile, so we allocate swapfile there.

roydahan · 2023-12-06T23:15:12Z

We can have the AMI set to 10GB and in the documentation or anywhere it's used (terraform?) we ask to set with bigger disk.
However, this must be coordinated with product and in all other places that uses it.

This reverts commit 069318b. Related with scylladb#491, we need to reduce snapshot size of the rootfs, we should not pre-allocate swapfile while building image.

This reverts commit 069318b. Related with #491, we need to reduce snapshot size of the rootfs, we should not pre-allocate swapfile while building image.

yaronkaikov · 2023-12-20T06:44:27Z

Verified https://jenkins.scylladb.com/job/scylla-master/job/releng-testing/job/next-machine-image/257/ (including rebased)

yaronkaikov · 2023-12-21T14:17:18Z

We can have the AMI set to 10GB and in the documentation or anywhere it's used (terraform?) we ask to set with bigger disk. However, this must be coordinated with product and in all other places that uses it.

@roydahan When someone wants to set an instance based on our images, we must set the size of the root disk anyway. So I don't think we should have any impact on that.

@tzach Any reason not to reduce the size to the minimum?

yaronkaikov · 2023-12-21T14:17:28Z

@benipeled @Annamikhlin Please review

benipeled · 2023-12-21T14:36:55Z

@benipeled @Annamikhlin Please review

Nothing changed since my last review - #491 (comment) - if it's verified by SCT and the swap change handled, go ahead

roydahan · 2023-12-21T19:04:43Z

I tried to look in our documentation and I couldn't find anyware in AMI launch for scylladb, explaination on how much to configure for root FS (swap but also logs and other things).
This is what I found:
https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/cluster-management/ec2-dc.html
https://opensource.docs.scylladb.com/stable/getting-started/install-scylla/launch-on-aws.html
https://www.scylladb.com/product/release-notes/scylla-ami-for-aws/

yaronkaikov · 2023-12-21T19:21:26Z

I tried to look in our documentation and I couldn't find anyware in AMI launch for scylladb, explaination on how much to configure for root FS (swap but also logs and other things). This is what I found: https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/cluster-management/ec2-dc.html https://opensource.docs.scylladb.com/stable/getting-started/install-scylla/launch-on-aws.html https://www.scylladb.com/product/release-notes/scylla-ami-for-aws/

I am not sure we have such

we do have https://opensource.docs.scylladb.com/stable/kb/set-up-swap.html . so i am not sure why not set it during scylla_image_setup. it shouldn't take long

tzach · 2023-12-24T19:37:22Z

@tzach Any reason not to reduce the size to the minimum?

Not that I'm aware of. Primarily, we need to run all standard tests with this image.

roydahan · 2023-12-24T22:51:53Z

Not that I'm aware of. Primarily, we need to run all standard tests with this image.

It won't matter because the tests are configuring it with the required root FS size that will have enough space for swap.
The product question here is about the fact that we won't have the minimum size by default and one needs to set it correctly when deploying from our AMI.

TBH, I think it's a change that confuse users and may break some users automations.

Every AMI we create today holds a 30Gb EBS snapshot, if we take into account the fact that we copy those images to other regions and have multiple images (dev, debug, releases) it is adding up to our cost. This commit reduces the size of the snapshot to 8Gb (it's the minimum value allowed for a snapshot) In addition, it's reducing also the time for the build process from 30 minutes to ~12 minutes (which is also good) Refs: scylladb/scylla-pkg#3712

tzach · 2023-12-28T08:35:43Z

It won't matter because the tests are configuring it with the required root FS size that will have enough space for swap.

@roydahan Does it make sense to do this automatically for all users?

I'm not comfortable updating production image settings without testing!

fruch · 2023-12-28T09:31:17Z

It won't matter because the tests are configuring it with the required root FS size that will have enough space for swap.

@roydahan Does it make sense to do this automatically for all users?

the only why you might force people to set bigger root disks, if the image root disk is bigger
so if they set it too small, it would fails to create an instance

you can automate in any other way, if you are giving them ami-id.

now it's 30Gb, and no on can create something smaller
and the by product of it, that all of our images created take 30Gb of storage, and that's why yaron is trying to make it smaller

it's a UX question, cause now if user were counting someone on their process on the default size to o.k. for them and with the swap size that comes default,
now on the next release they might get something else out of the box, that won't have enough swap.

all other users that specific the sizes of root-disks regardless, won't notice it. (they already use >=30Gb now)
I think also scylla-cloud those that, and doesn't rely on the default

I'm not comfortable updating production image settings without testing!

It would be test once it hit master, but with tools that all specify the root disk size with >=30Gb

roydahan · 2023-12-28T09:42:45Z

TBH I think this change doesn't worth the pain and bad UX for fresh install.
I suggest we won't do this for now.

mykaul · 2023-12-28T10:29:52Z

TBH I think this change doesn't worth the pain and bad UX for fresh install. I suggest we won't do this for now.

It is thousands of dollars on EBS costs, btw.
If not now - when?

roydahan · 2023-12-28T21:38:09Z

TBH I think this change doesn't worth the pain and bad UX for fresh install. I suggest we won't do this for now.

It is thousands of dollars on EBS costs, btw. If not now - when?

It's thousands of dollars if you don't delete the snapshots, but AFAIU we do now, and won't save more than 180 days.
So, these extra 22GB will cost us 1.76$ per month, per AMI (0.08$ * 22GB).
Assuming we have around 1 AMI per day in avg, it shouldn't cost thousands of dollars, probably few hundreds.

I think it's worth the user experience of launching our AMI with the defaults value and scylla fails to setup since there is not enough disk space to configure swap.

We're spending much much more money on instances that people don't terminate and live for months...

mykaul · 2023-12-31T15:49:52Z

TBH I think this change doesn't worth the pain and bad UX for fresh install. I suggest we won't do this for now.

It is thousands of dollars on EBS costs, btw. If not now - when?

It's thousands of dollars if you don't delete the snapshots, but AFAIU we do now, and won't save more than 180 days. So, these extra 22GB will cost us 1.76$ per month, per AMI (0.08$ * 22GB). Assuming we have around 1 AMI per day in avg, it shouldn't cost thousands of dollars, probably few hundreds.

Just 1 AMI per day would be great - but it's per region, no?

I think it's worth the user experience of launching our AMI with the defaults value and scylla fails to setup since there is not enough disk space to configure swap.

We can work on improving the user experience too.
Those 30GB may or may not be enough, depending on the instance type anyway, no?

We're spending much much more money on instances that people don't terminate and live for months...

We'll get there too. The fact we have instances running >1 week that are not production ones - should be one of our next item.
We'll also reduce container images (SCT's for example). Every minute run counts. Eventually, things add up.

yaronkaikov · 2024-01-01T07:13:23Z

TBH I think this change doesn't worth the pain and bad UX for fresh install. I suggest we won't do this for now.

It is thousands of dollars on EBS costs, btw. If not now - when?

It's thousands of dollars if you don't delete the snapshots, but AFAIU we do now, and won't save more than 180 days. So, these extra 22GB will cost us 1.76$ per month, per AMI (0.08$ * 22GB). Assuming we have around 1 AMI per day in avg, it shouldn't cost thousands of dollars, probably few hundreds.

Just 1 AMI per day would be great - but it's per region, no?

Actually, we are creating many more AMI's , during December we created 173 AMI's in us-east-1 only , in other testing regions such as eu-west- we have 111 AMIs

Some of them for debug and some are official releases/master

I think it's worth the user experience of launching our AMI with the defaults value and scylla fails to setup since there is not enough disk space to configure swap.

We can work on improving the user experience too. Those 30GB may or may not be enough, depending on the instance type anyway, no?

We're spending much much more money on instances that people don't terminate and live for months...

We'll get there too. The fact we have instances running >1 week that are not production ones - should be one of our next item. We'll also reduce container images (SCT's for example). Every minute run counts. Eventually, things add up.

roydahan · 2024-01-07T15:36:44Z

Reducing the AMI size saves only 500$ in a course of 6 months, calculating 2 AMIs for master and 2 AMIs for enterprise.
It's less than 0.4% of our costs in 6 moths period.

There are other places where we can save for snapshots, taking them in this task: https://github.com/scylladb/scylla-pkg/issues/3712

yaronkaikov requested review from fruch, roydahan, benipeled and Annamikhlin December 5, 2023 08:55

yaronkaikov force-pushed the reduce-ami-snapshot branch 2 times, most recently from 861e66e to a8b5b64 Compare December 5, 2023 08:59

fruch reviewed Dec 5, 2023

View reviewed changes

yaronkaikov requested a review from fruch December 5, 2023 11:24

yaronkaikov force-pushed the reduce-ami-snapshot branch from a8b5b64 to fda06bc Compare December 5, 2023 11:39

yaronkaikov force-pushed the reduce-ami-snapshot branch from 1b435e1 to 249d66e Compare December 5, 2023 12:03

yaronkaikov changed the title ~~[ami]: reduce snapshot size to minimum allowed~~ [ami,gce]: reduce snapshot size to minimum allowed Dec 5, 2023

syuu1228 mentioned this pull request Dec 6, 2023

Move swapfile to data volume (/var/lib/scylla) #492

Closed

syuu1228 mentioned this pull request Dec 11, 2023

Revert "pre-allocate swapfile while building image" #493

Merged

yaronkaikov pushed a commit that referenced this pull request Dec 20, 2023

Revert "pre-allocate swapfile while building image"

26b93d5

This reverts commit 069318b. Related with #491, we need to reduce snapshot size of the rootfs, we should not pre-allocate swapfile while building image.

yaronkaikov force-pushed the reduce-ami-snapshot branch from 249d66e to 3b506d9 Compare December 20, 2023 05:12

yaronkaikov added 3 commits December 25, 2023 09:55

scylla.json: remove un-used code

4f05c74

[gce]: reduce snapshot size to minimum allowed

cbf8845

yaronkaikov force-pushed the reduce-ami-snapshot branch from 3b506d9 to cbf8845 Compare December 25, 2023 07:55

roydahan closed this Jan 7, 2024

yaronkaikov deleted the reduce-ami-snapshot branch January 7, 2024 15:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ami,gce]: reduce snapshot size to minimum allowed #491

[ami,gce]: reduce snapshot size to minimum allowed #491

yaronkaikov commented Dec 5, 2023 •

edited

Loading

yaronkaikov commented Dec 5, 2023

benipeled commented Dec 5, 2023

yaronkaikov commented Dec 5, 2023 •

edited

Loading

yaronkaikov commented Dec 5, 2023

fruch commented Dec 5, 2023

fruch commented Dec 5, 2023

fruch Dec 5, 2023

yaronkaikov Dec 5, 2023

fruch commented Dec 5, 2023

yaronkaikov commented Dec 5, 2023

yaronkaikov commented Dec 5, 2023

yaronkaikov commented Dec 5, 2023 •

edited

Loading

yaronkaikov commented Dec 5, 2023

yaronkaikov commented Dec 5, 2023 •

edited

Loading

yaronkaikov commented Dec 5, 2023

fruch commented Dec 5, 2023

syuu1228 commented Dec 6, 2023

roydahan commented Dec 6, 2023

yaronkaikov commented Dec 20, 2023

yaronkaikov commented Dec 21, 2023

yaronkaikov commented Dec 21, 2023

benipeled commented Dec 21, 2023

roydahan commented Dec 21, 2023

yaronkaikov commented Dec 21, 2023

tzach commented Dec 24, 2023

roydahan commented Dec 24, 2023

tzach commented Dec 28, 2023

fruch commented Dec 28, 2023

roydahan commented Dec 28, 2023

mykaul commented Dec 28, 2023

roydahan commented Dec 28, 2023

mykaul commented Dec 31, 2023

yaronkaikov commented Jan 1, 2024

roydahan commented Jan 7, 2024

[ami,gce]: reduce snapshot size to minimum allowed #491

[ami,gce]: reduce snapshot size to minimum allowed #491

Conversation

yaronkaikov commented Dec 5, 2023 • edited Loading

yaronkaikov commented Dec 5, 2023

benipeled commented Dec 5, 2023

yaronkaikov commented Dec 5, 2023 • edited Loading

yaronkaikov commented Dec 5, 2023

fruch commented Dec 5, 2023

fruch commented Dec 5, 2023

fruch Dec 5, 2023

Choose a reason for hiding this comment

yaronkaikov Dec 5, 2023

Choose a reason for hiding this comment

fruch commented Dec 5, 2023

yaronkaikov commented Dec 5, 2023

yaronkaikov commented Dec 5, 2023

yaronkaikov commented Dec 5, 2023 • edited Loading

yaronkaikov commented Dec 5, 2023

yaronkaikov commented Dec 5, 2023 • edited Loading

yaronkaikov commented Dec 5, 2023

fruch commented Dec 5, 2023

syuu1228 commented Dec 6, 2023

roydahan commented Dec 6, 2023

yaronkaikov commented Dec 20, 2023

yaronkaikov commented Dec 21, 2023

yaronkaikov commented Dec 21, 2023

benipeled commented Dec 21, 2023

roydahan commented Dec 21, 2023

yaronkaikov commented Dec 21, 2023

tzach commented Dec 24, 2023

roydahan commented Dec 24, 2023

tzach commented Dec 28, 2023

fruch commented Dec 28, 2023

roydahan commented Dec 28, 2023

mykaul commented Dec 28, 2023

roydahan commented Dec 28, 2023

mykaul commented Dec 31, 2023

yaronkaikov commented Jan 1, 2024

roydahan commented Jan 7, 2024

yaronkaikov commented Dec 5, 2023 •

edited

Loading

yaronkaikov commented Dec 5, 2023 •

edited

Loading

yaronkaikov commented Dec 5, 2023 •

edited

Loading

yaronkaikov commented Dec 5, 2023 •

edited

Loading