Skip to content

[NC | DBS3] Add support for reserved bucket tags #8967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tangledbytes
Copy link
Member

@tangledbytes tangledbytes commented Apr 10, 2025

Explain the Changes

This PR adds the concept of "reserved" bucket tags which are bucket tags which can be set via noobaa cli or via put-bucket-tagging but they can come with further restrictions like a schema, immutability, etc.

DBS3 Example:

// In config-local.js or in config.json
config.NSFS_GLACIER_RESERVED_BUCKET_TAGS = {
    'deep-archive-copies': {
        schema: {
            $id: 'deep-archive-copies-schema-v0',
            enum: ['1', '2']
        }, // JSON Schema
        immutable: 'if-data',
        default: '1',
        event: true,
    }
}

Usage Example:

$ noobaa-cli bucket update --tag '[{ "key": "k1", "value": "v1" }]' ...
$ noobaa-cli bucket update --merge_tag '[{ "key": "k1", "value": "v1" }]' ...
$ aws s3api put-bucket-tagging TagSet=[{Key=deep-archive-copies,Value=2}]'

Issues: Fixed #xxx / Gap #xxx

Testing Instructions:

  • Doc added/updated
  • Tests added

@tangledbytes tangledbytes force-pushed the utkarsh/feat/nc-cli-bucket-tags branch 2 times, most recently from 6126797 to 779c0bf Compare April 11, 2025 12:30
@tangledbytes tangledbytes requested a review from shirady April 21, 2025 07:01
@@ -256,25 +265,14 @@ async function update_bucket(data) {
*/
async function delete_bucket(data, force) {
try {
const temp_dir_name = native_fs_utils.get_bucket_tmpdir_name(data._id);
const bucket_empty = await _is_bucket_empty(data);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is a bit confusing, suggesting to change it to:

Suggested change
const bucket_empty = await _is_bucket_empty(data);
const is_bucket_empty = await empty_bucket(data);

From what I understand the operation is to empty the bucket and return true when it ends.

try {
const { name, tagging } = params;
dbg.log0('BucketSpaceFS.put_bucket_tagging: Bucket name, tagging', name, tagging);
const bucket = await this.config_fs.get_bucket_by_name(name);
bucket.tag = tagging;
const { ns } = await object_sdk.read_bucket_full_info(name);
const is_bucket_empty = await BucketSpaceFS.#_is_bucket_empty(name, null, ns, object_sdk);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you pass null as the params?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because I wanted to not provide and params in this case.

undefined vs null - this is quite opinionated. I prefer using null over undefined when I intentionally want to have a zero value. It helps me tell when the zero value was intentional and when the zero value is system generated.

I am aware that Guy disagrees with this opinion as null is an object.


let list;
try {
if (ns._is_versioning_disabled()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

versioning is a property of the bucket, I think you can check if the bucket versioning is disabled in a simpler way.

versioning: {
$ref: 'common_api#/definitions/versioning',

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to keep the check consistent with the rest of bucketspace_fs.

Comment on lines +997 to +1059
list = await ns.list_objects({ ...params, bucket: name, limit: 1 }, object_sdk);
} else {
list = await ns.list_object_versions({ ...params, bucket: name, limit: 1 }, object_sdk);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this code used more than once?
We can move it to a function and reuse it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is I think but don't think its worth moving these lines into a new function. This code is concise enough, wdyt?

@shirady
Copy link
Contributor

shirady commented Apr 21, 2025

@tangledbytes some general comments / questions:

  • Could you update the NC NSFS docs with the new change?
  • Could you add testing instructions and automated tests for the code changes? (especially on the reserved case)
  • Does it mean that now every bucket would have the property tag in it? Do we need to consider upgrade changes for older buckets?

@tangledbytes tangledbytes force-pushed the utkarsh/feat/nc-cli-bucket-tags branch 2 times, most recently from d312bae to 5cf067e Compare April 28, 2025 07:40
Signed-off-by: Utkarsh Srivastava <[email protected]>

fix put_bucket_tagging test - pass dummy objectsdk

Signed-off-by: Utkarsh Srivastava <[email protected]>

add support for events, race safe tag manipulation and --merge_tag flag

Signed-off-by: Utkarsh Srivastava <[email protected]>

fix lint issues

Signed-off-by: Utkarsh Srivastava <[email protected]>

clarify the reserved tags config comment

Signed-off-by: Utkarsh Srivastava <[email protected]>

add docs for the CLI changes

Signed-off-by: Utkarsh Srivastava <[email protected]>

address PR comments

Signed-off-by: Utkarsh Srivastava <[email protected]>
@tangledbytes tangledbytes force-pushed the utkarsh/feat/nc-cli-bucket-tags branch from 5cf067e to d52195c Compare April 28, 2025 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants