Unable to delete the containers using sregistry-cli

[ashwinidr23]

I am unable to delete the containers in sregistryhub using sregistry-cli. however i can delete using GUI. but since i want to delete more containers at once i would like to use sregistry-cli for this.

i get the following error when i try to delete the repository
sregistry delete inadr/inadr-lolcow2:222
[client|registry] [database|sqlite:////home/inadr/.singularity/sregistry.db]
Are you sure you want to remove inadr/inadr-lolcow2:222?y
ERROR Beep boop! Internal Server Error: 500

these are the error logs
docker-compose logs --tail=10 -f uwsgi
Attaching to sregistry_uwsgi_1
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/ratelimit/decorators.py", line 27, in _wrapped
uwsgi_1 | increment=True)
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/ratelimit/utils.py", line 127, in is_ratelimited
uwsgi_1 | usage = get_usage_count(request, group, fn, key, rate, method, increment)
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/ratelimit/utils.py", line 154, in get_usage_count
uwsgi_1 | value = _SIMPLE_KEYSkey
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/ratelimit/utils.py", line 34, in
uwsgi_1 | 'ip': lambda r: r.META['REMOTE_ADDR'],
uwsgi_1 | AttributeError: 'Container' object has no attribute 'META'
uwsgi_1 | [pid: 43|app: 0|req: 1070/2027] 144.199.213.236 () {44 vars in 805 bytes} [Tue Nov 17 04:45:21 2020] DELETE /api/container/inadr3/inadr3-lolcow2:222 => generated 850 bytes in 16 msecs (HTTP/1.1 500) 4 headers in 142 bytes (1 switches on core 2)

I tried to figure out for a solution online, but i could not, any pointers please?

PS: i am still using sregistry-cli and not singularity remote feature because i am having trouble configuring with HTTPS due to certificate issues which are beyond my control at the moment!

[vsoch]

hey @ashwinidr23 - I'd be happy to work on this for you! The error message above actually looks related to rate limiting. We could go back to this (what I consider deprecated) client, but another idea is to add the singularity delete library endpoint directly (which isn't currently implemented):

https://sylabs.io/guides/3.5/user-guide/cli/singularity_delete.html

I'm quite busy this week but I think I should be able to make some time to work on this later this month. In the mean time, I would strongly encourage you to figure out adding https! You can compile Singularity in a way to not require it, but it's much better that it's secure.

[ashwinidr23]

Thank you very much. Either the sregistry-cli or singularity delete end point should help. There were some certificate issues that I noticed the last time I tried to configure with HTTPS. I did not try later, but it is in my list to make it work.

[ashwinidr23]

hey @vsoch ,
I tried updating the sregistry version to latest version(1.1.29) and configuring the sregistry with HTTPS and here are few updates.
i was getting minio.error.AccessDenied error
i went through the known issues and found that i was getting the same error as mentioned in #301. only in my case uwsgi was not getting started instead of scheduler and worker.
After going through this issue, i updated uwsgi part of docker-compose.yml file with
"env_file:
- ./.minio-env"
which resolved my problem.(i am not sure if this problem is just for me or the code needs to be updated.)

PS: i did not get this error with http

However- configuring the registry did not resolve the primary issue that i was facing (i.e., pulling/pushing/deleting images to sregistry using sregistry-cli or singularity)
when i use sregistry-cli,
push - works fine
pull - works after i export "REQUESTS_CA_BUNDLE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem "
delete - does not work

-bash-4.2$ sregistry delete registry://adr23/lolcow:2
[client|registry] [database|sqlite:////var/lib/builder/.singularity/sregistry.db]
Are you sure you want to remove adr23/lolcow:2?y
ERROR Beep boop! Internal Server Error: 500 (i understand this is something to do with rate limit from your previous comment)

when i use singularity:
i am able to add the remote end point and verify the token, however when i try to push i get this error
push:
singularity push -U --library https://domain_name lolcow_latest.sif library://adr23/test/lolcow:1
WARNING: Skipping container verifying
0 B / 79.91 MiB [------------------------------------------------------------------------------------------------------] 0.00% 11s
FATAL: Unable to push image to library: request did not succeed: http status code: 500
or
singularity push -U lolcow_latest.sif library://adr23/test/lolcow:v1
WARNING: Skipping container verifying
0 B / 79.91 MiB [-------------------------------------------------------------------------------------------------------] 0.00% 6s
FATAL: Unable to push image to library: request did not succeed: http status code: 500

pull : works fine

i did not find any details about deleting the containers using singularity in the document here : https://singularityhub.github.io/sregistry/docs/client#pancakes-installation
can you please help me with details or point me to a document for deleting a container using singularity?(since u mentioned sregistry-cli is almost "depreciated")

i want to be able to delete containers using command-line either using sregistry-cli/singularity(which i can automate rather than using GUI).

Thanks in advance!

[vsoch]

Okay let's work through your comments one by one! First, you are correct that the https docker-compose.yml file is missing the mount of the minio environment - I'll add that shortly in a PR. For sregistry, I don't consider sregistry-cli to be supported anymore - you should just be using Singularity. And then the 500 error indicates a server error, so you need to look at the uwsgi logs to see what that is.

[vsoch]

i did not find any details about deleting the containers using singularity in the document here : https://singularityhub.github.io/sregistry/docs/client#pancakes-installation
can you please help me with details or point me to a document for deleting a container using singularity?(since u mentioned sregistry-cli is almost "depreciated")

As far as I know, deleting containers with the Singularity client is not supported. But you could ask the hpcng folks if this is changed!

[vsoch]

Missing minio env is added: #334

[ashwinidr23]

Okay let's work through your comments one by one! First, you are correct that the https docker-compose.yml file is missing the mount of the minio environment - I'll add that shortly in a PR. For sregistry, I don't consider sregistry-cli to be supported anymore - you should just be using Singularity. And then the 500 error indicates a server error, so you need to look at the uwsgi logs to see what that is.

here is the log for push error
singularity push -U lolcow_latest.sif library://builder/test/lolcow:v1
WARNING: Skipping container verifying
0 B / 79.91 MiB [------------------------------------------------------------------------------------------------------] 0.00% 10s
FATAL: Unable to push image to library: request did not succeed: http status code: 500

docker-compose logs --tail=30 uwsgi
uwsgi_1 | [pid: 63|app: 0|req: 35/49] () {34 vars in 448 bytes} [Wed Jan 13 04:43:00 2021] GET /version => generated 58 bytes in 1 msecs (HTTP/1.1 200) 5 headers in 141 bytes (1 switches on core 3)
uwsgi_1 | POST RequestMultiPartPushImageFileView
uwsgi_1 | Internal Server Error: /v2/imagefile/1/_multipart
uwsgi_1 | Traceback (most recent call last):
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner
uwsgi_1 | response = get_response(request)
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 115, in _get_response
uwsgi_1 | response = self.process_exception_by_middleware(e, request)
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 113, in _get_response
uwsgi_1 | response = wrapped_callback(request, *callback_args, **callback_kwargs)
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
uwsgi_1 | return view_func(*args, **kwargs)
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/django/views/generic/base.py", line 71, in view
uwsgi_1 | return self.dispatch(request, *args, **kwargs)
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 505, in dispatch
uwsgi_1 | response = self.handle_exception(exc)
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 465, in handle_exception
uwsgi_1 | self.raise_uncaught_exception(exc)
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
uwsgi_1 | raise exc
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/rest_framework/views.py", line 502, in dispatch
uwsgi_1 | response = handler(request, *args, **kwargs)
uwsgi_1 | File "./shub/apps/library/views/images.py", line 289, in post
uwsgi_1 | res = s3.create_multipart_upload(Bucket=MINIO_BUCKET, Key=storage)
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 357, in _api_call
uwsgi_1 | return self._make_api_call(operation_name, kwargs)
uwsgi_1 | File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 676, in _make_api_call
uwsgi_1 | raise error_class(parsed_response, operation_name)
uwsgi_1 | botocore.exceptions.ClientError: An error occurred (503) when calling the CreateMultipartUpload operation (reached max retries: 4): Service Unavailable
uwsgi_1 | [pid: 63|app: 0|req: 36/50] () {36 vars in 512 bytes} [Wed Jan 13 04:43:00 2021] POST /v2/imagefile/1/_multipart => generated 850 bytes in 9732 msecs (HTTP/1.1 500) 4 headers in 142 bytes (1 switches on core 1)

and when i look at the sregistry GUI, i find a build created after this but when i navigate to the container, it is empty.
since the collection is not empty i am not able to delete the collection even on GUI now. please find the screen shots attached

-bash-4.2$sregistry search gives the name of the file which was "not pushed"
-bash-4.2$ sregistry search
[client|registry] [database|sqlite:////var/lib/builder/.singularity/sregistry.db]
Collections
1 test/lolcow:DUMMY-ed186922-9d3c-4537-9399-51b622fdf55b https://hpcs-sregistry.shell.com/containers/1

contents in images folder
[root@myvm images]# ls
_upload

[vsoch]

That error message tells us that the minio container is not running. Your next thing to debug is:

docker-compose logs --tail=30 minio

As I said before, the sregistry-cli command is not expected to work.

[ashwinidr23]

[root@myvm sregistry]# docker-compose logs --tail=30 minio
Attaching to sregistry_minio_1
minio_1 | Attempting encryption of all config, IAM users and policies on MinIO backend
minio_1 | Endpoint: http://172.17.0.3:9000 http://127.0.0.1:9000
minio_1 |
minio_1 | Browser Access:
minio_1 | http://172.17.0.3:9000 http://127.0.0.1:9000
minio_1 |
minio_1 | Object API (Amazon S3 compatible):
minio_1 | Go: https://docs.min.io/docs/golang-client-quickstart-guide
minio_1 | Java: https://docs.min.io/docs/java-client-quickstart-guide
minio_1 | Python: https://docs.min.io/docs/python-client-quickstart-guide
minio_1 | JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
minio_1 | .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide

[ashwinidr23]

i did not find any errors in minio container

[vsoch]

Have you changed anything about the setup? If you follow the instructions in the documentation to use the mc binary to look at logs, do you see that the request is hitting it?

[ashwinidr23]

i have not changed anything in the setup.
./mc config host ls - this was successfull

[root@myvm sregistry]# ./mc config host ls
gcs
URL : https://storage.googleapis.com
AccessKey : YOUR-ACCESS-KEY-HERE
SecretKey : YOUR-SECRET-KEY-HERE
API : S3v2
Path : dns

local
URL : http://localhost:9000
AccessKey :
SecretKey :
API :
Path : auto

myminio
URL : http://127.0.0.1:9000
AccessKey : access_key_here
SecretKey : secret_key_here
API : s3v4
Path : auto

play
URL : https://play.min.io
AccessKey : access_key_here
SecretKey : secret_key_here
API : S3v4
Path : auto

s3
URL : https://s3.amazonaws.com
AccessKey : YOUR-ACCESS-KEY-HERE
SecretKey : YOUR-SECRET-KEY-HERE
API : S3v4
Path : dns

but i do not get any logs. i tried to push a container while running the below command. but i cud not get any logs !
[root@myvm sregistry]# ./mc admin trace -v myminio
^Cmc: Unable to listen to http trace. Canceling upon user request.

[vsoch]

That’s the issue then - the request never reaches the Minio server, and that’s exactly what the 500 error is telling you. There is something off about your setup and you’ll have to debug. I would try shelling into the various containers, Minio and uwsgi, and seeing if you can see (ping) the others based on their hostnames that are supposed to be setup by Docker compose. You can also check the /etc/hosts file. If that works, the next step is to interactively create the Minio client from the uwsgi container (the file Minio.py) and test interactions with your Minio container. I suspect the first test won’t work and there is something off about the networking for your containers, but that’s just a hunch.

[ashwinidr23]

do you mean to login to uwsgi container and install mc there and try to run trace there?
can you please give me some details about the minio.py file as i have not worked with minio before?

[vsoch]

I mean to shell into the uwsgi container, and then get an interactive python shell (python manage.py shell) and use the minio.py file as example code to create connections to the server (the minioClient and minioExternalClient) and then test different uses of it manually from the images.py file, and keep the minio logs open in another shell to always see the server response (if it even hits, which from what you've shown me it's not). You will likely trigger that same error, and you should debug from there. But first you should test the basic networking, because if it's an issue with docker-compose not allowing the containers to see one another, that's one level up and should be addressed first. I'm telling you exactly how I would debug this if it was sitting in front of me.

[ashwinidr23]

Here is my observation so far,
network seems to be working fine, i am able to ping the containers from inside one container to another, /etc/hosts also looks ok.
like mentioned in the setup documents if i change the minio access key and secret key , i am unable to get the sregistry up and running
logs from docker-compose:
minio_1 |
minio_1 |
minio_1 | Attempting rotation of encrypted config, IAM users and policies on MinIO with newly supplied credentials
minio_1 | ERROR Unable to initialize the server: Unable to initialize sub-systems: Unable to migrate all config at .minio.sys/co nfig/: Invalid rotating credentials
sregistry_minio_1 exited with code 1

[root@myvm sregistry]# ./mc config host add myminio http://127.0.0.1:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
Enter Access Key: newaccesskey
Enter Secret Key:
mc: Unable to initialize new alias from the provided credentials. Get "http://127.0.0.1:9000/probe-bucket-sign-kumm11bw9rxn/?location=": dial tcp 127.0.0.1:9000: connect: connection refused.

however if i do not change the access key and secret key , i can get the sregistry up and running, but singularity pull/push does not seem to work as none of the requests are reaching minio in this case.