provider removed the deployment as it was unable to (re-)deploy it after receiving updated manifest file, leaving bid/lease open/active #121

andy108369 · 2023-09-08T14:09:12Z

provider removed the deployment as it was unable to (re-)deploy it after receiving updated manifest file, leaving bid/lease open/active

provider-services 0.4.6
00:42 provider restarts due to account sequence mismatch error (which is expected mechanism 1 2 to tackle the case when provider would just hang (stop bid on new order requests until manually restarted))
00:48 provider fails to update (redeploy) the deployment (send-manifest) (only the image version was bumped, the day before it worked without the issues with the same provider-services v0.4.6)

Provider removed the deployment manifest and its namespace after it failed to deploy it upon receiving, leaving bid/lease in the active/open state.

I've manually recovered the lease this way.

Logs

Error

E[2023-09-08|00:48:41.045] execution error                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud state=deploy-active err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))"

More logs

root@worker-01:~/9964422-patterns# strings 9964422-patterns.txt |grep -Ev 'check result|running check|{' | grep 2023-09-08
23502225736:I[2023-09-08|00:48:31.966] update received                              module=provider-manifest cmp=provider deployment=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422 version=187BAEB5D7DA26147B4146AE50ACF3F7F7D946C5B2FC997B3884CBF42D7A3460
23502261951:I[2023-09-08|00:48:41.033] manifest received                            module=manifest-manager cmp=provider deployment=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422
23502262124:I[2023-09-08|00:48:41.036] data received                                module=manifest-manager cmp=provider deployment=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422 version=187baeb5d7da26147b4146ae50acf3f7f7d946c5b2fc997b3884cbf42d7a3460
23502262370:D[2023-09-08|00:48:41.036] requests valid                               module=manifest-manager cmp=provider deployment=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422 num-requests=1
23502262558:D[2023-09-08|00:48:41.036] publishing manifest received                 module=manifest-manager cmp=provider deployment=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422 num-leases=1
23502262744:D[2023-09-08|00:48:41.036] publishing manifest received for lease       module=manifest-manager cmp=provider deployment=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422 lease_id=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk
23502263028:I[2023-09-08|00:48:41.036] manifest received                            module=provider-cluster cmp=provider cmp=service lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk
23502263257:D[2023-09-08|00:48:41.037] shutting down                                module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud cmp=deployment-monitor
23502263554:D[2023-09-08|00:48:41.037] shutdown complete                            module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud cmp=deployment-monitor
23502263851:I[2023-09-08|00:48:41.045] hostnames withheld                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud cnt=0
23502264131:E[2023-09-08|00:48:41.045] deploying workload                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))"
23502264489:E[2023-09-08|00:48:41.045] execution error                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud state=deploy-active err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))"
23502264867:D[2023-09-08|00:48:41.052] purged ips                                   module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
23502265141:D[2023-09-08|00:48:41.095] purged hostnames                             module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
23502265415:D[2023-09-08|00:48:41.095] teardown complete                            module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
23502265689:D[2023-09-08|00:48:41.095] shutting down                                module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
23502265963:D[2023-09-08|00:48:41.095] waiting on dm.wg                             module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
23502266237:I[2023-09-08|00:48:41.095] shutdown complete                            module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
23502266511:D[2023-09-08|00:48:41.095] hostnames released                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
23502266785:D[2023-09-08|00:48:41.095] sending manager into channel                 module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
23502267059:I[2023-09-08|00:48:41.095] manager done                                 module=provider-cluster cmp=provider cmp=service lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk
23502267288:D[2023-09-08|00:48:41.095] unreserving capacity                         module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1
23502267494:I[2023-09-08|00:48:41.095] attempting to removing reservation           module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1
23502267700:I[2023-09-08|00:48:41.095] removing reservation                         module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1
23502267906:I[2023-09-08|00:48:41.095] unreserve capacity complete                  module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1
232630548480:I[2023-09-08|01:47:22.102] manifest received                            module=manifest-manager cmp=provider deployment=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422
232630548653:I[2023-09-08|01:47:22.104] data received                                module=manifest-manager cmp=provider deployment=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422 version=187baeb5d7da26147b4146ae50acf3f7f7d946c5b2fc997b3884cbf42d7a3460
232630548899:D[2023-09-08|01:47:22.104] requests valid                               module=manifest-manager cmp=provider deployment=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422 num-requests=1
232630549087:D[2023-09-08|01:47:22.104] publishing manifest received                 module=manifest-manager cmp=provider deployment=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422 num-leases=1
232630549273:D[2023-09-08|01:47:22.104] publishing manifest received for lease       module=manifest-manager cmp=provider deployment=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422 lease_id=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk
232630549557:I[2023-09-08|01:47:22.104] manifest received                            module=provider-cluster cmp=provider cmp=service lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk
232630549786:E[2023-09-08|01:47:22.105] error looking up manifest                    module=provider-cluster cmp=provider cmp=service err="reservation not found" lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9964422/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk group-name=dcloud

The text was updated successfully, but these errors were encountered:

andy108369 · 2023-09-12T08:02:53Z

Another case with the kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>)) error.
This time I have the complete provider logs. (provider v0.4.6)

initially there was a typo in the image name, missing @ before sha256:<sha256>; though this was not seem to be the cause of the issue (although, it could)

The log file also contained:

E[2023-09-12|00:45:40.165] execution error                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/9966260/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud state=deploy-active err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))"

More of the log file https://gist.githubusercontent.com/andy108369/b159c06b4f0d57e13f1915a0b3d94a5f/raw/dc56a3dd240970bd64dcb6d14f4ef23e1493440b/err.log

The deployment was terminated, manifest & namespace were removed by the akash-provider.
The bid/lease/order/deployment were in active/open states as before.

Recovering from the issue

noticed the bid/lease weren't closed for the prod console;
so I've recovered the console manifest the same way as before https://gist.github.com/arno01/8a97a3f7bdbc3ec8d82d4aa20fd9fab2?permalink_comment_id=4685270#gistcomment-4685270
deployment received an error (the same one we've seen before, due to which the staging console got closed) - manifest-group=dcloud state=deploy-active err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))" -- provider removed the deployment as it was unable to (re-)deploy it after receiving updated manifest file, leaving bid/lease open/active #121 ; I was able to collect the entire akash-provider log. The error was triggered by running this job (2nd run) https://github.com/ovrclk/console-infrastructure/actions/runs/6156222573/job/16704584131

Logs - The logs of a failure after send-manifest hurricane-provider-console-kube-builder-ClusterParams-error.log

Hint: just run grep -w 9966260 against the log file.

Did repeat the above steps, except for:

bounced the akash-provider pod after recovering the manifest;
waited for 1-2 minutes longer after I bounced the akash-provider;
Success!
job logs (3rd run) https://github.com/ovrclk/console-infrastructure/actions/runs/6156222573/job/16704886598

Logs - The logs of a successful result after send-manifest hurricane-provider-console-NO-kube-builder-ClusterParams-error.log

Hint: just run grep -w 9966260 against the log file.

andy108369 · 2023-09-15T08:08:42Z

Have occurred again on the Hurricane provider.

err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))" was not related to the bad image version.

I've nuked the deployments and have redeployed them from scratch now:

Closed the Staging and Prod Console & Console-Proxy using the Destroy GH action
Redeployed them with the Deploy GH action (deployments have enough deposit to run for 376 days)
console-proxy (staging & prod): added the CORS headers to allow them talk (and hence deploy) to the provider it is deployed at ( provider to support CORS headers #http_options #89 )
console-proxy (staging & prod): patched netpol, as console-proxy needs to access the provider it is deployed at ( no ingress, nodePort network access between two deployments running on the same provider #1 )
Updated the Cloudflare CNAME records for the Console, Console-Proxy (Staging & Prod)

andy108369 · 2023-09-28T16:20:51Z

Issue happens after the provider has been rebuilt from scratch (including the VM's / OS / newer kubespray/K8s)

E[2023-09-28|16:09:37.269] deploying workload                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/12989462/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))"
E[2023-09-28|16:09:37.269] execution error                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/12989462/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud state=deploy-active err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))"

(I've sent you the provider logs in Slack)

andy108369 · 2023-09-30T18:31:38Z

New guess for err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))", potential reason is unattended ubuntu upgrades breaking the K8s cluster.

I'll try to disable them, reboot the worker node and see whether this issue occurs again.

refs.
#131

andy108369 · 2023-10-17T09:04:30Z

Can't reproduce this manually.
Have tried:

bumping the image field and sending the manifest (3 times);
applying custom ingress and netpol rules to simulate the deployment that has been failing;
bumping the image field and sending the manifest (2 times);
restarting akash-provider service;
bumping the image field and sending the manifest (2 times);

I believe the enabled unattended upgrades were the root cause of the issue #131

Neither I've had any reports for this issue.
Closing for now.

andy108369 · 2023-11-22T16:58:25Z

Re-opening as am still seeing this error (err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))") on Hurricane provider with k8s v1.27.5 (delivered with kubespray v2.23.0) when sending-manifest to the provider (using the CLI) and that's not limited to the image update in SDL, but also env update.

provider-services 0.4.8 (provider & client [CLI])
akash network 0.28.2

Provider logs:

D[2023-11-22|16:45:37.103] running check                                module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud cmp=deployment-monitor attempt=1
I[2023-11-22|16:45:37.135] check result                                 module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud cmp=deployment-monitor ok=true attempt=1
I[2023-11-22|16:45:47.516] update received                              module=provider-manifest cmp=provider deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605 version=C761DDE12EAAD74D36ACD78EB57DFF035836BD85C162E9B1A071B38313D57BEE
D[2023-11-22|16:45:48.377] running check                                module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud cmp=deployment-monitor attempt=1
I[2023-11-22|16:45:48.403] check result                                 module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud cmp=deployment-monitor ok=true attempt=1
I[2023-11-22|16:45:54.207] manifest received                            module=manifest-manager cmp=provider deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605
I[2023-11-22|16:45:54.210] data received                                module=manifest-manager cmp=provider deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605 version=c761dde12eaad74d36acd78eb57dff035836bd85c162e9b1a071b38313d57bee
D[2023-11-22|16:45:54.210] requests valid                               module=manifest-manager cmp=provider deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605 num-requests=1
D[2023-11-22|16:45:54.210] publishing manifest received                 module=manifest-manager cmp=provider deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605 num-leases=1
D[2023-11-22|16:45:54.210] publishing manifest received for lease       module=manifest-manager cmp=provider deployment=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605 lease_id=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk
I[2023-11-22|16:45:54.210] manifest received                            module=provider-cluster cmp=provider cmp=service lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk
D[2023-11-22|16:45:54.211] shutting down                                module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud cmp=deployment-monitor
D[2023-11-22|16:45:54.211] shutdown complete                            module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud cmp=deployment-monitor
I[2023-11-22|16:45:54.219] hostnames withheld                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud cnt=0
E[2023-11-22|16:45:54.219] deploying workload                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))"
E[2023-11-22|16:45:54.219] execution error                              module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud state=deploy-active err="kube-builder: ClusterParams() returned result of unexpected type (%!s(<nil>))"
D[2023-11-22|16:45:54.232] purged ips                                   module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
D[2023-11-22|16:45:54.248] purged hostnames                             module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
D[2023-11-22|16:45:54.248] teardown complete                            module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
D[2023-11-22|16:45:54.248] shutting down                                module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
D[2023-11-22|16:45:54.248] waiting on dm.wg                             module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
I[2023-11-22|16:45:54.248] shutdown complete                            module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
D[2023-11-22|16:45:54.248] hostnames released                           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
D[2023-11-22|16:45:54.248] sending manager into channel                 module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk manifest-group=dcloud
I[2023-11-22|16:45:54.248] manager done                                 module=provider-cluster cmp=provider cmp=service lease=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1/akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk
D[2023-11-22|16:45:54.248] unreserving capacity                         module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1
I[2023-11-22|16:45:54.248] attempting to removing reservation           module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1
I[2023-11-22|16:45:54.248] removing reservation                         module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1
I[2023-11-22|16:45:54.248] unreserve capacity complete                  module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h/13697605/1/1

andy108369 · 2023-11-22T17:00:55Z

followed-up by #152

andy108369 added repo/provider Akash provider-services repo issues awaiting-triage labels Sep 8, 2023

andy108369 assigned troian Sep 8, 2023

troian mentioned this issue Sep 27, 2023

all leases got deleted upon akash-provider pod restart on Hurricane provider #128

Closed

troian added sev1 and removed awaiting-triage labels Sep 27, 2023

troian mentioned this issue Sep 27, 2023

provider will remove the leases during the k8s maintenance #14

Closed

andy108369 closed this as completed Oct 17, 2023

andy108369 reopened this Nov 22, 2023

andy108369 removed the sev1 label Nov 22, 2023

andy108369 unassigned troian Nov 22, 2023

andy108369 added the awaiting-triage label Nov 22, 2023

andy108369 closed this as completed Nov 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

provider removed the deployment as it was unable to (re-)deploy it after receiving updated manifest file, leaving bid/lease open/active #121

provider removed the deployment as it was unable to (re-)deploy it after receiving updated manifest file, leaving bid/lease open/active #121

andy108369 commented Sep 8, 2023

andy108369 commented Sep 12, 2023

andy108369 commented Sep 15, 2023 •

edited

Loading

andy108369 commented Sep 28, 2023 •

edited

Loading

andy108369 commented Sep 30, 2023 •

edited

Loading

andy108369 commented Oct 17, 2023

andy108369 commented Nov 22, 2023

andy108369 commented Nov 22, 2023

provider removed the deployment as it was unable to (re-)deploy it after receiving updated manifest file, leaving bid/lease open/active #121

provider removed the deployment as it was unable to (re-)deploy it after receiving updated manifest file, leaving bid/lease open/active #121

Comments

andy108369 commented Sep 8, 2023

Logs

andy108369 commented Sep 12, 2023

Recovering from the issue

andy108369 commented Sep 15, 2023 • edited Loading

andy108369 commented Sep 28, 2023 • edited Loading

andy108369 commented Sep 30, 2023 • edited Loading

andy108369 commented Oct 17, 2023

andy108369 commented Nov 22, 2023

andy108369 commented Nov 22, 2023

andy108369 commented Sep 15, 2023 •

edited

Loading

andy108369 commented Sep 28, 2023 •

edited

Loading

andy108369 commented Sep 30, 2023 •

edited

Loading