deployment's gone from the provider but not the chain / unreserving unmanaged order

[andy108369]

The deployment 11648596's gone from the provider on 12th of July 2023; In the chain it was active
dseq 11648596
dseq owner akash1c5xu5xmauq4zkam0t4hlddvwk327ny20y68ddt
provider akash1pl9sm32kt0xmcyfwjsf9guu9f93yf8p6k7grtn

Versions

network: mainnet-2
provider-services 0.2.1
akash v0.22.8 (the RPC his provider is using)

Provider Logs

Checking the 11648596 dseq:

provider-logs-12July-0411UTC$ grep 1648596 13.log 
2023-07-13T12:02:17.40270005+01:00 stdout F I[2023-07-13|11:02:17.402] unreserving unmanaged order                  module=provider-cluster cmp=provider cmp=service lease=akash1c5xu5xmauq4zkam0t4hlddvwk327ny20y68ddt/11648596/1/1/akash1pl9sm32kt0xmcyfwjsf9guu9f93yf8p6k7grtn
2023-07-13T12:02:17.402755+01:00 stdout F D[2023-07-13|11:02:17.402] unreserving capacity                         module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1c5xu5xmauq4zkam0t4hlddvwk327ny20y68ddt/11648596/1/1
2023-07-13T12:02:17.402775911+01:00 stdout F I[2023-07-13|11:02:17.402] attempting to removing reservation           module=provider-cluster cmp=provider cmp=service cmp=inventory-service order=akash1c5xu5xmauq4zkam0t4hlddvwk327ny20y68ddt/11648596/1/1

• akash-provider logs https://transfer.sh/HvGdFXW1EU/provider-logs-12July-0411UTC.tar.gz
• helm_list-provider.akash1.pxcloudcompute.net-r_1_0.txt
• ingress_nginx_config-provider.akash1.pxcloudcompute.net-r_1_0.txt
• provider_config-provider.akash1.pxcloudcompute.net-r_1_0.txt

11648596 dseq is only visible against the RPC that has history when that dseq was created

PROVIDER=akash1pl9sm32kt0xmcyfwjsf9guu9f93yf8p6k7grtn; HEIGHT=$(akash query block | jq -r '.block.header.height'); provider-services query market lease list --height $HEIGHT --provider $PROVIDER --gseq 0 --oseq 0 --page 1 --limit 10000 --state active -o json | jq --argjson h $HEIGHT -r '["owner","dseq/gseq/oseq","rate","monthly","withdrawn","consumed","days"], (.leases[] | [(.lease.lease_id | .owner, (.dseq|tostring) + "/" + (.gseq|tostring) + "/" + (.oseq|tostring)), (.escrow_payment | (.rate.amount|tonumber), (.rate.amount|tonumber), (.withdrawn.amount|tonumber)), (($h-(.lease.created_at|tonumber))*(.escrow_payment.rate.amount|tonumber)/pow(10;6)), (($h-(.lease.created_at|tonumber))/((60/6)*60*24))]) | @csv' | awk -F ',' '{if (NR==1) {$1=$1; printf $0"\n"} else {block_time=6; rate_akt=(($4*((60/block_time)*60*24*30.436875))/10^6); $4=rate_akt; withdrawn_akt=($5/10^6); $5=withdrawn_akt; $6; $7; print $0}}' | column -t 
"owner"                                         "dseq/gseq/oseq"  "rate"  "monthly"  "withdrawn"  "consumed"  "days"
"akash1h24fljt7p0nh82cq0za0uhsct3sfwsfu9w3c9h"  "11902730/1/1"    43      18.8465    0            0.006966    0.01125
"akash1c5xu5xmauq4zkam0t4hlddvwk327ny20y68ddt"  "11648596/1/1"    7       3.06804    1.59281      1.780002    17.65875
"akash1m73mjlq7n0qeda82qdh7laflczrxtf7nahkyjf"  "11888436/1/1"    146     63.9905    2.08576      2.109846    1.0035416666666668

Chain view for 11648596 dseq

$ provider-services query txs --events "akash.v1.dseq=11648596" --page 1 --limit 100 -o json | jq -r '(["timestamp","height","code","txhash","provider/owner","type"]), (.txs[] | [.timestamp, .height, .code, .txhash, (.tx.body.messages[] | .provider+.id.owner+.bid_id.owner, ."@type")]) | @csv' | column -t -s","
"timestamp"             "height"    "code"  "txhash"                                                            "provider/owner"                                "type"
"2023-06-25T15:47:34Z"  "11648598"  0       "ABC0DD6AF3F477155A1F177CD0323DB613C95962789E1F7BF92E2B73E259146D"  "akash1c5xu5xmauq4zkam0t4hlddvwk327ny20y68ddt"  "/akash.deployment.v1beta2.MsgCreateDeployment"
"2023-06-25T15:47:46Z"  "11648600"  0       "C4920BCB0693D1AA13EF81394EB46CD500D142F1A57F563BD49B25603105F042"  "akash10l6t6qxlcsae2vwfh0wtt4792z3dftrl7ct53m"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "6D07DAA540C0D0488E9ACC0CD773977DFDF7AAF29ABDD341C9641576BD7A4FFD"  "akash1tweev0k42guyv3a2jtgphmgfrl2h5y2884vh9d"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "57D6A0C927F4EA7FE46F8D1E6AC3047EA1A63A75E87B0A2ACE693F518F2294CF"  "akash1f7ajaj2sswmqmlrxjhje6du7umwgpuxvpg80xx"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "6D287F090DBAFFCCEAB444C1B28DB54959EB41A5A9683D93CE0F6DA5BC33E8DA"  "akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "8FFC2E98BA96B1C3370714AA8EEE61737076B5E586DE9D3BD5C13F62DA877AF9"  "akash15ksejj7g4su7ljufsg0a8eglvkje94z8qsh68a"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "C54C7EA3115CFE2EAB68CD4EE3FDD02F79646E8FD2F4C3EAB8E327823E5E6F34"  "akash144v0qmrrjxjf9402c2f026yr7jzg9dzdfeyx8c"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "95C19A8426FC2FD632D18E3731FEBFF41D2FDF50B741CA8C33D1E8392C6A00DB"  "akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "604EFA1DBDBD3C48D4AD789547FCB89939E78BD764446668EDEC6554B1F10304"  "akash17l0f3kf7gv4kmgqjmgc0ksj3em6lqgcc4kl4dg"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "C52B8AB86A9314CBCF6599665C786DE3CAB5FC7D9C18D8D912B0C02452F0833D"  "akash1p5vh8dsck7mnvc5zha6u3aes0fya3lq4ec9dpy"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "1FC467207D5F31BAEE33D1D65286ACB809177D88A716CDC836217FC9ABB95A96"  "akash1kdhkqwqm9m4yzjfvzn50fgpxyy0mrqgd80eeqt"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "004BC67F96E21ED019EEA535C6A489BD48DC4B76145936C9E7BFA16059D920A8"  "akash15cdrr6sha4zp79gy0g4wqlslgxhwqwmal3uwfp"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "933CDA234D8BE5F0F6CF08D3F7F732F1B0F74B71F01BCBB524F494A8FA7862D9"  "akash167y7j0yg5nct9uv3efww0sut5ck20y4xj3xk4f"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "4B1E616D9C221769ED48D0655C43C93EB2EA1BA438FD07B677DEAAE7B5BE015D"  "akash1hvs8ma8naujens7lsugrs3dc4lkfgcyrvrz579"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "E65014CAA422D9375BC35F2DD71336FFC3F5B597224432693051A35713DA0FC9"  "akash175llqyjvxfle9qwt740vm46772dzaznpzgm576"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "64766BEAC6A19DED60141DF753F042C63B7B07BD64C6B09FC8DC97F63EDD45A7"  "akash1mn3fx53ngzptq7l4h4f0mqa0yj6fd8pd0s2xgm"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "864B3CD7345795D0AA835BFC9AAFEEE45EC874222211F3999303389009D9088D"  "akash1pl9sm32kt0xmcyfwjsf9guu9f93yf8p6k7grtn"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "312DB48554176CEA54C8477441777F7BEBA4889E6A9BBD8B795A0A6EA2BE1B56"  "akash18ga02jzaq8cw52anyhzkwta5wygufgu6zsz6xc"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "55062AA140BFCE86CF376BE9AC2775AE42B8B5AF5482CA6E86BA5ED943C1ECB7"  "akash1e4xwlgc3pyqcngxy73chc8c4ju84l8l2vv8074"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "20C9D21A4B4F892C39AA8BCAB85B129CC6486835981385C238DF94A5F85D9096"  "akash10fl5f6ukr8kc03mtmf8vckm6kqqwqpc04eruqa"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "16E4994D4CE9FCCC2AF8F3D04A9761181270A082DE0A51FEA37D40F600BA3FA2"  "akash1pev2y734effvxxn8rzkp5vqq0jmqq0y0s8hvjc"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:47:46Z"  "11648600"  0       "644F06277660A60DC4A3C180C348F26D5E8E0261A1A90ABE537E46906953C30E"  "akash1u5cdg7k3gl43mukca4aeultuz8x2j68mgwn28e"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:48:10Z"  "11648604"  0       "0156CC6B618707F000D918B0D4C157E453884177E5C3A55211860AA76099CD3E"  "akash1zlsep362zz46qlwzttm06t8lv9qtg8gtaya97u"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:48:10Z"  "11648604"  0       "BD292BB5AA70A964BCC8C191220CBFDADF25086A5A39F64C42778D5935645205"  "akash1qhjtxmacslmefm3v4sn5ggq6ed9jn83cy2rjd0"  "/akash.market.v1beta2.MsgCreateBid"
"2023-06-25T15:48:40Z"  "11648609"  0       "3D3DE73DF38BA0351D049DA6767E9FE32E9A01C9484BD79907BFB6A9809CEF69"  "akash1c5xu5xmauq4zkam0t4hlddvwk327ny20y68ddt"  "/akash.market.v1beta2.MsgCreateLease"
"2023-07-13T11:02:10Z"  "11903018"  0       "DEB6731A8B785C90944463DE2E935BB9160E90C7B4798662B6F8DFB737C72452"  "akash1c5xu5xmauq4zkam0t4hlddvwk327ny20y68ddt"  "/akash.market.v1beta2.MsgCloseBid" <<< this is us closed it manually

Potential reason

I'm thinking if that could do anything with that the RPC's history begins with the 11754001 height whereas dseq 11648596 was deployed at 11648598 height .. https://www.mintscan.io/akash/txs/ABC0DD6AF3F477155A1F177CD0323DB613C95962789E1F7BF92E2B73E259146D

# curl -s http://akash001.rcp-private.pxcloudcompute.net:26657/status | jq -r | grep
 earliest_block_
      "earliest_block_hash": "B00B8B9E13AE7DEBB76CEF9C5583FBE2C9D324335E85FD9C307D1678C00E9C6F",
      "earliest_block_height": "11754001",
      "earliest_block_time": "2023-07-03T01:15:37.371436235Z",

Next steps

Find the reproducer

To reproduce one would need to follow this path:

1. Deploy the akash-provider and akash-node
2. Deploy some app to that provider
3. Re-deploy the akash-node so that its history starts after when the app has been deployed (in step 2.)
4. Restart the akash-provider service and see what happens to that deployment from the step 2.

Refs

#17

[andy108369]

can't confirm the reproducer

Hurricane provider has 31 active leases deployed much earlier than the RPC node's earliest block height 12606084.
AKT withdrawing from the leaess is working as expected.

root@control-01:~# export AKASH_NODE="http://$(kubectl -n akash-services get ep akash-node-1 -o jsonpath='{.subsets[0].addresses[0].ip}'):26657"
root@control-01:~# curl -s $AKASH_NODE/status | jq -r | grep catch
      "catching_up": false

root@control-01:~# curl -s $AKASH_NODE/status | jq -r | grep _block
      "latest_block_hash": "49DFD4B4F173ABA1D7E99C43A293662884C96D7293DC5BBB4A62DF6582A9E957",
      "latest_block_height": "12922323",
      "latest_block_time": "2023-09-22T19:43:21.516873004Z",
      "earliest_block_hash": "B65FA78EDD5F8956777994A515BABD87F21C14D8958058A3CC99031482F67287",
      "earliest_block_height": "12606084",
      "earliest_block_time": "2023-08-31T17:22:29.328603681Z",

$ PROVIDER=akash15tl6v6gd0nte0syyxnv57zmmspgju4c3xfmdhk; HEIGHT=$(akash query block | jq -r '.block.header.height'); provider-services query market lease list --height $HEIGHT --provider $PROVIDER --gseq 0 --oseq 0 --page 1 --limit 10000 --state active -o json | jq --argjson h $HEIGHT -r '["owner","dseq/gseq/oseq","rate","monthly","withdrawn","consumed","days"], (.leases[] | [(.lease.lease_id | .owner, (.dseq|tostring) + "/" + (.gseq|tostring) + "/" + (.oseq|tostring)), (.escrow_payment | (.rate.amount|tonumber), (.rate.amount|tonumber), (.withdrawn.amount|tonumber)), (($h-(.lease.created_at|tonumber))*(.escrow_payment.rate.amount|tonumber)/pow(10;6)), (($h-(.lease.created_at|tonumber))/((60/6)*60*24))]) | @csv' | awk -F ',' '{if (NR==1) {$1=$1; printf $0"\n"} else {block_time=6; rate_akt=(($4*((60/block_time)*60*24*30.436875))/10^6); $4=rate_akt; withdrawn_akt=($5/10^6); $5=withdrawn_akt; $6; $7; print $0}}' | column -t 
"owner"                                         "dseq/gseq/oseq"  "rate"      "monthly"  "withdrawn"  "consumed"       "days"
"akash1qnuldv7z06pjzkqr9khttntpaz6fuzzkkjgs89"  "12007878/1/1"    3           1.31487    2.73019      2.743257         63.50131944444445
"akash1zg9f8plgjrvu5edvumdkh6s7t8ea3pc4l7lkyn"  "12771889/1/1"    15.979899   7.00385    2.33418      2.403344849802   10.444305555555555
"akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys"  "9973354/1/1"     1           0.438291   2.9446       2.948943         204.78770833333334
"akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys"  "9973374/1/1"     1           0.438291   2.94457      2.948923         204.78631944444444
"akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys"  "9973472/1/1"     1           0.438291   2.94449      2.948824         204.77944444444444
"akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys"  "10076251/1/1"    1           0.438291   2.84171      2.846045         197.6420138888889
"akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys"  "12601114/1/1"    1           0.438291   0.316815     0.321173         22.303680555555555
"akash1z6ql9vzhsumpvumj4zs8juv7l5u2zyr5yax2ys"  "12792603/1/1"    1           0.438291   0.12535      0.129693         9.006458333333333
"akash1rp23kw9lk7gaa29fwn4ujahsmpfr7602958elw"  "11626106/1/1"    24          10.519     31.0038      31.108368        90.01263888888889
"akash1y48wwg95plz4ht5sakdqg5st8pmeuuljw6y9tc"  "12773635/1/1"    36.867034   16.1585    5.32002      5.479768465624   10.321944444444444
"akash1gydqg3sfjkgxgjuvyvmhfp4lc75g2n3p3lutux"  "12609645/1/1"    15.458521   6.77533    0.115645     4.832998381003   21.711319444444445
"akash1gydqg3sfjkgxgjuvyvmhfp4lc75g2n3p3lutux"  "12609835/1/1"    92.16255    40.394     0.671219     28.79701821045   21.698541666666667
"akash1fcjm7k0dyyyj2g0thq3u5axmqgmqc0jqqxrc93"  "12834013/1/1"    20.150519   8.83179    1.69117      1.778827365763   6.130347222222222
"akash1tccyrt46zfygeve328dr0727s2n56864kgk9yx"  "11996487/1/1"    9           3.94462    8.29295      8.332164         64.29138888888889
"akash1dvg3qsesz4m83pgvfrxjesvdw7kqdyhzvglc7g"  "12610922/1/1"    104.309728  45.718     0.74498      32.479232936544  21.623125
"akash130xhrhruhmv65w3awpgzylqkhzj7km5jgfyez9"  "12543605/1/1"    16          7.01266    5.98947      6.059024         26.297847222222224
"akash1numg2898s79u8qspt3fk7yg9vecdmckwg5q2tl"  "12588141/1/1"    3           1.31487    0.989403     1.002468         23.205277777777777
"akash1numg2898s79u8qspt3fk7yg9vecdmckwg5q2tl"  "12672375/1/1"    3.027221    1.3268     0.743388     0.756569126762   17.355694444444445
"akash155q5556fwlnusglyuq6jrxmnpvevyeum6er0s3"  "12862374/1/1"    3.280523    1.43782    0.182249     0.196513169269   4.1599305555555555
"akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90"  "9972897/1/1"     1           0.438291   2.94507      2.949397         204.8192361111111
"akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90"  "9972979/1/1"     1           0.438291   2.94497      2.949318         204.81375
"akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90"  "9973061/1/1"     1           0.438291   2.9449       2.949234         204.80791666666667
"akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90"  "9973224/1/1"     1           0.438291   2.94474      2.949073         204.7967361111111
"akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90"  "12814829/1/1"    1           0.438291   0.103136     0.107465         7.462847222222222
"akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90"  "12814840/1/1"    1           0.438291   0.103117     0.107453         7.4620138888888885
"akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90"  "12814868/1/1"    1           0.438291   0.103075     0.107426         7.460138888888889
"akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90"  "12814878/1/1"    1           0.438291   0.103085     0.107415         7.459375
"akash1czvvvcaf7wzvmdlayxwx3xyl2qggsfh6n9yrde"  "11563187/1/1"    28          12.2721    37.9331      38.054884        94.38215277777778
"akash16p6lrlxf7f03c0ka8cv4sznr29rym27uv0qz0d"  "12655391/1/1"    37.208968   16.3084    9.76877      9.930738678488   18.534097222222222
"akash163vrm08flp5qr0ksrnes0qwzwk8ty3jj582ayn"  "12674385/1/1"    6.391165    2.80119    1.55673      1.58443371515    17.215972222222224
"akash1a9er6ylly34fevj6pmkljyz2mt5um2pe26rdvw"  "11250055/1/1"    6           2.62975    10.0074      10.03341         116.12743055555555

cause can't be determined based on the logs

Can't find the cause but can see that the deployment manifest wasn't found.

It is possible that the deployment was accepted but the manifest file has never been sent.

Or there might have been a different issue where the manifest was removed.

$ provider-services show-cluster-ns --dseq 11648596 --provider akash1pl9sm32kt0xmcyfwjsf9guu9f93yf8p6k7grtn --owner akash1c5xu5xmauq4zkam0t4hlddvwk327ny20y68ddt
bjaqv8ue0l5f0jj8klii4orh6ska2dpdvk2f2jo6djjm6

$ grep bjaqv8ue0l5f0jj8klii4orh6ska2dpdvk2f2jo6djjm6 -r .
./13.log:2023-07-13T11:45:06.997427098+01:00 stdout F I[2023-07-13|10:45:06.997] CRD manifest not found                       cmp=provider client=kube lease-ns=bjaqv8ue0l5f0jj8klii4orh6ska2dpdvk2f2jo6djjm6
./13.log:2023-07-13T11:48:08.384721386+01:00 stdout F I[2023-07-13|10:48:08.384] CRD manifest not found                       cmp=provider client=kube lease-ns=bjaqv8ue0l5f0jj8klii4orh6ska2dpdvk2f2jo6djjm6
./13.log:2023-07-13T11:54:10.540802517+01:00 stdout F I[2023-07-13|10:54:10.540] CRD manifest not found                       cmp=provider client=kube lease-ns=bjaqv8ue0l5f0jj8klii4orh6ska2dpdvk2f2jo6djjm6
./13.log:2023-07-13T11:55:53.178432679+01:00 stdout F I[2023-07-13|10:55:53.177] CRD manifest not found                       cmp=provider client=kube lease-ns=bjaqv8ue0l5f0jj8klii4orh6ska2dpdvk2f2jo6djjm6
./13.log:2023-07-13T11:57:37.663881008+01:00 stdout F I[2023-07-13|10:57:37.663] CRD manifest not found                       cmp=provider client=kube lease-ns=bjaqv8ue0l5f0jj8klii4orh6ska2dpdvk2f2jo6djjm6