Context deadline exceeded leads to the stack overflow error #247
Labels
bug
Something isn't working
Comments
|
Thank you for reporting this! There is definitely a bug |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Celestia Node version
v0.21.5
OS
Ubuntu 22.04.5 LTS jammy
Install tools
No response
Others
No response
Steps to reproduce it
Expected result
Context deadline exceeded error is being occurred until connection recovers to core node. The stack overflow panic should not be appeared while waiting for it.
Actual result
Context deadline exceeded error leads to get stack overflow panic.
Relevant log output
2025-02-12T13:04:15.003Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWFrPYYiyVw628zGeyw9oVFxc7s4cbEUpmShA5TPUNpTZ6 2025-02-12T13:04:22.847Z INFO header/store store/store.go:367 new head {"height": 3991372, "hash": "73C8792248C92A89F5425C4A550425387BB747669B7893FBCB73CCE39FDDD2E7"} 2025-02-12T13:04:28.969Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWBTY6cd4obNhyJkoKpESHhiziyRVZX5ZytWLw5rDGcPSW 2025-02-12T13:04:33.762Z ERROR core core/fetcher.go:208 fetcher: error receiving signed block {"height": 3991373, "err": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:36.892Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:38.773Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWJGhxHWkKcGwMfCAAzKDrMcizfw5GQxG5ahcLBTTAGm6K 2025-02-12T13:04:38.893Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:40.893Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:42.894Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:43.966Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWSb4fZjgbYMbntjpw82qwUUvDJevb6JfVcC73LEkGfxeS 2025-02-12T13:04:44.895Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:45.007Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWFrPYYiyVw628zGeyw9oVFxc7s4cbEUpmShA5TPUNpTZ6 2025-02-12T13:04:46.896Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:48.896Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:50.897Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:52.898Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:54.899Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:56.900Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:04:57.234Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWH8A9heXtuqZLWGCvojFx8SFkbHB7RErsoFtVrWDNXG3Q 2025-02-12T13:04:58.834Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWBTY6cd4obNhyJkoKpESHhiziyRVZX5ZytWLw5rDGcPSW 2025-02-12T13:04:58.901Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:00.902Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:02.903Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:04.904Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:06.905Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:08.797Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWJGhxHWkKcGwMfCAAzKDrMcizfw5GQxG5ahcLBTTAGm6K 2025-02-12T13:05:08.905Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:10.907Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:11.934Z WARN shrex/eds shrexeds/server.go:123 server: request height not found {"peer": "12D3KooWF3cU7rCy6GKyT9CKff6dDxjqtWApAw3ayZLgugqo82C8", "height": 3991382} 2025-02-12T13:05:12.849Z INFO core core/listener.go:217 listener: listening stopped 2025-02-12T13:05:12.849Z WARN core core/listener.go:150 listener: subscriber error, resubscribing... {"err": "underlying subscription is stuck"} 2025-02-12T13:05:12.849Z ERROR core core/fetcher.go:195 fetcher: error receiving new height {"err": "rpc error: code = Canceled desc = context canceled"} 2025-02-12T13:05:12.908Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:14.910Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:15.008Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWFrPYYiyVw628zGeyw9oVFxc7s4cbEUpmShA5TPUNpTZ6 2025-02-12T13:05:16.911Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:18.913Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:20.913Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:22.914Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:24.916Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:26.918Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:27.228Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWH8A9heXtuqZLWGCvojFx8SFkbHB7RErsoFtVrWDNXG3Q 2025-02-12T13:05:28.010Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWBTY6cd4obNhyJkoKpESHhiziyRVZX5ZytWLw5rDGcPSW 2025-02-12T13:05:28.919Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:30.919Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:32.920Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:34.203Z INFO canonical-log swarm/swarm_listen.go:168 CANONICAL_PEER_STATUS: peer=12D3KooWGeJxXKTLgSXJzw51YggHDA8oVEsiQ6X7FEDTBcyJyMoe addr=/ip4/116.99.121.31/udp/62291/quic-v1 sample_rate=100 connection_status="established" dir="inbound" 2025-02-12T13:05:34.925Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:36.767Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWJxNX1YhsBZn52Sb15RxvrMTLb7inxeUvcbde8k9GzayB 2025-02-12T13:05:36.927Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:38.929Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:40.930Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:42.932Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:43.968Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWSb4fZjgbYMbntjpw82qwUUvDJevb6JfVcC73LEkGfxeS 2025-02-12T13:05:44.934Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:45.014Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWFrPYYiyVw628zGeyw9oVFxc7s4cbEUpmShA5TPUNpTZ6 2025-02-12T13:05:46.936Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:48.937Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:50.957Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:52.986Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:54.988Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:56.990Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:05:57.201Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWH8A9heXtuqZLWGCvojFx8SFkbHB7RErsoFtVrWDNXG3Q 2025-02-12T13:05:58.016Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWBTY6cd4obNhyJkoKpESHhiziyRVZX5ZytWLw5rDGcPSW 2025-02-12T13:05:59.030Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:01.070Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:02.850Z INFO core core/listener.go:217 listener: listening stopped 2025-02-12T13:06:02.850Z WARN core core/listener.go:150 listener: subscriber error, resubscribing... {"err": "underlying subscription is stuck"} 2025-02-12T13:06:02.850Z ERROR core core/fetcher.go:195 fetcher: error receiving new height {"err": "rpc error: code = Canceled desc = context canceled"} 2025-02-12T13:06:03.117Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:05.131Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:07.341Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:09.399Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:11.577Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:13.665Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:13.967Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWSb4fZjgbYMbntjpw82qwUUvDJevb6JfVcC73LEkGfxeS 2025-02-12T13:06:15.019Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWFrPYYiyVw628zGeyw9oVFxc7s4cbEUpmShA5TPUNpTZ6 2025-02-12T13:06:15.759Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:18.145Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:20.274Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:22.875Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:24.975Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:27.362Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:30.173Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:34.097Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:36.221Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 2025-02-12T13:06:39.082Z INFO bitswap/server/decision decision/engine.go:679 received empty message from 12D3KooWJGhxHWkKcGwMfCAAzKDrMcizfw5GQxG5ahcLBTTAGm6K 2025-02-12T13:06:42.178Z WARN shrex/eds shrexeds/server.go:123 server: request height not found {"peer": "12D3KooWDDsyz2TEcKQuHqLQ8rzPoGi7XQXFAvA4ei2pajFWSSm3", "height": 3991398} 2025-02-12T13:06:43.303Z WARN header/sync sync/sync_head.go:46 failed to get recent head, returning current subjective {"sbjHead": 3991372, "err": "fetching signed block at height 0 from core: rpc error: code = DeadlineExceeded desc = context deadline exceeded"} runtime: goroutine stack exceeds 1000000000-byte limit runtime: sp=0xc4c3102350 stack=[0xc4c3102000, 0xc4e3102000] fatal error: stack overflow runtime stack: runtime.throw({0x2f86258?, 0x443ce7?}) /home/admin/go/pkg/mod/golang.org/[email protected]/src/runtime/panic.go:1067 +0x48 fp=0x7ffd32703508 sp=0x7ffd327034d8 pc=0x478228 runtime.newstack() /home/admin/go/pkg/mod/golang.org/[email protected]/src/runtime/stack.go:1117 +0x5bd fp=0x7ffd32703648 sp=0x7ffd32703508 pc=0x45a6dd runtime.morestack() /home/admin/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:621 +0x7a fp=0x7ffd32703650 sp=0x7ffd32703648 pc=0x47ef7a goroutine 1480068315 gp=0xc003767880 m=0 mp=0x5771c60 [running]: github.com/celestiaorg/go-header/sync.(*ranges[...]).Head(0x3c54f00?)Is the node "stuck"? Has it stopped syncing?
yes, until connection is being recovered.
Notes
I also added full logs from getting error at the first time to getting started new heads.
celestia_da_logs.txt
In my case it was huge network spike that leaded to the behaviour.
CPU and RAM were as usual, just a little spikes.
The text was updated successfully, but these errors were encountered: