Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FATAL: called a nakamoto function outside of epoch 3 #5502

Open
jzethar opened this issue Nov 25, 2024 · 3 comments
Open

FATAL: called a nakamoto function outside of epoch 3 #5502

jzethar opened this issue Nov 25, 2024 · 3 comments

Comments

@jzethar
Copy link

jzethar commented Nov 25, 2024

Describe the bug
On running Stacks node using docker with image release:3.0.0.0.3 node crashes with panic: FATAL: called a nakamoto function outside of epoch 3

Steps To Reproduce
It is not clear if it is reproducible or not. However, the suggestion to resync the node to try reproducing it could be challenging at the moment due to the node's slow syncing speed

Environment (please complete the following information):

  • OS: Debian GNU/Linux 12 as a host machine
  • Docker release: release-3.0.0.0.3

Additional context
Logs:

INFO [1732537880.760897] [stackslib/src/chainstate/nakamoto/coordinator/mod.rs:1085] [chains-coordinator-0.0.0.0:20443] Process burn block 817251 reward cycle 72 in /var/stacks-blockchain-data/mainnet/burnchain, in_prepare_phase: false, is_rc_start: true, is_prior_in_prepare_phase: true, burn_block_hash: 000000000000000000036061c227a69ea9ef8cd4459b4992982c989eb5f5591a
ERRO [1732537880.761261] [testnet/stacks-node/src/main.rs:257] [chains-coordinator-0.0.0.0:20443] Process abort due to thread panic: panicked at /build/stackslib/src/chainstate/nakamoto/coordinator/mod.rs:314:5:
FATAL: called a nakamoto function outside of epoch 3
ERRO [1732537880.829133] [testnet/stacks-node/src/main.rs:259] [chains-coordinator-0.0.0.0:20443] Panic backtrace:    0: stacks_node::main::{{closure}}
             at build/testnet/stacks-node/src/main.rs:258:18
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/boxed.rs:2245:9
      std::panicking::rust_panic_with_hook
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:805:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:664:13
   3: std::sys::backtrace::__rust_end_short_backtrace
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/sys/backtrace.rs:170:18
   4: rust_begin_unwind
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:662:5
   5: core::panicking::panic_fmt
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:74:14
   6: blockstack_lib::chainstate::nakamoto::coordinator::get_nakamoto_reward_cycle_info
      blockstack_lib::chainstate::nakamoto::coordinator::<impl blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,N,U,CE,FE,B>>::get_nakamoto_reward_cycle_info
             at build/stackslib/src/chainstate/nakamoto/coordinator/mod.rs:989:9
      blockstack_lib::chainstate::nakamoto::coordinator::<impl blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,N,U,CE,FE,B>>::handle_new_nakamoto_burnchain_block
             at build/stackslib/src/chainstate/nakamoto/coordinator/mod.rs:1140:21
   7: blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,N,U,CE,FE,B>::handle_new_burnchain_block
             at build/stackslib/src/chainstate/coordinator/mod.rs:2421:9
      blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,blockstack_lib::chainstate::coordinator::comm::ArcCounterCoordinatorNotices,blockstack_lib::chainstate::coordinator::OnChainRewardSetProvider<T>,CE,FE,B>::handle_comms_epoch2
             at build/stackslib/src/chainstate/coordinator/mod.rs:599:19
   8: blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,blockstack_lib::chainstate::coordinator::comm::ArcCounterCoordinatorNotices,blockstack_lib::chainstate::coordinator::OnChainRewardSetProvider<T>,CE,FE,B>::run
             at build/stackslib/src/chainstate/coordinator/mod.rs:565:21
   9: stacks_node::run_loop::neon::RunLoop::spawn_chains_coordinator::{{closure}}
             at build/testnet/stacks-node/src/run_loop/neon.rs:633:17
      std::sys::backtrace::__rust_begin_short_backtrace
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/sys/backtrace.rs:154:18
  10: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/thread/mod.rs:522:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panic/unwind_safe.rs:272:9
      std::panicking::try::do_call
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:554:40
      std::panicking::try
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:518:19
      std::panic::catch_unwind
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panic.rs:345:14
      std::thread::Builder::spawn_unchecked_::{{closure}}
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/thread/mod.rs:521:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/ops/function.rs:250:5
  11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/boxed.rs:2231:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/boxed.rs:2231:9
      std::sys::pal::unix::thread::Thread::new::thread_start
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/sys/pal/unix/thread.rs:105:17
  12: <unknown>
  13: __clone

Dumping core for pid 7
@github-project-automation github-project-automation bot moved this to Status: 🆕 New in Stacks Core Eng Nov 25, 2024
@jcnelson
Copy link
Member

Hey, thank you for reporting this!

Do you happen to know the node's last reported Stacks and Bitcoin block heights? Also, do you happen to have logs from the node from right before the crash? Were you running a genesis sync, or were you booting from a chain snapshot. Thanks in advance!

@jzethar
Copy link
Author

jzethar commented Nov 26, 2024

Of course, there is a response from API about latest blocks:

    "server_version": "stacks-blockchain-api v8.2.2 (master:c75e9fb15)",
    "status": "ready",
    "pox_v1_unlock_height": 781552,
    "pox_v2_unlock_height": 787652,
    "pox_v3_unlock_height": 840361,
    "chain_tip": {
        "block_height": 128620,
        "block_hash": "0x6a90c750c89a29e7be4e3678a5d84eb14808d018300bcc20fee2fce37f8bb2bd",
        "index_block_hash": "0xfca14f7824785316d82598bf23e61a6d5e96114fdf81f291d0b0926be3b5f2ec",
        "burn_block_height": 816867
    }

And here is a full log with information before the crash:

INFO [1732624431.991648] [stackslib/src/chainstate/coordinator/mod.rs:799] [chains-coordinator-0.0.0.0:20443] PoX Anchor block selected, cycle: 72, consensus_hash: ae77adf41d4aec67e9daf7ca40494d528a7af47f, stacks_block_hash: cde8c67e5abdf5aaed52b5a062c547f64e1977aa31382630067f521d5457d24f, stacks_block_id: d5116ed8d557601f7515a1cd201b784e52cc956b92dbd457043efd29c9d76284, is_known: false, commit_txid: 7ef530500356260035d2009329dbb53dd972c527f41e6028cb5287e1af34c909, cycle_burn_height: 817251
INFO [1732624431.992245] [stackslib/src/chainstate/coordinator/mod.rs:2224] [chains-coordinator-0.0.0.0:20443] Anchor block cde8c67e5abdf5aaed52b5a062c547f64e1977aa31382630067f521d5457d24f (txid 7ef530500356260035d2009329dbb53dd972c527f41e6028cb5287e1af34c909) for reward cycle 71 is affirmed by the network (pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppa), but must be downloaded
INFO [1732624431.992257] [stackslib/src/chainstate/coordinator/mod.rs:2353] [chains-coordinator-0.0.0.0:20443] Burnchain block processing stops due to missing affirmed anchor stacks block hash cde8c67e5abdf5aaed52b5a062c547f64e1977aa31382630067f521d5457d24f
INFO [1732624431.992260] [stackslib/src/chainstate/coordinator/mod.rs:2614] [chains-coordinator-0.0.0.0:20443] Burnchain block processing stops due to missing affirmed anchor stacks block hash cde8c67e5abdf5aaed52b5a062c547f64e1977aa31382630067f521d5457d24f
INFO [1732624433.352478] [stackslib/src/net/relay.rs:2559] [relayer-http://0.0.0.0:20443] Got chunks for unconfigured StackerDB replica, stackerdb_contract_id: SP000000000000000000002Q6VF78.signers-0-1
INFO [1732624434.564717] [stackslib/src/net/p2p.rs:1814] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Neighbor accepted!, public key: None, address: 00000000000000000000ffff5b861604
INFO [1732624434.566796] [stackslib/src/net/p2p.rs:1814] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Neighbor accepted!, public key: None, address: 00000000000000000000ffff2d9de96e
INFO [1732624434.573629] [stackslib/src/net/p2p.rs:1814] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Neighbor accepted!, public key: None, address: 00000000000000000000ffff593a108b
INFO [1732624434.610731] [stackslib/src/net/p2p.rs:1814] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Neighbor accepted!, public key: None, address: 00000000000000000000ffff8fc62e82
INFO [1732624434.612559] [stackslib/src/net/p2p.rs:1814] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Neighbor accepted!, public key: None, address: 00000000000000000000ffff9fcb00b6
INFO [1732624434.627105] [stackslib/src/net/p2p.rs:1814] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Neighbor accepted!, public key: None, address: 00000000000000000000ffff96887f28
INFO [1732624434.633561] [stackslib/src/net/p2p.rs:1814] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Neighbor accepted!, public key: None, address: 00000000000000000000ffff0384f125
INFO [1732624434.635582] [stackslib/src/net/p2p.rs:1814] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Neighbor accepted!, public key: None, address: 00000000000000000000ffff59754ebb
INFO [1732624434.774786] [stackslib/src/net/p2p.rs:1814] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Neighbor accepted!, public key: None, address: 00000000000000000000ffff344fab39
INFO [1732624435.513853] [stackslib/src/net/rpc.rs:551] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Handled StacksHTTPRequest, verb: GET, path: /v2/info, processing_time_ms: 0, latency_ms: 0, conn_id: 27, peer_addr: 172.18.0.3:52472, p2p_msg: None
INFO [1732624435.899769] [stackslib/src/chainstate/nakamoto/coordinator/mod.rs:1085] [chains-coordinator-0.0.0.0:20443] Process burn block 817251 reward cycle 72 in /var/stacks-blockchain-data/mainnet/burnchain, in_prepare_phase: false, is_rc_start: true, is_prior_in_prepare_phase: true, burn_block_hash: 000000000000000000036061c227a69ea9ef8cd4459b4992982c989eb5f5591a
ERRO [1732624435.900310] [testnet/stacks-node/src/main.rs:257] [chains-coordinator-0.0.0.0:20443] Process abort due to thread panic: panicked at /build/stackslib/src/chainstate/nakamoto/coordinator/mod.rs:314:5:
FATAL: called a nakamoto function outside of epoch 3
ERRO [1732624435.973191] [testnet/stacks-node/src/main.rs:259] [chains-coordinator-0.0.0.0:20443] Panic backtrace:    0: stacks_node::main::{{closure}}
             at build/testnet/stacks-node/src/main.rs:258:18
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/boxed.rs:2245:9
      std::panicking::rust_panic_with_hook
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:805:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:664:13
   3: std::sys::backtrace::__rust_end_short_backtrace
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/sys/backtrace.rs:170:18
   4: rust_begin_unwind
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:662:5
   5: core::panicking::panic_fmt
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:74:14
   6: blockstack_lib::chainstate::nakamoto::coordinator::get_nakamoto_reward_cycle_info
      blockstack_lib::chainstate::nakamoto::coordinator::<impl blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,N,U,CE,FE,B>>::get_nakamoto_reward_cycle_info
             at build/stackslib/src/chainstate/nakamoto/coordinator/mod.rs:989:9
      blockstack_lib::chainstate::nakamoto::coordinator::<impl blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,N,U,CE,FE,B>>::handle_new_nakamoto_burnchain_block
             at build/stackslib/src/chainstate/nakamoto/coordinator/mod.rs:1140:21
   7: blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,N,U,CE,FE,B>::handle_new_burnchain_block
             at build/stackslib/src/chainstate/coordinator/mod.rs:2421:9
      blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,blockstack_lib::chainstate::coordinator::comm::ArcCounterCoordinatorNotices,blockstack_lib::chainstate::coordinator::OnChainRewardSetProvider<T>,CE,FE,B>::handle_comms_epoch2
             at build/stackslib/src/chainstate/coordinator/mod.rs:599:19
   8: blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,blockstack_lib::chainstate::coordinator::comm::ArcCounterCoordinatorNotices,blockstack_lib::chainstate::coordinator::OnChainRewardSetProvider<T>,CE,FE,B>::run
             at build/stackslib/src/chainstate/coordinator/mod.rs:565:21
   9: stacks_node::run_loop::neon::RunLoop::spawn_chains_coordinator::{{closure}}
             at build/testnet/stacks-node/src/run_loop/neon.rs:633:17
      std::sys::backtrace::__rust_begin_short_backtrace
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/sys/backtrace.rs:154:18
  10: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/thread/mod.rs:522:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panic/unwind_safe.rs:272:9
      std::panicking::try::do_call
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:554:40
      std::panicking::try
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:518:19
      std::panic::catch_unwind
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panic.rs:345:14
      std::thread::Builder::spawn_unchecked_::{{closure}}
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/thread/mod.rs:521:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/ops/function.rs:250:5
  11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/boxed.rs:2231:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/boxed.rs:2231:9
      std::sys::pal::unix::thread::Thread::new::thread_start
             at rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/sys/pal/unix/thread.rs:105:17
  12: <unknown>
  13: __clone

Dumping core for pid 7

Yes, I'm running a genesis sync, using bootstrap nodes:

bootstrap_node = "02196f005965cebe6ddc3901b7b1cc1aa7a88f305bb8c5893456b8f9a605923893@seed.mainnet.hiro.so:20444,02539449ad94e6e6392d8c1deb2b4e61f80ae2a18964349bc14336d8b903c46a8c@cet.stacksnodes.org:20444,02ececc8ce79b8adf813f13a0255f8ae58d4357309ba0cedd523d9f1a306fcfb79@sgt.stacksnodes.org:20444,0303144ba518fe7a0fb56a8a7d488f950307a4330f146e1e1458fc63fb33defe96@est.stacksnodes.org:20444,02da7a464ac770ae8337a343670778b93410f2f3fef6bea98dd1c3e9224459d36b@seed-0.mainnet.stacks.co:20444,02afeae522aab5f8c99a00ddf75fbcb4a641e052dd48836408d9cf437344b63516@seed-1.mainnet.stacks.co:20444,03652212ea76be0ed4cd83a25c06e57819993029a7b9999f7d63c36340b34a4e62@seed-2.mainnet.stacks.co:20444"

Is there any chain archive that I can just download and unpack?

@diwakergupta
Copy link
Member

Is there any chain archive that I can just download and unpack?

Yes, please see https://docs.hiro.so/stacks/archive/guides/stacks-blockchain

jcnelson added a commit that referenced this issue Dec 1, 2024
@jcnelson
chore: fix #5502 by reducing the PoX sync watchdog
fb306ea
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Stacks Core Eng
Status: Status: 🆕 New
Milestone
No milestone
Development

No branches or pull requests
3 participants
@diwakergupta @jcnelson @jzethar