Troubleshooting: Run a node

Troubleshoot common issues when running an Arbitrum node, with configuration-specific guidance and a report generator for getting help.

The guidance displayed on this page will change based on your selected configuration:

Info
An interactive widget (MultiDimensionalContentWidget) from the original documentation is not yet available in this version of the docs.
Thank you!

At the end of this troubleshooting guide, you'll find a Generate troubleshooting report button. Clicking this button will generate a report that includes your selected configuration. You can include this report when asking for help.

Using this page to generate a troubleshooting report is helpful because it gathers the information that we need in order to resolve your issue.

Step 1: Try the troubleshooting checklist

If you're running into unexpected outputs or errors, the following checklist may help you independently resolve your issue.

The guidance displayed on this page will change based on your selected configuration.

The How to run a full node (Nitro) may address your issue.

The How to run a full node (Nitro) may address your issue.

The How to run a full node (Nitro) may address your issue.

The How to run a local dev node may address your issue.

How to run an archive node may address your issue.

How to run a validator may address your issue.

Answers to frequently asked questions can be found in Frequently asked questions: Run a node.

Step 2: Look for your scenario

Common troubleshooting scenarios and solutions are detailed below.

You can check logs by different log types: info, warn, and error.

ScenarioSolution

You see Unindex transactions.

This is expected behavior. You'll see this when your node removes old txlookup indices. This is emitted from the base Geth node, so you'd see the same output from a mainnet Geth node.

You see Head state missing, repairing.

This is usually because your node shuts down ungracefully. In most cases, it will recover in a few minutes, but if it not, you may have to re-sync your node. Remember to shut down your node gracefully with the following command: docker stop —time=1800 $(docker ps -aq).

Your local machine is running out of memoryNitro (and Geth) can consume a lot of memory depending on the request load. It's possible that your machine may run out of memory when receiving tons of requests.

Your Arbitrum node can’t connect to your parent chain node on localhost:8545

This is often because of a Docker port configuration issue. See https://stackoverflow.com/questions/43884981/unable-to-connect-localhost-in-docker.

You specified your snapshot file path via the --init.url parameter, but the snapshot file isn't found.

This is usually because the snapshot file isn't mounted to your Docker container. Mount it and change the file path to your Docker container’s mount point.

You get 403 errors from the feed URL.

This often happens when Cloudflare attempts to block botnets and other malicious actors but accidentally blocks node runners.

You see latest assertion not yet in our node

This usually because your node hasn’t synced to the latest state, it’s a normal behavior.

You see "Post "xxx_url": context deadline exceeded"

Please check your parent chain endpoint because there is something going wrong on that endpoint; you can check it.

You see Resuming state snapshot generation

This is a normal behavior while the node is catching up to the tip of the chain; once a node has been fully synced, "Resuming state snapshot generation" shouldn't be logged unless it falls behind again.

You see track-block-metadata-from is set but blockMetadata fetcher is not enabled.

You started the node with --node.transaction-streamer.track-block-metadata-from set to a non-zero block but did not enable the blockMetadata fetcher. The node will still run, but blockMetadata won't be backfilled. Either remove the flag, or enable the fetcher with --node.block-metadata-fetcher.enable—and when enabling the fetcher you'll typically also need to set --node.block-metadata-fetcher.source.url to point at a node that serves block metadata.

ScenarioSolution

You see error reading inbox err="sequencer batches out of order; after batch A got batch B”

This is because you get two discontinuous batches; this might be because of your parent chain endpoint issues. You can change to another endpoint and set nitro --init.reorg-to-batch A

You see error reading inbox err="failed to get previous message for pos x: leveldb: not found”

This is because your node db crashed and lost some messages. You can try to set --init.reorg-to-message-batch x-1

You see Failed to load snapshot err="head doesn't match snapshot: have a, want b”

This is usually because an ungraceful shutdown caused a corrupted database; try restarting the node without a prune flag, and after your node goes back to normal, then graceful shut it down and restart to prune it.

You see failed to get blobs: expected at least six blobs for slot [slot_number] but only got 0

This often happens when you connect to a beacon chain endpoint while the blob you are querying is expired. To resolve this error, connect to a beacon endpoint that supports historical blob data (see List of Ethereum beacon chain RPC providers ).

You see P2P server will be useless, neither dialing nor listening

Arbitrum Nitro doesn’t need P2P mode, so you can ignore this log.

You see Getting file info dir=<path>/machines error="stat <path>/machines: no such file or directory".

Nitro is checking one of the expected locations for WASM machine artifacts, but that particular machines/ directory does not exist. This warning can be harmless if Nitro finds the required machine artifacts in another location. If it is followed by validation or WASM module-root errors, make sure you're running a supported/current Nitro image for your chain, and that the image or mounted machines/ directory contains artifacts matching the chain's onchain WASM module root. If you're using an older Docker image, upgrade to a newer Nitro release; if you're building or running from source, mount or download the matching machine artifacts for that Nitro version/module root.

You see broadcaster queue jumped positions queuedMessages=N expectedNextIdx=M.

The feed delivered messages out of the expected order; the node skipped ahead in its broadcast queue. This typically self-heals as later messages arrive. If it persists, check your feed URL connectivity or try a fallback feed via --node.feed.input.url.

You see Compression was not negotiated when connecting to feed server, non-critical: node will continue without compression.

The feed server didn't agree to compression during the WebSocket handshake. This is informational—the node continues without compression and your sync is unaffected. You can ignore this warning.

You see readData returned EOF url=wss://...feed... opcode=0.

The feed server closed the WebSocket connection. The broadcast client will automatically reconnect; no operator action is needed unless EOF repeats continuously. Persistent EOFs suggest a network/firewall issue or that all configured feeds are unreachable.

You see error reading inbox err="previous delayed accumulator mismatch for message N".

The delayed-inbox accumulator your node computed doesn't match the onchain one at message N—typically caused by a parent-chain endpoint serving inconsistent data or by a reorg. Switch to a different parent-chain RPC and restart; if it persists, re-sync from a recent snapshot.

You see error reading inbox err="unexpected delayed sequence number N, expected M".

The delayed inbox produced a sequence number that is not the next expected one—usually a parent-chain RPC inconsistency or a reorg on the parent chain. Switch to a different parent-chain RPC; if the gap persists, re-sync the node.

You see Trie prefetcher failed opening storage trie root=... err="missing trie node ...".

Your local state database is inconsistent—typically caused by an ungraceful shutdown or by enabling --init.recreate-missing-state-from on a non-archive node. Restart without pruning; if the node still can't recover, re-sync from a recent official snapshot.

You see error reading inbox err="...pebble: not found" (same root cause as the leveldb: not found case above; different database backend).

Same root cause as the leveldb: not found case above: your local DB is missing a message it expected to have. Use --init.reorg-to-message-batch x-1 to roll back to before the missing index, or re-sync from a snapshot.

ScenarioSolution

You see no contract code at given address

Your parent chain node might not sync to the latest state, please wait after it finishes syncing.

You see staker: error checking latest staked err="latest assertion of x: globalstate not in chain: count a hash b expected c, sendroot d expected f"

Once it catches up, the node will check the state against the latest confirmed assertion bonded onchain; if it doesn’t match, it will log this error. Usually, this is because of db corruption, so you might need to re-sync the blockchain; using a snapshot might help: https://snapshot-explorer.arbitrum.io/

You see disabling L1 bound as batch posting message is close to the maximum delay blockNumber or batch is within reorg resistance margin from layer 1 minimum block or timestamp bounds

It indicates that there has been an issue with batch posting on the network. This could occur if your batch poster didn't post a batch for an extended period. Common reasons include the node being shut down inadvertently or the batch poster running out of funds, leading to no new blocks being produced or posted by the batch poster.
To resolve this, If re-org doesn’t matter, you can just start the batch poster with --node.batch-poster.reorg-resistance-margin=0 and node.batch-poster.l1-block-bound to ignore, If it does, you'd want to modify the time bounds on the sequencer inbox to allow the sequencer to post a batch containing the transactions with the old timestamp

You see on-chain WASM module root did not match with any of the allowed WASM module roots

Usually, because you are running on an old node version, try to upgrade your node. Also, you modify your node’s code; please refer to the continue set.

You see error acting as staker

In most cases, this error is caused by your parent chain endpoint's rate limit or other issues, you can check your parent chain endpoint. If the error still persists, please ask in our discord node-running channel.

You see wrong msgIdx got X expected Y.

The transaction streamer received a message whose index doesn't match the next expected one—your node's local DB is behind or ahead of the sequencer feed. Restart the node; if the gap persists, re-sync from a snapshot or use --init.reorg-to-message-batch to roll back to a known-good index.

You see error reading inbox err="...header not found" or header for hash not found.

The parent-chain RPC returned a block header your node expected to exist—usually your parent-chain endpoint hasn't synced to the height nitro asked for, or the endpoint serves a different chain history. Wait for the parent-chain node to finish syncing, or switch to a fully-synced parent-chain RPC.

You see accumulator not found or delayed accumulator not found for index N.

Nitro tried to read an inbox accumulator or sequencer-batch metadata entry that is not yet present in the local consensus database. This can happen transiently while the inbox reader or message extractor is still catching up, and batch-posting paths intentionally treat this as an ephemeral error for the first few minutes. If the error keeps repeating, make sure the node has caught up, your parent-chain RPC can serve the relevant SequencerInbox / delayed inbox logs, and the local database or snapshot is not missing historical inbox data. Try restarting with a healthy parent-chain RPC; if the same index remains missing, re-sync from a recent snapshot.

You see error initializing database err="found N unexpected files in database directory, including: ...".

The data directory you pointed nitro at already contains files that don't belong to a fresh DB. Either empty the directory before init, or point --persistent.chain at a fresh path and extract your snapshot there.

You see error validating feed signature error="signature not verified: signer ..." sequence number=N.

A feed message was signed by an address that is not in your allowed signer list. Check that --node.feed.input.verify.allowed-addresses includes the expected sequencer feed signer for your chain; for Arbitrum One/Sepolia this should be the official feed signer published in the docs.

You see no connected feed or no connected feed on startup.

The broadcast client failed to connect to any feed URL you configured. Check network/firewall rules for outbound WebSocket connections, verify the feed URLs in --node.feed.input.url are reachable, and confirm you can curl them from the node host.

Step 3: Generate a troubleshooting report

  1. Complete the above troubleshooting checklist.
  2. Fill in the below form.
  3. Click Generate troubleshooting report.
  4. Copy and paste the generated report text when asking for support on Discord or any other support channel.

Info
An interactive widget (GenerateTroubleshootingReportWidget) from the original documentation is not yet available in this version of the docs.

Node startup command (make sure to remove any sensitive information like, i.e., private keys)

Unexpected output

Tip: Paste the ~100 lines of output before and including the unexpected output you're asking about. You can use the following command to get the logs:

docker logs --tail 100 YOUR_CONTAINER_ID

Generate troubleshooting report

Complete the checklist above before generating...

On this page