I’m currently doing the Initial Block Download (IBD) of a Bitcoin Core node.
For most of the synchronization, everything behaved as expected. The node was using the resources I made available to it: CPU usage was high during validation, RAM usage increased according to the configured cache, network bandwidth was being used, and the synchronization rate varied roughly between 2% and 5% per hour, with a peak around 5%/h.
I then left the node running for about a week while I was away.
When I came back, it was around 68–69% synchronized, but the effective synchronization rate had fallen to approximately 0.03% per hour.
This had apparently happened several days before I returned. Before leaving, the node was around 20%, so even a sustained 1%/h rate should have put it much further ahead after several days.
There was also an unexpected power outage while I was away, which caused the node to shut down abruptly. However, I don’t see any indication that the outage damaged either the machine or the blockchain data. The node starts normally, the chain verifies, and everything else on the computer is functioning normally.
So I started investigating the network side.
Network observations
The most obvious symptom is that the node is receiving almost no useful data.
The machine itself is basically idle during these periods:
- CPU: ~1–3%
- RAM: <1 GB actually used by the whole system
- HDD: very little activity
- Network: often almost completely idle
At one point I measured only about 76 MB received over two hours.
This is not consistent with the machine being unable to validate blocks. When data actually arrives, it can process it reasonably well. The bottleneck appears to be that blocks simply aren’t arriving.
I initially suspected my ISP might be interfering with Bitcoin P2P traffic. I tested the node through Tor, however, and observed essentially the same behavior. Tor itself was working normally. This made an ISP-specific protocol or traffic-shaping problem seem much less likely.
Peer behavior
I then started looking more closely at getpeerinfo.
At one point I had around eight peers. Most of the outbound peers were outbound-full-relay.
The interesting part was the inflight field.
Each peer could have roughly 6–15 blocks in flight, but the set of blocks appeared essentially unchanged when I checked it again 5, 10, or 15 minutes later.
In other words, the node appeared to have requested blocks, but those requests weren’t producing useful progress.
The CPU was nowhere near saturated, so this didn’t look like a situation where validating a block was taking several minutes.
Looking at the network logs
The network logs made the behavior much clearer.
The node repeatedly goes through something roughly like:
- Try connecting to a peer.
- Establish the connection.
- Attempt a v2 handshake.
- Sometimes fall back to v1.
- Exchange a small amount of data.
- Receive a few blocks / headers / other messages.
- Stop receiving useful block data.
- Keep the connection alive for a significant amount of time.
- Eventually hit a stale/timeout condition.
- Disconnect the peer.
- Move on to another peer.
Many candidate peers don’t even get that far: the connection attempt times out or fails.
I have seen this happen over and over again. At one point the peer IDs had progressed from roughly 45 to 90+ while very little useful synchronization had occurred.
The node can therefore spend a very large amount of time cycling through peers without making significant progress.
The peers that remain connected are not necessarily dead. They can respond to PING/PONG and appear healthy from the perspective of basic connectivity. The problem is that they may stop delivering the blocks being requested.
This distinction seems important:
A peer being alive is not necessarily the same thing as being useful for IBD.
Why this seems particularly strange near the end of IBD
I expected synchronization to potentially become slower as the chain gets closer to the tip, but I wasn’t expecting it to collapse from a few percent per hour to approximately 0.03%/h.
In fact, I would have expected finding useful peers to become easier in some respects.
A full archival node has the historical blocks and, of course, everything after them. A pruned node has only the recent part of the chain. Therefore, as the IBD progresses toward the tip, more peers should theoretically have the blocks I need, not fewer.
Instead, I’m seeing the opposite: the node spends enormous amounts of time establishing connections, keeping apparently healthy peers around, requesting blocks, and then receiving very little.
Could this be a P2P attack?
This led me to wonder whether there is a possible denial-of-service attack against IBD specifically, distinct from an eclipse attack or an attempt to make a node accept an invalid chain.
For example, imagine adversarial peers that don’t need to control consensus at all. They would simply behave just well enough to remain useful-looking:
- establish a connection;
- complete the handshake;
- respond to keepalive traffic;
- accept block requests;
- perhaps provide a small amount of data;
- then delay further block delivery for as long as possible;
- remain connected until the node eventually decides that the peer is stale;
- and then repeat with another peer.
The goal would not be to feed the node a fraudulent blockchain. The goal would simply be to consume as much of the node’s connection/latency budget as possible while providing almost no synchronization progress.
If an attacker could populate the address space with sufficiently many such peers, it seems conceivable that an IBD node could spend most of its time waiting for unproductive connections rather than downloading blocks.
I’m not claiming that this is what is happening to my node. There are plenty of benign explanations — poor peers, network conditions, peer selection, IBD behavior, etc. I’m interested in whether the behavior I’m observing is known and expected.
Questions
- Is it normal for IBD to become this extremely slow around ~68–70%?
- Is there a known reason why peers can remain connected and respond to PING/PONG while apparently not delivering blocks for many minutes?
- Is there a diagnostic I should use to distinguish "slow peers" from peers that are effectively useless for block download?
- Does Bitcoin Core intentionally limit how aggressively it replaces connected but unproductive peers?
- Is there a configurable parameter for the block-download/stale-peer timeout, as opposed to the general peer/PING timeout?
- More generally, is the DoS scenario I described a known consideration in Bitcoin Core’s peer-management design?
I’m running Bitcoin Core 31.0 with pruning enabled. The machine has 4 physical CPU cores, 16 GB RAM, and the blockchain is stored on a mechanical HDD. The HDD and CPU don’t appear to be the bottleneck during the periods where synchronization stalls.
I’d particularly appreciate an explanation from someone familiar with the peer-selection and block-download code, because I’m trying to determine whether what I’m seeing is simply an unfortunate P2P environment or whether I’ve stumbled into an actual weakness/edge case in the IBD peer-management behavior.
