On Tuesday, the Solana blockchain encountered a significant setback, experiencing an outage that halted block production for four hours and 46 minutes. This interruption in service is not the first for Solana, a network celebrated for its high throughput and speed, but also scrutinized for its operational stability. Validators were compelled to initiate a restart with updated software provided by Solana Labs, incorporating a crucial patch intended to rectify the underlying issue.
The Solana Foundation has yet to publish a comprehensive report detailing the cause of the outage. However, insights shared via X (formerly Twitter) by Matthew Sigel, Head of Digital Assets Research at VanEck, and reshared by co-founder Anatoly Yakovenko, provide a technical explanation of the events leading up to the disruption.
Here’s Why Solana Went Down Yesterday
Sigel’s analysis points to a critical flaw in the Berkley Packet Filter (BPF) loader—a fundamental component for deploying, upgrading, and executing programs on the Solana network. This mechanism is essential for deploying, upgrading, and executing programs on the Solana network.
He elaborated, “BPF loader, the ‘Berkley Packet Filter,’ which is the mechanism to deploy upgrade and execute programs on Solana, failed due to a bug linked to a recent Solana Improvement Proposal (SMID) that altered BPF features, including the removal of metadata usage which was deemed redundant.”
Sigel further elaborated on the reason for the bug’s activation, suggesting, “There is speculation that the bug was manually triggered, leading to the network’s downtime.”
The bug, identified during tests on the testnet, had a fix that was not yet deployed to the main network due to the ongoing testing phase. Addressing the steps taken to mitigate the issue, Sigel stated, “Developers have re-written the BPF code lines to eliminate the bug, necessitating a critical patch to the core software. This ensures that once patched, the network can resume its operations securely.”
The process for restarting the network involves validators creating a snapshot of the last block verified by 66% of the network, achieving consensus on this block, and then restarting the chain. The network can only fully resume once 80% of validators agree on the last block, with a risk of halting if the fix does not perform as expected.
Sigel also touched on second order effects of the outage, suggesting, “Once restarted, we anticipate a significant uptick in DeFi activity as arbitrage bots leverage existing arbitrages, potentially leading to $25M in MEV. This incident may caution future development and SMID discussions, particularly around the debated changes to fee markets.”
Moreover, Sigel touched on the long-term implications of this outage for Solana’s innovation trajectory, noting, “This may slow down innovation on Solana as future SMIDs will be more heavily debated.” He specifically referenced the contentious changes to fee markets as an example of the complex dynamics at play within the ecosystem.
Remarkably, the outage had no lasting effect on the SOL price. SOL even closed yesterday’s trading day with a green candle. At press time, SOL was trading at $95.76.
Featured image from Shutterstock, chart from TradingView.com
