Shapella Testing

As the Shapella network upgrade is approaching in April 2023, rigorous testing is being conducted to ensure a smooth transition and minimize any potential risks or issues during the upgrade. The Shapella upgrade consists of two parts, the Shanghai Execution Layer upgrade, and the Capella Consensus Layer upgrade. In this article, we will discuss all the testing done in the Shapella upgrade and the different testing techniques used.

• Shapella Testing Timeline
• Shandong Testnet
• Withdrawals Devnets
• Mainnet Shadow Forks
• Zhejiang Testnet
• Goerli Testnet
• Sepolia Testnet
• Withdrawal Testing Techniques

Shapella Testing Timeline

• 13th Oct 2022: Shandong Testnet
• 9th Dec 2022: Withdrawal Devnet 0
• 22nd Dec 2022: Withdrawal Devnet 1
• 10th Jan 2023: Withdrawal Devnet 2
• 17th Jan 2023: Withdrawal Devnet 3
• 24th Jan 2023: Withdrawal Mainnet Shadow Fork 1
• 24th Jan 2023: Withdrawal Devnet 4
• 26th Jan 2023: Withdrawal Devnet 5
• 27th Jan 2023: Withdrawal Devnet 6
• 1st Feb 2023: Zhejiang Testnet
• 16th Feb 2023: Withdrawal Devnet 7
• 28th Feb 2023: Sepolia Testnet
• 3rd Mar 2023: Withdrawal Mainnet Shadow Fork 2
• 14th Mar 2023: Goerli Testnet
• 6th Apr 2023: Withdrawal Mainnet Shadow Fork 3

Shandong Testnet

It was an experimental testnet that ran in association between EF JavaScript Team and EF DevOps Team. A set of Shanghai-considered EIPs was selected for early client testing on this Testnet.

It was a Pure JS-based testnet, i.e., running on a set of Lodestar (CL) and EthereumJS (EL) client boot nodes.

To learn more, readers can follow our earlier article on Shandong Testnet.

Withdrawals Devnets

Testnets are the testing environment where community members can participate and test. On the other hand, devnets are only for developers and testers.

• Withdrawal Devnet 0:

It was mainly used to test all the essential testing functionalities after the Capella side of the fork.

Bugs: Core Developers found a bug in Lodestar, where the pool content of the BLS changes was not distributed to lodestar nodes.

It was again relaunched with 28000 validators with updated specs.

Bugs: Core Developers again found a bug in Lodestar. It was unable to parse the payload fetched from Nethermind. Also, Shanghai Timestamp was no longer printed on Geth Nodes.

TX-Fuzz was also used in the Devnet 0. However, core Developers decided to make it standard on every devnet now.

TX-Fuzz is a package containing helpful functions to create random transactions.

• Withdrawal Devnet 1:

It was a semi-public testnet with some external entities running validators to test features. It was successfully running and finalized.

Bugs: Lighthouse called V2 methods, whereas Teku, Prysm, and Lodestar call V1 methods. This was a minor issue. However, some bad blocks were seen in clients.

• Withdrawal Devnet 2:

This devnet followed CL spec v1.3.0-rc.0. It also tested EIP-6122: Forkid checks based on timestamps in addition to Shanghai EIPs.

Bugs: There was an issue in EthereumJS where it sometimes produced bad blocks. Also, an issue in Geth was found where the transaction pool would accept bad transactions. Both these issues were found by TX-Fuzz.

• Withdrawal Devnet 3:

After the discussions on Consensus-Layer Call 101, Core Developers wanted to explore harmonization in the withdrawals format from CL to EL. As a result, a minor change was introduced in the withdrawal format, where the field amount in the withdrawal structure was changed from wei to gwei.

This devnet followed CL spec v1.3.0-rc.1 and Engine API spec with respect to commit 939255f . In addition, it also tested EIP-6122: Forkid checks based on timestamps in addition to Shanghai EIPs.

Bugs: Besu nodes saw a bad block.

• Withdrawal Devnet 4:

There were a total of 605k validators, where 565k validators were run by EF and 40k validators by other client teams. Spec & features were similar to Devnet 3. Developers wanted to mimic the mainnet environment as much as possible. They have also increased the gas limit to 25M from 4M to include larger contracts. Its main goal was to test BLS submission to the local pool before Capella.

• Withdrawal Devnet 5:

Here, developers were running 4 bootnodes instead of 1 for easier peering and less load on a single bootnode.

• Withdrawal Devnet 6:

Some Besu Nodes were producing bad blocks.

• Withdrawal Devnet 7:

Withdrawal Devnet 7 was launched on February 14th. The Capella fork took place 48 hours later on February 16th, bringing together 600K validators, 300K BLS changes, and 100K exits, all in an effort to simulate worst-case scenarios.

Bugs: The Besu Client recently encountered a problem where it was creating bad blocks due to a misreporting of proposed bad blocks. The developers quickly identified the problem and deployed a fix to address the issue.

Withdrawal Mainnet Shadow Forks

Do you know there were 19 Shadow Forks done during The Merge Upgrade (13 Mainnet Shadow Forks + 6 Goerli Testnet Shadow Forks)?

Shadow Forking refers to using data from a testnet or the mainnet to test sync assumptions for a network upgrade so that developers can test features before deploying the actual upgrade to the mainnet.

• It possibly exposes bugs in clients and would suggest optimizations needed.
• It helps developers test the realistic scenario.
• It helps developers increase their confidence that implementations work as expected.

Withdrawal Mainnet Shadow Fork 1: It started with a few issues because the configuration wasn't correctly applied on Geth. Core Developers have also used some evil nodes that send bad blocks, bad network messages, etc., and trick other nodes into the bad chain. These nodes try to spam invalid blocks/messages Both on the execution layer and the consensus layer.

We saw 12 bad blocks during the night, none of them were accepted by any client. I changed the weights a bit so it will create more issues wrt. withdrawals. Let's see if the clients can handle it https://t.co/1FDQ1W3fYm pic.twitter.com/Eq8nkEJDZp

— MariusVanDerWijden (@vdWijden) January 24, 2023

Withdrawal Mainnet Shadow Fork 2: Shapella Fork on Withdrawal Mainnet Shadow Fork 2 was successfully launched on March 3, 2023, and performed well with all client combinations. To ensure the network's stability, developers also ran several rigorous scenario tests, such as taking the mock relay offline and triggering circuit breaker conditions of missing blocks in a row/epoch. These tests proved successful, as the circuit breaker was triggered as expected in clients with it implemented, and the network continued to finalize.

Withdrawal Mainnet Shadow Fork 3: Developers prepared for the MSF-3 by implementing the latest recommended release of clients and lining up BLS changes. They also configured MEV-Boost at a reasonable level and had two builders present on the network. In addition, they set up validator operations, depositing a few validators, and exiting a few without BLS changes applied. This thorough setup allowed the developers to test the Capella and Shanghai upgrades effectively.

Zhejiang Testnet

Zhejiang was the first Shanghai-Capella (Shapella) public testnet. It meant to invite community to test how Staked ETH Withdrawal process will look like after the mainnet upgrade.

And we hit Shapella on Zhejiang testnet. Withdrawals and BLS changes are already included in the first slot at 43200 (epoch 1350). Full and partial withdrawals both processed. BLS changes are also processed. For more info on withdrawals check out: https://t.co/AHMt2x1AbV pic.twitter.com/SlgpaOQx6f

— Barnabas Busa (@BarnabasBusa) February 7, 2023

To learn more, readers can follow our earlier article on Zhejiang Testnet.

Sepolia Testnet

The activation of the Shapella fork on the Sepolia testnet at epoch 56832 was successful, occurring at 4:04:48 AM UTC on February 28, 2023. Sepolia marks the second public testnet to test the Shapella Fork after Zhejiang, and we're thrilled to report that it's performing exceptionally well with all client combinations.

To learn more, readers can follow our earlier article on Sepolia Testnet.

Goerli Testnet

The Shapella fork has successfully been activated on the Goerli testnet at epoch 162304, occurring at 10:25:36 PM UTC on March 14, 2023. This is the third public testing of the Shapella fork, following its earlier tests on the Zhejiang and Sepolia testnets.

To learn more, readers can follow our earlier article on Goerli Testnet.

Withdrawal Testing Techniques

Shapella network upgrade is being extensively tested using various testing techniques to ensure a smooth and secure transition. The testing has covered all the different aspects of the upgrade, interoperability, and all the expected EIPs. In this section, we will discuss various techniques used in Shapella Testing.

• Local and Interop Devnets: Testing the compatibility and interoperability of different clients is crucial in ensuring that the upgrade works seamlessly and as intended. The first step is to verify that the clients follow the specs and have implemented all the functions correctly. This is essential to avoid any potential compatibility issues during the upgrade process.
• Spec Tests: This acts as a sanity check to ensure that clients aren't implementing a spec that won't pass tests. It is of 3 types; Consensus spec tests, State tests, and Execution spec tests.
• Hive Tests: It is an interoperability tool that is used in testing between multiple clients at the same time. It is very short-lived and deterministic. It is highly useful in happy path tests,i.e., a well-defined test case using known input, which executes without exception, produces an expected output, and also, in bad scenarios. It runs using a simulator that starts up the clients and runs the tests against a predefined interface. It acts as an integration and regression check to ensure clients aren't failing defined edge cases. Here are examples of various test scenarios used in Withdrawals Testing.
• Kurtosis Tests: It spins up a local testnet with the required EL/CL combinations and then allows them to transition. It then asserts some happy case conditions. Finally, it acts as an integration test to make sure clients are compatible. It is helpful in rapidly iterating ideas. It is also available as a Go library, and developers can write Go tests with kurtosis.

Related Articles

• Shandong: Pre-Shanghai Testnet
• Zhejiang Public Testnet is Finalizing
• Why Ethereum Clients prefer SSZ over RLP?
• Upcoming Changes to Ethereum Blockchain
• How Warm COINBASE helps in Gas Cost Reduction?
• Activating ETH Withdrawal with Shanghai-Capella
• Transient Storage for Beginners: EIP-1153 Explained
• How Layer 3 in Future will look like?
• An Overview of Beacon Chain API

Related Videos

• Reth: Ethereum Execution Layer Client Written in Rust
• Sign-In with Ethereum: EIP-4361
• TWAMM: Time-Weighted Average Market Maker
• MobyMask: An Initiative to Eliminate Phishers
• Fractional NFTs: EIP-4675 using EIP-1155 & EIP-1633