Introduction

Scalability has long been a challenge for most public blockchains in the blockchain field. For instance, Bitcoin experienced a three-year-long scalability debate, and Ethereum suffered network congestion due to a simple game, CryptoKitties. To address this issue, various solutions have been proposed in the industry, including short-term scalability by increasing block size, partially sacrificing decentralization through the DPoS consensus mechanism, using alternative structures like DAG, and off-chain scaling methods such as subchains and sidechains.

Among these, sharding technology is considered an effective and fundamental solution. At the 2016 Developer Conference, Ethereum founder Vitalik Buterin published the Ethereum 2.0 “purple paper,” introducing the idea of processing transactions through sharding. As an important direction for blockchain scalability, sharding technology dynamically allocates computing resources through parallel processing, improving blockchain network scalability and laying a technical foundation for supporting high-frequency global transactions.

Current Blockchain Scalability Solutions

Overview of Sharding Technology

Origin of the Idea

Sharding technology originated from database partitioning, which aimed to divide large databases into smaller segments for more efficient data processing. The idea of combining sharding technology with blockchain was first proposed in 2015. A pair of researchers from the National University of Singapore, Prateek Saxena, and Loi Luu, presented a paper at the CCS International Security Conference. They innovatively divided blockchain networks into “fragments” capable of processing transactions simultaneously, providing a new solution to the scalability issue of public blockchains.

Later, this pair of researchers turned theory into practice, developing the first sharding-based project, Zilliqa. Zilliqa adopted a hybrid consensus mechanism of pBFT and PoW, becoming the most efficient public chain for transaction processing. Subsequently, sharding technology also received recognition from Ethereum founder Vitalik Buterin. In 2016, Ethereum proposed a two-layer sharding design, dividing the Ethereum 2.0 network into the main chain and shard chains. The main chain, through the Validator Management Contract (VMC), manages the operation of shard chains, while the shard chains use the PoS consensus mechanism to package transaction data and generate validation blocks. Meanwhile, VMC ensures the validity of transactions and smooth inter-shard data transfer through the UTXO model and receipt trees.

Ethereum 2.0 Sharding Upgrade Flowchart

Since then, as sharding technology continues to evolve, a series of innovative projects have emerged, further driving breakthroughs in blockchain scalability. These projects not only explore the potential of sharding in processing speed and network efficiency but also provide strong support for potential large-scale commercial applications, promising to advance blockchain technology toward a new narrative of high efficiency and broad application.

Definition of Sharding

Sharding technology is a method for optimizing blockchain architecture by dividing the blockchain network into multiple independent “shards” to enable parallel processing of data. Each shard operates as an independent processing unit capable of executing transactions and handling data on its own, thereby effectively distributing the network’s computational and storage burdens. This approach not only significantly improves the transaction processing speed of the blockchain network but also optimizes node storage requirements. Nodes no longer need to maintain the complete data of the entire blockchain. Thus, sharding enhances the scalability and performance of blockchain networks without compromising overall network security, providing technical support for large-scale applications.

Source: New Architectures and Methodologies for High-Performance Sharding Blockchain

Types of Sharding

Sharding technology can be categorized into three main types: network sharding, transaction sharding, and state sharding. The core principle lies in “dividing a whole into parts and managing them separately,” allowing multiple shards to process different transactions simultaneously and then aggregate the results on the main chain, thereby improving the overall performance of the blockchain network.

1. Network Sharding
   Network sharding is the foundational form of sharding upon which other sharding mechanisms are built. The key to network sharding lies in ensuring security and preventing attacks by malicious nodes. Specifically, it involves randomly selecting a group of nodes to form a shard and establishing an independent consensus within the shard to handle transactions. This method significantly increases network concurrency as multiple shards simultaneously process unrelated transactions, thus improving system performance. Zilliqa is a typical example of a blockchain using network sharding, combining PoW and pBFT consensus mechanisms to enhance speed. PoW prevents Sybil attacks, ensuring only legitimate nodes participate in sharding, while pBFT facilitates fast transaction consensus, greatly improving confirmation speed.

2. Transaction Sharding
   Transaction sharding involves distributing different transactions to various shards for processing, thereby accelerating the transaction handling speed of the entire network. Transactions are generally allocated based on the sender’s address, grouping related transactions to prevent double-spending. For instance, if one address initiates two conflicting transactions, they will be quickly identified and prevented within the same shard. In cases where transactions occur across shards, inter-shard communication is used to detect and block double-spending. The UTXO model can further improve transaction sharding efficiency, despite potential issues like splitting large transactions. The maturity of transaction sharding has significantly advanced, allowing multiple consensus mechanisms to work in parallel.

3. State Sharding
   State sharding is the most complex and challenging type of sharding. The key lies in ensuring that each shard maintains only its internal state rather than the entire blockchain’s global state, thus distributing data storage requirements. However, when cross-shard transactions occur, the shards involved must share transaction states, requiring frequent inter-shard communication that can reduce performance. Furthermore, state sharding faces challenges in data consistency and fault tolerance: if a shard is attacked and goes offline, its data validation may be affected. Addressing this issue might require global state backups on each node, but such backups conflict with the decentralized storage intent of state sharding and could introduce centralization risks.

Sharding Implementation Strategies

Sharding Architecture

Sharding architecture design is the core of sharding technology, encompassing the design concepts of main chains and subchains, as well as node allocation within and across shards. In this architecture, the main chain maintains network consensus and security, functioning as the core of the blockchain, coordinating subchain operations, and ensuring global consistency. Subchains are independent regions derived from the main chain, each focusing on processing specific types of transactions and smart contracts, thus achieving independent parallelism to improve performance efficiency and scalability.

Additionally, nodes in the sharding architecture are divided into two roles: subchain nodes, responsible for maintaining transaction records and states within their shard while participating in consensus to validate transactions, and cross-subchain nodes, tasked with transmitting information and updating states across shards to ensure coordination and synchronization between the main chain and subchains. This detailed role division enhances resource utilization and boosts overall transaction processing capacity, laying a solid foundation for the expansion and efficient operation of blockchain networks.

Source: newcomputerworld

Random Sampling

Random sampling and selection mechanisms are critical to ensuring the security and fairness of sharding architectures. The key lies in randomly selecting nodes to form shards and preventing malicious attackers from concentrating control over a shard. During node selection, hash-based random number generation algorithms are often used to ensure fairness and decentralization, eliminating biases based on geographical location or historical behaviour. This ensures all nodes have an equal chance of being selected into different shards, enhancing the network’s decentralization and resistance to censorship.

To prevent attackers from manipulating a shard by controlling certain nodes, sharding architectures typically introduce multiple selection mechanisms and dynamic node allocation strategies. For example, when the number of nodes in a shard reaches a set threshold, the system automatically triggers shard reorganization, randomly selecting new nodes to join and ensuring the distribution of nodes within the shard does not become overly concentrated. Additionally, “shard rebalancing” mechanisms periodically adjust node distribution across shards, preventing attackers from exploiting node concentration to attack or manipulate a shard. These mechanisms effectively reduce the risk of single-point failures within the sharding architecture and strengthen the network’s defense against malicious attacks.

Source: An Effective Sharding Consensus Algorithm for Blockchain Systems

Challenges and Solutions in Sharding

Security Issues

Adaptive adversarial attacks refer to attacks where malicious actors exploit their knowledge of network conditions to target specific shards in a blockchain network. Attackers may manipulate transactions, tamper with data, or interfere with transaction confirmation processes to achieve their objectives. Since each shard in a sharded architecture has relatively fewer nodes, it becomes easier for attackers to concentrate their efforts on a single shard, increasing security risks. To address this issue, measures must be taken to ensure shard integrity.

One effective solution is to introduce multi-layered verification mechanisms and cross-shared consensus protocols. Specifically, multiple validation nodes should be established within each shard to collaboratively confirm transactions, thereby increasing the complexity and cost of attacks. Additionally, cross-shard consensus protocols facilitate information sharing and state validation between shards, ensuring coordination and consistency across the network and preventing attacks on a single shard from threatening the entire network. These mechanisms significantly enhance the resilience of sharded architectures against attacks and reduce the risks posed by adaptive adversarial threats.

Data Availability Challenges

Data availability is another critical challenge in sharding technology. As sharding becomes widely adopted, efficiently verifying the accessibility and integrity of data in each shard becomes essential for maintaining the blockchain network’s stability. One approach to address this challenge is to sample portions of the dataset to quickly verify the availability of the entire data set. This method reduces the computational overhead of inspecting all data, improving the system’s overall efficiency.

Moreover, effective verification mechanisms must be established. For instance, participating nodes should provide corresponding proof of data availability when generating new blocks. This is particularly important in cross-shard transactions to ensure consistency and accuracy of data between shards.

Case Studies

Ethereum 2.0 Sharding Technology

In Ethereum’s scalability roadmap, Danksharding represents a revolutionary upgrade and a core technology for achieving large-scale scalability in Ethereum 2.0. Unlike traditional sharding methods, Danksharding integrates “merged fee markets” and adopts a single block proposer mechanism, simplifying cross-shard transaction processes. The technical implementation will gradually transition to full sharding in Ethereum 2.0 through mechanisms like EIP-4844 and proto-danksharding.

Danksharding’s uniqueness lies in its innovative structural design. Traditional sharding divides blockchain networks into multiple parallel subchains, with each subchain independently handling transactions and reaching consensus. Danksharding, on the other hand, employs a single-block proposer to eliminate the complexity and performance bottlenecks caused by multiple proposers in traditional sharding. The Beacon Chain, as the core consensus layer of Ethereum 2.0, plays a crucial role in this process. It manages and coordinates all validators in the Ethereum network, ensuring security and consistency. Within the Danksharding framework, the Beacon Chain maintains validator states and facilitates cross-shard communication and data synchronization, collectively enhancing Ethereum 2.0’s overall performance.

The implementation of Danksharding will proceed in multiple phases. Initially, proto-danksharding is introduced as a transitional phase during Ethereum’s Cancun upgrade. Using EIP-4844, it supports Rollup technology to reduce data storage costs, laying the groundwork for the full implementation of Danksharding. Furthermore, Danksharding will enhance Ethereum’s security, preventing potential threats like 51% attacks, while optimizing computational and storage demands in the network to support large-scale decentralized applications.

Source: Breaking Down ETH 2.0 - Sharding Explained

Polkadot Sharding Technology

Polkadot achieves sharding through its innovative “parachain” architecture, enabling independent blockchains to operate within the same network while achieving interoperability. Each parachain is an independent blockchain network that processes its data and transactions. These parachains are coordinated and managed through the Relay Chain, which provides a unified consensus mechanism and ensures network security, as well as data synchronization and consistency across all parachains.

Parachains are also customizable, allowing for independent governance structures and tailored functionality to meet specific requirements, greatly enhancing network flexibility and scalability. Polkadot’s parachain architecture is particularly suited for decentralized applications (DApps) with high demands, especially in DeFi, NFT, and DAO sectors, where its scalability and flexibility have been proven. For instance, Polkadot’s parachain slot auction mechanism enables each parachain to secure connection rights to the Relay Chain and use specific computational resources during the lease period. With the addition of more parachains, Polkadot can achieve higher transaction throughput and lower fees.

In Polkadot 1.0, the allocation of core resources was determined through a two-year auction system. In version 2.0, resource allocation became more flexible. As more parachains join and resources are dynamically distributed, Polkadot is poised to become an efficient multi-chain ecosystem supporting a wide range of decentralized applications.

Source: Polkadot v1.0

NEAR Sharding Technology

NEAR Protocol utilizes the innovative Nightshade dynamic sharding technology, allowing the system to adjust the number of shards flexibly based on network demands, maintaining efficient and stable operations under varying loads. Nightshade architecture, successfully implemented on the NEAR mainnet, processes large transaction volumes and supports DApp development, particularly excelling under high-load conditions.

The core advantage of Nightshade lies in its dynamic sharding capability, which adjusts shard numbers in real-time to improve network performance and scalability. With the upcoming Phase 2 upgrade, NEAR introduces significant enhancements to its existing architecture, including “Stateless Validation” technology. This innovation allows NEAR validator nodes to operate without locally storing shard states, instead dynamically obtaining “state witness” information from the network for validation. This approach improves shard processing efficiency, reduces hardware requirements for validators, and enables broader participation. As sharding technology continues to evolve, NEAR is well-positioned to support large-scale user growth and provide the architectural foundation for the widespread adoption of decentralized applications.

Source: What is NEAR Protocol? The Blockchain Operating System (BOS)

TON Sharding Technology

The TON architecture adopts a multi-layer structure consisting of a masterchain and workchains, ensuring efficient network operation and seamless cross-chain communication. The masterchain serves as the core ledger of the network, storing block headers for all workchains and managing the overall network state, including protocol upgrades and validator elections. Workchains are independent subchains within the TON network, each specializing in specific application scenarios or business needs, thereby achieving network flexibility and specialization.

TON emphasizes cross-chain compatibility, enabling seamless interaction with other blockchain networks to enhance ecosystem usability and inter-blockchain functionality. One of TON’s most notable innovations is its infinite sharding paradigm, allowing the network to dynamically adjust the number of shards according to transaction load. Under high loads, TON splits shards to handle more transactions; under low loads, shards merge to conserve resources and improve overall efficiency. This horizontal scaling design allows TON to meet increasing transaction demands without sacrificing performance, supporting high-volume applications like DeFi.

Moreover, TON introduces Hypercube technology, where data transmission time scales logarithmically with the number of blockchains. This means that even as the TON network expands to millions of chains, its processing speed and response times remain unaffected. Theoretically, TON can support up to 4.3 billion workchains, although its current implementation includes only the masterchain and base chains. This innovative architecture showcases TON’s potential in high-load, high-concurrency environments, driving the broad adoption of blockchain technology.

Source: Shards | The Open Network

Future Research Directions

Potential Developments in Sharding Technology

• Cross-Chain Compatibility: With advancements in sharding technology, interchain communication will become increasingly critical, especially as the demand for exchanging information and assets between different blockchain networks grows. Future sharding technology may further integrate cross-chain communication protocols, such as Polkadot’s Relay Chain and Cosmos’s IBC, to enable seamless interactions across shards and chains.
• Enhanced Security through Shard Governance: Dynamic shard adjustments and flexible governance mechanisms will become focal points of future research. Sharded chains still face challenges in balancing security and decentralization. Emerging security models, such as economic incentive mechanisms and shard validator sharing, will be explored to reduce the risk of attacks on sharded chains.
• Integration with Privacy Protection: The combination of sharding and privacy protection will be crucial in data-sensitive applications. Technologies like zero-knowledge proofs and Trusted Execution Environments (TEE) may become integral parts of sharding, ensuring data security as sharded chains scale.

Potential Integrations and Innovations in Other Blockchain Architectures

• Hybrid Architecture Innovations: Future blockchain architectures may combine multiple technologies, such as integrating sharding with DAG (Directed Acyclic Graph) or multi-layer blockchain architectures. Multi-layer chains can leverage master and side chains to achieve more efficient data sharding and cross-chain expansion. For example, the master blockchain could focus on security and consensus, while side chains handle more flexible shard processing.
• Adapting to Quantum Computing: As quantum computing advances, blockchain architectures will increasingly consider quantum compatibility. The computational and encryption advantages of quantum computing could potentially enhance sharding efficiency. At the same time, precautions must be taken against quantum threats to current encryption algorithms, particularly in inter-shard communication and validation mechanisms.
• AI-Driven Intelligent Shard Management: Artificial intelligence and machine learning could be applied to automate and optimize sharding networks, especially in shard load prediction, traffic forecasting, and dynamic shard adjustments. In the future, AI-driven shard management will enable blockchains to adaptively optimize resource allocation, improving overall network efficiency and user experience.

Conclusion

Sharding technology divides blockchain networks into multiple independent and parallel “shards,” effectively reducing the load on individual nodes and enhancing transaction processing capabilities. It is becoming a core focus in empowering the blockchain field. From Ethereum 2.0’s Danksharding to TON’s infinite sharding paradigm, an increasing number of blockchain networks are exploring and implementing sharding technology to meet the growing demand for transaction throughput. Meanwhile, challenges such as cross-chain compatibility and data availability have fostered new technological innovations, enabling collaboration and asset flow between different blockchains.

However, the implementation of sharding technology is not without challenges. Issues such as security, data consistency, and the efficiency of cross-shared communication require further breakthroughs. Looking ahead, sharding technology will continue to drive blockchain toward a new era of high performance and widespread application. As the technology matures, sharding architectures will become more flexible and secure, supporting more decentralized applications (DApps) and financial innovations, ultimately bringing greater sustainability and innovation to the global blockchain ecosystem.