Introduction
In a traditional decentralized storage network, two parties (sometimes a single computer that does two jobs) collaborate to ensure data is stored and available to the network. In this model, one party (the provider), stores the data and reports to the second party, the validators (sometimes called miners), in charge of paying the first party for doing their job. This two-party system allows for a functional trustless environment without the interference of a centralized party. More formally, these validators are part of a consensus, in the case of Jackal, the Tendermint Consensus Layer. This allows every validator in the group to ensure they all report the same information and provide a single decentralized truth. Providers will then be able to submit a mathematical proof to the network of validators, which will independently verify the validity of the proof and come to a consensus, ensuring a majority of validators agree on the outcome.
The Scaling Problem
This consensus can take a long time, sometimes taking multiple seconds or even hours for a proof to be verified. To ensure the network is scalable, the consensus engine often limits how much data can be processed at a single time. We call this a block. On the surface, limiting how much data can be processed at a single time poses a theoretical limit to how much data the network can reasonably handle.
This scalability problem becomes harder to manage when we want the storage providers to prove large files quickly. All storage proofs exist on a sliding scale of speed to size; the bigger a proof is, the quicker it is to create, and vice versa. To jam as many proofs into a single block as possible, it is ideal to have small proofs. However, the problem with this is that making many slow proofs can slow down providers, and limit how many files they’re able to keep at a time.
For a storage network to scale, it must be able to handle many files at a time, both from the storage provider, and the blockchain consensus.
How Do We Prove Data Now?
Files are proven very similarly across decentralized storage networks via Merkle Trees. Merkle Trees are a hashing technique that allows us to prove a tiny piece of a file belongs to a greater set of pieces that make up a whole file. From the Ethereum website, we can see that each piece–denoted by a letter–is concatenated with the next piece in the set and put through a hashing algorithm. These subsequent hashes are recursively hashed together until all that’s left is a single hash result known as the root.
We can use this root along with a subset of nodes in this generated tree to create a proof that a piece belongs in this set. In the following diagram, also from the Ethereum website, we can see that one only needs both the piece itself and the green outline nodes to prove the piece is a child of the root. In decentralized storage networks, we can store the root of the file on the blockchain and periodically post proofs of random pieces of the file, essentially proving we have the whole file.
Scaling Solution in Web3
When trying to create a scaling solution for storage networks, it is best practice to look at storage systems and other domains in the blockchain space, such as Layer-2s and ZK-Snarks. We can start with the most obvious improvement: ZK-Snarks. In fact, Filecoin already uses ZK-Snarks to prove files to their network. In theory, snarks should enable us to prove to the network we have a file, without disclosing any information about the file itself. Furthermore, we do not need to provide the details about the Merkle Tree proof. In theory, this proof should be incredibly small, however with a few caveats, this proof technique requires very small leaf sizes resulting in the generation of proofs taking a very long time.
What we want is a solution that is fast without requiring a great deal of block space. Another route would be taking inspiration from Layer-2s. According to Binance, a Layer-2 is a framework for building networks where blockchain transactions take place off-chain and the state of the secondary network is periodically committed to the underlying blockchain. What if there was a way to not only create but verify proofs off-chain and then commit an attestation that the proof is valid, not the proof itself? In theory, this has two major advantages: the proofs can be quickly generated, and will not take up block space.
Layering Consensus
To scale effectively, we need to verify proofs off-chain. However, with no consensus layer like Tendermint to ensure a trustless environment, handing off the proof to a single party creates a significant security vulnerability. Therefore, We must create pseudo-consensus layers, or ‘consensus spheres,’ that rely on the underlying Tendermint consensus layer every time we want to verify a proof.
To do this, we can create a system where storage providers can request a new attestation form from the blockchain. This form is linked to a storage deal on-chain, and once the form has been 2/3 filled, it will automatically report to the blockchain that the deal is verified. Following Tendermint’s lead, we can assume for the sake of security, that at least 2/3 of the network is playing fair and responding accurately. The consensus sphere life cycle starts with the creation of the form with a random subset of providers. The provider that requested the form generates a proof for the file deal, and sends it to all of the providers on the form, expecting 2/3 of the providers to verify the proof using queried data from the blockchain and respond. Once the form is complete, the form is deleted, and the consensus sphere closes.
By creating many of these spheres simultaneously, we create parallel and overlapping consensus spheres that scale horizontally to offload the block space and proving computer power from the blockchain to the storage providers.
Security Concerns
When attempting an attack on the network, having control over 2/3 of the storage provider network is a surefire way of circumventing the security of the network. To combat this, we can add collateral when instantiating a new provider, creating an economic barrier of attack and ensuring the price of creating over 2/3 of the network’s worth of providers becomes more costly than a potential gain from executing a successful attack.
Conclusion
Less is more, especially when it comes to block space. With providers both creating and verifying files stored on the network, we push the bounds for what is possible in the realm of scaling decentralized protocols. Not only do parallel consensus spheres create scaling opportunities for the Jackal Storage network, but they may also have uses far beyond storage networks, something we are excited to explore.
References
