1:4 Good Architecture
As fascinating as the consensus layer of the distributed ledger stack can be, consensus is only a small fraction of a much larger architecture distributed ledgers need to deliver on their lofty promises.
Architecture in computer science has a surprising amount in common with architecture in the physical world.. as all buildings (or pick your complex systems metaphor: human body, ant colony, etc.) consist of networks of interconnected sub systems that must interact for the larger system to survive.
In this oversimplified model, we'll say buildings engage with four primary systems:
The public infrastructure such as the sewer system and electrical grid so the building can connect with the outside world.
The load bearing foundation upon which everything in the building is stored.
The "smart" systems of the building including electrical, climate control, & security sub systems that perform some type of useful logic.
The user interface of the building which occupants interact with including the light switches, finishings, and furniture.
In computer science, the architecture works largely the same way.
The public infrastructure is a series of interconnected protocols called "the internet" that allows for communication between buildings within a city.
If we view a building as an "application" (or a unified assembly of parts designed to accomplish some task) the foundation of the application consists database layers where all information created within the application is ultimately stored. Within this database layer, distributed ledgers an an additional layer of complexity with consensus mechanisms that traditional centralized database do not have.
Sitting on top of the database layer are the intelligence layers, which in distributed ledger speak is known as "smart contracts". In chapter 1 we defined as the programmatic logic that allows users to sell Beanie Babies, replace Facebook, etc.
The interface layers then sit on top of both the database and smart logic which gives lay users the ability to interact with the system without needing to program.
Keys & Membranes
These four basic levels of any complex system (be it building or application), must be able to selectively interact with each other for the system to function. In other words, we need semi-permeable membranes that can decide when to open themselves up, and when to close themselves off from the outside world.
For our building to function, we need to somehow separate it from the outside world, or else anyone could walk in at any time and potentially destroy the structure.
This is why we invented keys. Not only can we have keys to the main door, but within the structure we can have keys to protect the critical sub systems that only credentialed maintenance workers have access to.
A rough sketch of how a building fits inside a larger system looks something like this:
Where each system is protected from the others by a series of keys that wall of access unless granted to keep each individual sub system secure.
Cities represents the protocol layer allowing the building to communicate with other buildings via plumbing, electrical lines, and roads.
Bridges between cities represent technologies like atomic swaps and other cross chain systems that allow protocols to interact with each other.
Buildings represent the database layer where everything is stored and defines the boundaries of what exists inside the building, and what exists outside.
Plumbing represents a smart contracts that provides some functionality to the building
Flushing the toilet represents a user interface interaction with the system.
In distributed ledger speak the diagram looks like this:
Well run cities attract businesses, which in turn attracts talent, which drives the entire local economy. In the quest to attract the best people, cities are most concerned with creating a fertile ground in which newcomers feel welcome by tweaking the knobs such as:
Tax rates: How much rent is extracted use use the roads, plumbing, etc. based on usage? Tax rates = network transaction fees, or the cost incurred each time a record on the application is sent to the protocol for validation and storage.
Building codes: What rules do developers need to follow when creating new buildings? Building codes = The programming languages used, and ways smart contract programs are constructed and executed on the network.
Development constraints: How much land is available to build on, and how much does extra does it cost to scale to larger developments? Development constraints = Network bandwidth and storage needed to run applications at scale.
Bridges: How easy is it to travel to other cities? Bridges = programmatic tools in the protocol layer that allow tokens from other platforms to be exchanged for tokens on the native platform
Governance: How stable is the governance structure, and what is the process to change the rules? Governance = The rules of consensus outlined in the previous chapter, and methods for voting in new network parameters. More importantly, good governance determines if people trust the system or not. If users think a hard fork is just around the corner and the project will be abandoned, the value of the project sinks.
When you purchase a cryptographic token representing a portion of the base level protocol, in effect you are purchasing ownership in the collective rules that govern how the city operates.
Let's unpack this idea by trying and answer the question:
Where is amazon moving its headquarters?
If you knew the answer to this question, you could make a considerable fortune speculating on the price of land.
The same is true when buying platform level distributed ledger tokens.
The "bet", in simple terms is which ecosystem is going to attract the most talent, development, and adoption.
The winning city will most likely need:
Low taxes rates: The cost for performing transactions on the network is very low and does not inhibit the ability to effectively do business.
Flexible building codes: programming language allow programmers to build applications of any complexity, and make development easier and more intuitive.
Low development constraints: If the application is successful, the protocol needs to support massive scalability in the quantity of data stored on the protocol as well as the speed at which new information can be stored and retrieved.
Good bridges: that allow the protocol to talk to other protocols, etc.
Good governance: Trust in the people maintaining order in the system.
While these characteristics can create the fertile ground development, only developers creating useful applications will encourage end users to move to the platform, which in turn will spur more applications to move there.
This classic chicken and egg problem is called a "Network Effect" which was turned into simple mathematical logic in the 1950s called "Metcalfe's Law". This law states the value of any network is the square of the number of active users.
While not always exactly an exponent of 2, this basic trait is borne out in all modern networks. It makes logical sense that having 5 telephones in a small town vs 10 telephones in a small town would allow exponentially more people to communicate with each other.
Network effects are The Ultimate Incentive Knob
Cities and DLT protocols are successful because of the same network effects that make app stores and existing centralized social networks successful. More people move there because more people are already there, or conversely fail to take off as people move away and abandon the platform.
The effort and cost required to migrate from one City/App Store/DLT Protocol to another are known as switching costs. Network effects in the distributed ledger protocol space are much more fickle than in Cities as the switching costs are much lower.
For a company to move from one city to another takes considerable effort and expense, however for an Application to switch from say the Ethereum network to the NEM network takes considerably less effort.
Less effort but not 0 effort, changing platforms is still a major feat for any application to do which brings up two primary problems.
Migrating the Code Base: This is the effort and cost for project developers to learn how a new platform works.
Migrating the User Base: This is the effort to get your users to move to the new platform. To migrate a snapshot of current account balances need to be taken for all current users so they will have same percentage of tokens on the new network. This can add significant risk to early stage startups as changing such a fundamental fabric of the system early on can send the wrong message to investors that the company did not do its due diligence beforehand.
In a competitive landscape, dozens of platform level protocols compete everyday for projects to launch on their platform for a variety of cost, ease of deployment, and network effect reasons.
The line between the incentives offered by a platform such as low fees, mining rewards, fast execution, and security, from the technical underpinnings that allow such benefits to happen is tricky to separate.
While the goal is to keep the technical and the socio-cultural incentives separate between parts of the book, at some point they run together.
Closer to the Truth
We hope the cities analogy helped, but the truth is the technical architecture that allows distributed ledgers to work differs slightly from the more simplistic architecture presented above. A more complete picture of this development stacks needs to be broken out by the primary set of users that interact with each layer.
End Users: Incentivized by using the products
User interface layer - The pretty surface layer that allows users to interact with the application via a friendly UI which does not show what's going on beneath the iceberg. To end users, only the value they perceive they get out of the service matters, not the elegance of the underlying architecture.
dApp Developers: Incentivized by making the products
Smart Contract Layer - The first logic layer where the useful work happens bebit a calculator, or a mortgage application automator. Within the smart contract layer there is a massive body of computer science best practices for good architecture including class based approaches to prevent excessive resource usage by efficiently calling only what is needed to execute each specific part of the code. Most developers wanting to build something useful will use prewritten tools called libraries to help them create their smart contracts. In fact, terms like "ERC20 standard" in Ethereum are really just a set of programmatic best practices used to keep developers out of trouble by giving them a set of pre written instructions to make building new products easier.
Protocol Developers: Incentivized by making the products that allow developers to make the products
Virtual Machines, low level execution layers, APIs, Bridges... - The translators that turn smart contract code into a format that can be executed on a global network of distributed machines. This is where the critical heart of each project lies along with the consensus mechanisms, and elliptic curve cryptography that allow the network to process, validate, and store transactions.
Hosts: Incentivized to store copies of everything end users want to store for all time
Storage layer(s) - The actual databases where information is stored on a redundant distributed ledger for all time, or until the chain is abandoned.
Hardware Manufacturers: Incentivized by selling physical devices that support all of the above layers
Raw metal - The interaction the bytecode translated by the virtual machine has with the raw distributed server hardware that must work on any machine on the network regardless of what hardware configuration or supported server operating system the ledger happens to reside on.
Within this stack the virtual machine and storage layers are the most controversial as they do the heavy lifting for the entire system. In the next chapter we will discuss the virtual machine layer in detail, but first we need to address the philosophical trade offs within the storage layer.
Court System, hidden costs, and dueling philosophies
In the last chapter we briefly mentioned the big blocks vs lightning network scaling strategies. These two examples illustrate the fundamental difference between a uni-cast network and and multi-cast flood network.
In the early days of the internet, when someone wanted to send a file across the network, all peers needed to replicate the original data for the file to transfer (E.g., a flood network)
If the internet operated this way today, it would instantly grind to a halt as every tweet or Amazon purchase would need to be passed on to every other person in the network.
Hence today we use the TCP protocol to uni-cast our communications directly from one party to another without the need for this replication.
The distributed ledger world is currently in a philosophical quandary about what deserves to be replicated for all time on-chain, and what can safely happen off-chain.
Strategy 1: Keep everything on-chain. This strategy seeks to use ever falling data storage costs to host redundant copies of the entire ledger as it grows over time.
Strategy 2: Use the chain as a court system where the base level ledger is only used when disputes arise between parties. (E.g., a lightning channel is disputed and collateral locked up on the base level chain is transferred from Party A to Party B. Otherwise, if there is no dispute ,the transaction happens without being registered to the chain)
Keep in mind any consensus mechanism type (they-are, you-elect, you-are) can in theory operate either with an all on-chain, or multi level on chain + off-chain strategy.
The history of the original internet developing from Ethernet flood networks into TCP uni-cast networks points towards a multi layer strategy as the most likely step forward.
As long as the network provides incentives for a distributed enough base of users to host full copies of the base ledger, it is too early to rule out which strategy will succeed.
The important part here is that storing data onto a redundant global ledger is not free. The cost to post a transaction to an on chain ledger is not just the first time cost for the miner to validate the transaction, but the recurring cost of keeping the transaction in the ledger for all time by everyone continuing to host a copy of the full ledger.
The beauty of a backend revolution is much of what goes on beneath the iceberg is magically invisible to the end user.
Take converting a .JPEG image file into .PNG image for example. To accomplish this task, your computer needs to interact with all five levels of the development stack.
As an end user, you only see the drop down menu to select .PNG and hit submit.
But beneath you someone needed to write the photo editing software.
Beneath them someone needed to write the operating system.
Beneath that someone needs to store the results. (In this case it's your local hard drive)
And finally beneath all that someone needs to manufacture the hard drive, cpu, RAM, etc. that allow the physical transitor gates to open and close rendering the correct 1s and 0s.
Why should converting one token on protocol ledger X to another token on protocol ledger Y be any different?
Imagine the following use case.
Xoana (our hero from the foreword) wants to pay for 10 gigabytes of storage to back up her Harvard graduation pictures.
She paid for a 1 year supply of VPN tokens so she could access the internet freely outside of Venezuela, but now safely in the United States she no longer needs her VPN tokens.
Just like converting a .JPEG to .PNG, she can seamlessly convert her VPN tokens to Storage tokens, then upload her photos.
Beneath the user interface tip of the iceberg, a ballet of electrons takes place in milliseconds across the globe.
A application on the smart contract layer submits a sell order to swap her VPN tokens for storage tokens
The protocol layer the VPN tokens exist on initiates an atomic swap by first locking her VPN tokens, then releasing them when the Storage token protocol confirms their side is also locked.
The swap happens and the ledger storage layers of both the VPN protocol and the Storage protocol update their ledgers.
Physical hard drive space is unlocked by Xoana new storage tokens allowing her access to 10 gigabytes of sharded storage space across a global network of shared hard drives.
If this sounds like science fiction, it is not. While clunky, this entire use case is possible today. Scaling up, it's possible to see a new type of internet evolving out of this primordial soup.
Over the short term, choosing platforms with fast and loose governance, limited scalability, and high transaction costs can work when the value of applications built on the protocol is speculative, and real world usage is yet to happen.
However, protocols with good architecture will by design win out over the long term as applications begin run into the limits bad architecture imposes on successful applications, forcing developers to switch to better platforms.
In our final chapter of part I, we will dig deep into raw mathematical logic that determines if a protocol can be provably trusted to work the same way every time.
Without this core mathematical trust, no system should be trusted to execute tasks as important as organ donor matching, real estate title transfers, prison release schedules, etc.