1:4 Good Architecture
As fascinating as the consensus layer of the distributed ledger system can be, consensus is only a small fraction of a much larger architecture (in computer science lingo called “stack”) distributed ledgers need to deliver on their lofty promises.
Architecture in computer science has a surprising amount in common with architecture in the physical world.. as all buildings (or pick your complex systems metaphor: human body, ant colony, etc.) consist of networks of interconnected sub systems that must selectively interact for the larger system to survive.
In this oversimplified model, we'll say buildings engage with four primary systems:
The public infrastructure such as the sewer system and electrical grid so the building can connect with the outside world.
The load bearing foundation upon which everything in the building is stored.
The "smart" systems of the building including electrical, climate control, & security sub systems that perform some type of useful logic.
The user interface of the building which occupants interact with including the light switches, finishings, and furniture.
In computer science, the architecture works largely the same way.
The public infrastructure is a series of interconnected protocols called "the internet" that allows for communication between servers (individual or clusters of computers). Regardless of the hardware architecture of the servers, database languages, operating systems, etc. along long as each server can agree on basic packet switching standards, communication between any two machines globally is possible in nanoseconds.
If we view a building as an "application" (or a unified assembly of digital code parts designed to accomplish some task) the foundation of the application consists database layers where all information created within the application is ultimately stored. Within this database layer, distributed ledgers add an additional layer of complexity with consensus mechanisms that traditional centralized databases do not have.
Sitting on top of the database layer are the intelligence layers, which in distributed ledger speak is known as "smart contracts". In chapter 1 we defined as the programmatic logic that allows users to sell Beanie Babies, replace Facebook, etc. Once some piece of logic happens like If user clicks like then display like to other users, this record must be stored somewhere so it can be queried later.
The interface layers then sit on top of both the database and smart logic which gives lay users the ability to interact with the system without needing to program anything. Interfaces can be as simple as a command line terminal, all the way to swiping right on Tinder. As dumb monkeys, we tend to think in terms of what can see, but as we now know there is a massive iceberg sitting beneath user interfaces. When you flush a toilet (a user interface of sorts), your waste does not magically disappear, but is handled by a series of sanitation systems until hopefully it is remediated and dumped back into the local water supply.
As we live in the future, even sanitation systems generate enormous amount of digitizable data on water quality, flow rates, quantity of chemicals used, new species of bacteria and other organic life present in the waste. Even if there is no convenient interface to look at and analyze this data, it does exist and needs to be stored securely somewhere.
Keys & Membranes
These four basic levels of any complex system (be it building or application), must be able to selectively interact with each other for the system to function. In other words, we need semi-permeable membranes that can decide when to open themselves up, and when to close themselves off from the outside world.
For our building to function, we need to somehow separate it from the outside world, or else anyone could walk in at any time and potentially destroy the structure.
This is why we invented keys. Not only can we have keys to the main door, but within the structure we can have keys to protect the critical sub systems that only credentialed maintenance workers have access to.
A rough sketch of how a building fits inside a larger system looks something like this:
Where each system is protected from the others by a series of keys that selectively guard access to keep each individual sub system secure.
Cities represents the protocol layer allowing the building to communicate with other buildings via plumbing, electrical lines, and roads.
Bridges between cities represent technologies like atomic swaps and other cross chain systems that allow protocols to interact with each other.
Buildings represent the database layer where information is stored and defines the boundaries of what exists inside the building, and what exists outside.
Plumbing represents a some kind of logic that provides functionality to the building by performing useful work.
Flushing the toilet represents a user interface interaction with the system.
In distributed ledger speak the diagram looks like this:
Well run cities attract businesses, which in turn attracts talent, which drives the entire local economy. In the quest to attract the best people, cities are most concerned with creating fertile ground in which newcomers feel welcome. Cities accomplish this by tweaking knobs such as:
Tax rates: How much rent is extracted use use the roads, plumbing, etc. based on usage?
In a DLT system tax rates = network transaction fees, or the cost incurred each time a record on the application is sent to the protocol for validation and storage. Viewed more abstractly, tax rates can also be levied on applications within the protocol such as an Uber/Lyft-like ride sharing app. The “tax” paid to shareholders and employees comes out of the service fee they charge to drivers and passengers in the system. Of course, part of this tax is also paid to the lower level protocol that facilitates the IT infrastructure.
Building codes: What rules do developers need to follow when creating new buildings?
Building codes = The programming languages used, and ways smart contract programs are constructed and executed on the network. The most important part of building codes is SAFETY (eg we don’t want things like poor locks on our doors to open up liability when there is a break in)
Development constraints: How much land is available to build on, and how much does extra does it cost to scale to larger developments?
Development constraints = Network bandwidth and storage needed to run applications at scale. If the bedrock the city is built on is too deep and developers choose to develop on shaky ground, the applications they create can potentially topple when what they depend on to perform lower level computations fails.
Bridges: How easy is it to travel to other cities?
Bridges = programmatic tools in the protocol layer that allow data (tokens, metedata, etc.) from other platforms to be exchanged for data on the native platform. Bridges have the very difficult job of properly translating data from one system to another. If you send your Bitcoin to the Ethereum network and receive one Bitcoin equivalent token, how do you know you can send it back?
Governance: How stable is the governance structure, and what is the process to change the rules?
Governance = The rules of consensus outlined in the previous chapter, and methods for voting in new network parameters. More importantly, good governance determines if people trust the system or not. If users think a hard fork is just around the corner and the project will be abandoned, the value of the project sinks.
When you purchase a cryptographic token representing a portion of the base level protocol, in effect you are purchasing ownership in the collective rules that govern how the city operates.
Let's unpack this idea by trying and answer the question: Where is amazon moving its headquarters?
If you knew the answer to this question, you could make a considerable fortune speculating on the price of land.
The same is true when buying platform level distributed ledger tokens.
The "bet", in simple terms is which ecosystem is going to attract the most talent, development, and adoption.
The winning city will most likely need:
Low taxes rates: The cost for performing transactions on the network is very low and does not inhibit the ability to effectively do business.
Flexible building codes: programming languages and toolsets that allow programmers to build applications of any complexity, and make development easier and more intuitive. Instead of needed to reinvent the wheel each time, APIs are available that show how to easily perform pre-written functions such as sending payments, collateralizing assets, etc.
Low development constraints: If the application is successful, the protocol needs to support massive scalability in the quantity of data stored on the protocol as well as the speed at which new information can be stored and retrieved.
Good bridges: that allow the protocol to talk to other protocols, etc.
Good governance: Trust in the people maintaining order in the system with proper game theoretic incentives to keep bad actors out.
While these characteristics can create the fertile ground development, only developers creating useful applications will encourage end users to move to the platform, which in turn will spur more applications to move there.
This classic chicken and egg problem is called a "Network Effect" which was turned into simple mathematical logic in the 1950s called "Metcalfe's Law". This law states the value of any network is the square of the number of active users.
While not always exactly an exponent of 2, this basic trait is borne out in all modern networks. It makes logical sense that having 50 telephones in a small town vs 10 telephones in a small town would allow for exponentially more network interconnections between users.
As we will discuss further in Part II, developers may choose to build on open DLT protocols over closed FAANG systems as there is a history of centralized platforms stealing all of the valued created for themselves after initially using developers to jump start adoption. (See Farmville, Zenga, etc.)
Cities and DLT protocols are successful because of the same network effects that make app stores and existing centralized social networks successful. More people move there because more people are already there, or conversely fail to take off as people move away and abandon the platform.
The effort and cost required to migrate from one City/App Store/DLT Protocol to another are known as switching costs. Network effects in the distributed ledger protocol space are much more fickle than in Cities or centralize protocols as the switching costs are much lower.
For a company or individual to move from one city to another takes considerable effort and expense. Real physical expenses like diesel to power the moving trucks, and inefficiency expenses like closing costs on a new home, or losing governmental benefits that do not transfer across state lines.
Migrating the movie you purchased from the Amazon platform to a competing platform like Netflix is next to impossible. Not because there is any physical or longstanding inefficiency barriers, but because each platform is trying to become an anti-capitalist monopoly (Thiel’s “unfair advantage”)
However, for an Application to switch from say the Ethereum network to the NEM network takes considerably less effort. Less effort but not 0 effort, changing platforms is still a major feat for any application to do which brings up two primary problems.
Migrating the Code Base: This is the effort and cost for project developers to learn how a new platform works.
Migrating the User Base: This is the effort to get your users to move to the new platform. To migrate a snapshot of current account balances need to be taken for all current users so they will have same percentage of tokens on the new network. This can add significant risk to early stage startups as changing such a fundamental fabric of the system early on can send the wrong message to investors that the company did not do its due diligence beforehand.
In a competitive landscape, dozens of platform level protocols compete everyday for projects to launch on their platform for a variety of cost, ease of deployment, and network effect reasons.
The line between the incentives offered by a platform such as low fees, mining rewards, fast execution, and security, from the technical underpinnings that allow such benefits to happen is tricky to separate.
We hope the cities analogy helped, but the truth is the technical architecture that allows distributed ledgers to work differs markedly from the more simplistic architecture presented above. A more complete picture of the development stack needs to be broken out by the primary set of users that interact with each layer.
End Users: Incentivized by using the products
User interface layer - The pretty surface layer that allows users to interact with the application via a friendly UI which does not show what's going on beneath the iceberg. To end users, only the value they perceive they get out of the service matters, not the elegance of the underlying architecture.
dApp Developers: Incentivized by making the products
Smart Contract Layer - The first logic layer where the useful work happens bebit a calculator, or a mortgage application automator. Within the smart contract layer there is a massive body of computer science best practices for good architecture including class based approaches to prevent excessive resource usage by efficiently calling only what is needed to execute each specific part of the code. Most developers wanting to build something useful will use prewritten tools called libraries to help them create their smart contracts. In fact, terms like "ERC20 standard" in Ethereum are really just a set of programmatic best practices used to keep developers out of trouble by giving them a set of pre written instructions to make building new products easier.
Protocol Developers: Incentivized by making the products that allow developers to make the products
Virtual Machines, low level execution layers, APIs, Bridges... - The translators that turn smart contract code into a format that can be executed on a global network of distributed machines. This is where the critical heart of each project lies along with the consensus mechanisms, and elliptic curve cryptography that allow the network to process, validate, and store transactions.
Hosts: Incentivized to store copies of everything end users want to store for all time. These are the IT professionals or eventually even end users that support the network with hard drive space, bandwidth, and compute resources.
Storage layer(s) - The actual databases where information is stored on a redundant distributed ledger for all time, or until the chain is abandoned.
Hardware Manufacturers: Incentivized by selling physical devices that support all of the above layers
Raw metal - The interaction the bytecode translated by the virtual machine has with the raw distributed server hardware that must work on any machine on the network regardless of what hardware configuration or supported server operating system the ledger happens to reside on.
Within this stack the virtual machine and storage layers are the most controversial as they do the heavy lifting for the entire system. In the next chapter we will discuss the virtual machine layer in detail, but first we need to address the philosophical trade offs within the storage layer.
Court Systems, Hidden Costs, and Dueling Philosophies
In the last chapter we briefly mentioned the big blocks vs lightning network scaling strategies. These two examples illustrate the fundamental difference between a uni-cast network and and multi-cast flood network.
In the early days of the internet, when someone wanted to send a file across the network, all peers needed to replicate the original data for the file to transfer (E.g., a flood network)
If the internet operated this way today, it would instantly grind to a halt as every tweet or Amazon purchase would need to be passed on to every other person in the network.
Hence today we use the TCP protocol to uni-cast our communications directly from one party to another without the need for this replication.
The distributed ledger world is currently in a philosophical quandary about what deserves to be replicated for all time on-chain, and what can safely happen off-chain.
Strategy 1: Keep everything on-chain. This strategy seeks to use ever falling data storage costs to host redundant copies of the entire ledger as it grows over time.
Strategy 2: Use the chain as a court system where the base level ledger is only used when disputes arise between parties. (E.g., a lightning channel is disputed and collateral locked up on the base level chain is transferred from Party A to Party B. Otherwise, if there is no dispute ,the transaction happens without being registered to the chain)
Keep in mind any consensus mechanism type (they-are, you-elect, you-are) can in theory operate either with an all on-chain, or multi level on chain + off-chain strategy.
As long as the network provides incentives for a distributed enough base of users to host full copies of the base ledger, it is too early to rule out which strategy will succeed.
The important part here is that storing data onto a redundant global ledger is not free. The cost to post a transaction to an on chain ledger is not just the first time cost for the miner to validate the transaction, but the recurring cost of keeping the transaction in the ledger for all time by everyone continuing to host a copy of the full ledger.
The beauty of a backend revolution is that much of what goes on beneath the iceberg is magically invisible to the end user.
Take converting a .JPEG image file into .PNG image for example. To accomplish this task, your computer needs to interact with all five levels of the development stack.
As an end user, you only see the drop down menu to select .PNG and hit submit.
But beneath you someone needed to write the photo editing software.
Beneath them someone needed to write the operating system, whether it is a virtual machine or a regular computer.
Beneath that someone needs to store the results. This could be your local hard drive, a FAANG data center, or a DLT.
And finally beneath all that someone needs to manufacture the hard drive, cpu, RAM, etc. that allows the physical transistor gates to open and close rendering the correct 1s and 0s.
Today, Converting one token on protocol ledger X to another token on protocol ledger Y is technically possible, yet is not built out to consumer internet (swipe right for sex now) standards.
Imagine the following use case:
Xoana (our hero from the foreword) wants to pay for 10 gigabytes of storage to back up her Harvard graduation pictures.
However, months earlier she paid for a 1 year supply of VPN tokens so she could access the internet freely outside of Venezuela.
Now that she is safely in America, just like converting a .JPEG to .PNG, she can seamlessly convert her VPN tokens to Storage tokens.
Beneath her user interface tip of the iceberg, a ballet of electrons takes place in milliseconds across the globe.
A application on the smart contract layer submits a sell order to swap her VPN tokens for storage tokens
The protocol layer the VPN tokens exist on initiates an atomic swap by first locking her VPN tokens, then releasing them when the Storage token protocol confirms their side is also locked.
The swap happens and the ledger storage layers of both the VPN protocol and the Storage protocol update their ledgers.
Physical hard drive space is unlocked by Xoana new storage tokens allowing her access to 10 gigabytes of storage space across a global network of shared hard drives.
If this sounds like science fiction, it is not. While clunky, this entire use case is possible today. By analyzing early proto-applications like Filecoin/Siacoin/Burstcoin (shared storage) and Mysterium (decentralized VPN), it's possible to see a new type of internet evolving out of this primordial soup.
Over the short term, choosing platforms with fast and loose governance, limited scalability, and high transaction costs can work when the value of applications built on the protocol is speculative, and real world usage is yet to happen.
However, protocols with good architecture will by design win out over the long term as applications begin run into the limits bad architecture imposes on successful applications, forcing developers to switch to better platforms.
In our next chapter, we will explore how an actual computer science technique used in database architecture for decades holds one of the major technical keys to unlock good architecture. When trying to maintain a shared global truth, scaling an immutable flood network can become extremely difficult.