Keep the privacy issues aside for this post. If you don’t think of ChatGPT as a web app with an LLM but as an actual assistant, it makes sense that it would remember all this. A personal assistant who remembers details about your life is way more valuable than an assistant with amnesia.

Many people think that an AI assistant with a memory will lock you in, like streaming services do by offering a personalized experience. It isn’t easy to move away from Spotify if you have years of listening history. No other music streaming service could instantly provide you with the same level of music discovery and recommendations. A year ago, I unsubscribed from Spotify, hoping that I could simply use YouTube to listen to music. Every piece of audio content on Spotify must also be on YouTube. The truth is that I still miss Spotify and might resubscribe at some point.

Alok’s question on how on defensible is AI memory

However, I don’t think this is true for AI assistants. The previous generation of machine learning systems were task-specific, such as recommending movies, music, etc. Today’s transformer based models can learn anything just from a stream of tokens. Imagine exporting your chat history from ChatGPT to Claude. Now Calude will have the same memories. The lock-in effect is not comparable to what we had in the Web2 services.

It opens up a whole new design space for stateless agents. Imagine an AI assistant that you can spin up on demand, feeding it only the memories you explicitly store and manage yourself. Think of it like entries in a personal journal. Each time you interact, the assistant reads from your journal, provides insights or summaries, and then writes the outcome back into your journal. After your interaction, you shut the assistant down completely. Users can truly own their interactions when AI models can seamlessly import, export, and interpret standardized data formats. This approach lets you retain complete control over your data, prevents you from getting locked into a particular AI assistant provider, and, crucially, it doesn’t have to compromise the UX.

Imagine a future with a feed of AI agents, each providing access to specialized tools and the latest AI models. To interact, you’d simply prepare your prompt, provide your personal memory journal, and enjoy a seamless experience. This portability doesn’t dilute the AI’s effectiveness.

Moreover, specialized, task-focused AI agents could leverage your journal entries for highly contextual interactions. Need a financial planning session? Spin up an agent explicitly trained for the financial domain, temporarily granting it access to relevant sections of your journal. Let the agent update your accounting books. Finished? Shut it down.

2022 was also the year when I took a sabbatical. I was wondering what to work on next. Sometime around June, I got in touch with the Textile team to work on their IPFS tooling as a part-time contractor. Later that year, I decided to join the team full-time to work on Tableland. Before the bubble popped, NFT projects were thriving, and Tableland was starting to get some usage.

this is fine

Let me briefly describe Tableland at this point. In a nutshell, it is an off-chain database that is driven through onchain events. Every table is represented as an NFT onchain. Any wallet with adequate permissions can write to the table. All tables are public for anyone to read. You could simply spin up a Tableland node, and your database will be synced in a few minutes.

The Tableland node builds on top of SQLite and listens to EVM events. Every insert or update SQL query is sent to a specific contract. The underlying EVM blockchain determines the order of events. So, there is no need for a consensus protocol at the Tableland layer. The node simply extracts the queries from the event stream and materializes them into its local database.

Tableland Architecture

The NFT projects liked Tableland because it enabled them to store and update the NFT metadata in a structured way. Tableland also has sophisticated ACL features. For instance, you could restrict a user to write on a table based on which NFTs they hold or if they are part of a DAO or whatnot. However, no one actually used these features to their full potential.

After the bubble popped in late 2022, most teams working on such NFT projects vanished and never returned, even after the market conditions improved in late 2023 and early 2024.

During this time, some DePin projects expressed interest in using Tableland. However, their requirements were different. They wanted an open database with a high write throughput but didn’t care about table schemas or structured data.

Tableland’s write throughput is fundamentally bound by the block time of the underlying blockchain. This inspired us to work on an open object storage system. This time, we built using Tendermint with a bespoke validator set. It is a sidechain that checkpoints to Filecoin. Essentially, you can think of it as Filecoin layer 2.

We initially called it Basin but later renamed it Recall. In the last few months, as we have seen AI agents take off, we are exploring the possibility of using Recall to store agents’ context windows. This would make the context window shareable in a permissionless way, making AI agents composable.

I will write more about it in the future as the protocol evolves. Meanwhile, you can take a look at the docs and feel free to reach out to me on X or Farcaster.

I

The first post describes how to extract Write Ahead Logs from a Postgres instance, sign them with an Ethereum wallet, and send them to a remote server for replication onto another Postgres instance. The remote server verifies the signature to ensure that the database change is authorized and materializes the WAL message into a row within a table.

II

The second post accomplishes the same goals as the first but uses a different approach. Instead of using Postgres’ logical replication protocol, we use DuckDB with the postgres_scanner extension. In this approach, we perform a table scan instead of streaming Write Ahead Logs and produce a materialized parquet file as the output. This file is the point-in-time copy of the original database. While DuckDB does all the heavy lifting, this approach has its limits. Scanning the table too often is resource-intensive, especially if the table is large.

III

In the third post, we modify the approach described in the first. We materialize the WAL locally instead of sending the signed updates to a remote server. We use a DuckDB instance to replicate the source Postgres database locally. At regular intervals, we export the state of the local DuckDB as parquet files, sign it, and send it to the remote storage of persistence.

IV

The fourth post describes the same thing as the third, but I implemented the prototype in Rust this time. I found that the libraries implementing Postgres’s replication protocol are less mature than in Go. Rust libraries are more low-level and lack documentation. At the time of the writing, the rust-postgres project has a few open Pull Requests to enable working with logical replication, but these PRs have been open for a long time. However, Materialize’s fork is stable and offers these features.

V

The fifth and sixth posts are about ECMH and its applications. ECMH stands for Elliptic Curve MultiSet hashes. It is a cryptographic hashing scheme with set semantics. The output of the usual hash function is a digest that looks like a string of random characters. The same applies to ECMH hashes, but they have an additional structure. ECMH hash digests represent a set. You can perform set operations on these, such as inserting or deleting elements. I describe implementing an ECMH-based hashing scheme in Go using the go-ristretto library in the last two posts.

VI

The sixth and the last post describes an application for ECMH. I used set-based hashes to fingerprint a directory in the file system. The fingerprint provides data integrity. By keeping track of the 32-byte fingerprint, we can tell if any of its files are tampered with.

In the first part, we will unpack the Finite Field arithmetic. We will implement basic operations such as addition, subtraction, multiplication, and modular inverse. In the next part, we will implement elliptic curve operations for the signature scheme. The original paper used C. I have instead used Rust. However, the following code snippets only use rudimentary language features in order to focus on lower level operations. Needless to say, this code is only for illustration purposes and should not be used in production.

Before we begin, we should take note of two different data types used here. A Field element is represented as a byte array of 32 bytes in the packed form. In the unpacked form, it will be represented as an array containing 16 elements, each 16-bit wide. The size of a Field element is 256 bits in both representations.

Note: we will use an i64 type for each element in the unpacked array. The remaining bits will be left empty initially. This extra space ensures that we are not risking overflows when performing additions and multiplications

The field struct

We will perform arithmetic operations such as addition, multiplication, and inverse in the unpacked form and finally output our result in the packed form to save space.

The most important thing to remember is that any algorithm we implement must be constant time (i.e., O(1)) relative to the input size. Constant time algorithms are more performant and less vulnerable to attacks. If the execution time of cryptographic operations depends on its inputs, then it can be analyzed and reverse-engineered. A cipher system using such algorithms can be broken, leading to dire consequences.^[1]

Addition and Subtraction

Addition and Subtraction implementations are straightforward. We iterate over our Field array and perform an addition or subtraction operation at every index.

At this point, we will ignore the possibility of a “carry” or a negative result (where the most significant bit is 1). Remember, we are using an i64 array to represent one Field element. We can count on the unused bits to handle these scenarios. Later, when we convert the result to the packed format, we will deal with the carry and negative results.

add and sub methods for finite field elements

Carry

Before we get to multiplication, we should consider handling the carry bits. In the unpacked format, each array element is an i64 integer. Out of which, we have only used 16 bits.

Here is how we will handle carryovers. We iterate over the items of the Field element array and divide each integer by . There is a carry if the division doesn’t make all the bits 0. We move the carry over to the subsequent element.

carry method used in addition and multiplications

If we find carry bits in the last (15th) element, we add that back to the first element, but only after multiplying the carry by 38. The number 38 seems arbitrary. We must look at the multiplication algorithm to understand its origin. So, stay tuned.

Calling this function once can cause the element at the first index to exceed or even become negative (due to a wrap-around). A second call can fix this. However, if items between the 1st and 15th index are close to , the carry bits from the 0th index can cause a cascade of carries. It will result in a further carry from the last element to the first, causing it to exceed again.

We must call the carry function three times to ensure no carry bits are left in the Field element. Such a carry cascade is no longer possible on a third call since the middle values are now zero.

Multiplication

Now we are ready to look at multiplication. In the Ed25519 curve, the parameter is set to . It is the prime modulo of the Finite Field underlying elliptic curve^[2]. In practice, all the arithmetic operations on the Field are done modulo .

Since is , therefore is . There are two points to note here.

First, the size of our data structure is . It means that reducing any value by modulo is easy. Bigger values will automatically overflow.

Second, in the multiplication function, we can get away by reducing the result by modulo as long as we later reduce it by . We will do so while packing our result into a byte array before returning the final result.

We start the multiplication by taking the product of inputs (in the unpacked form) element by element. We store the individual results in the product array so we can add that up to get the final result. This representation is equivalent to a reduction modulo or .

product calculation expressed algebraically

After doing some modulo arithmetic, as shown in the figure above, we can arrive at the equation . Note how the higher powers of are reduced because of the modulo in step 2. Finally, the reduced powers are grouped in step 3. The same thing is implemented below on line 63. We can now ignore the elements between product[16] and product[30] in our initial array.

Finally, we call the carry function twice. It is good enough to account for carry bits and ensure each array element doesn’t exceed the 16-bit size.

The multiplication method for finite fields

Fermat’s Little Theorem

In Finite Field arithmetic, in place of division we have the modular inverse. The modular inverse of is if by multiplying with in the given modulo, we get (multiplicative identity of the field).

Fermat’s little theorem says the following. Here, is a prime and is any integer.

For our purpose, . We can use the formula to find the inverse of a given field element.

Modular inverse

Using the theorem, we can see that:

Essentially, we need to calculate where is the given field element. And is . In hex, is represented as:

bytes in i.e. shown in hexadecimal

The hex notation makes it easy to see that besides the first and the last byte, all other bytes are ff (i.e., all 8 bits of a byte are 1). If we look closer at the first and the last byte, we notice that 0x7f = 01111111 and 0xeb = 11101011. It means that all the bits of are 1 except the 2nd, 4th, and the most significant bit.

Here is the inverse algorithm. We iterate in a loop times and maintain an accumulator in the result variable. We will initialize this accumulator with the input (line 91).

The inverse method for finite fields

In every iteration, we will square the accumulator and reassign the value. Further, if the bit is set for an index (that is, every index except the 2nd and 4th), then we also multiply the accumulator with the input value.

To understand how the algorithm works, we can take a smaller example. Let’s say we want to calculate the modular inverse of 3 in the prime field where is 13. In this case, and . The binary representation for 11 is 1011. We will initialize the result with 3, skip the most significant bit, and iterate 3 times for the remaining bits: 011.

Mod inverse table

Note: all the multiplications in the above table are in mod 13. We can quickly verify our result in a Python shell

Mod inverse in python

With that, we are done with the arithmetic operations. Next, we will focus on transforming between the two data types we discussed earlier.

Unpacking

As the name suggests, the unpack function will convert a packed representation (byte array) to an unpacked one. This is straightforward. We start by creating a new Field element. Initialize it with 0 values. Next, we iterate through all 16 elements and update them. We combine two adjacent bytes from the packed byte array into a 16-bit value. To do so, we multiply the 2nd byte by and add them together.

Unpacking the byte array to operate on field elements

Finally, we make sure that the most significant bit is zero. We do this by taking the last element modulo , since, in our prime field arithmetic, every number is less than (i.e., ).^[3]

Bit Swap

bit swapping

Before we start with the pack function, we need a swap function. It will take two Field elements and swap their bits only if the respective bit of the third argument is 1.

Notice the i64 variable c. Due to the bitwise negation, it will have all its bits set to 1 when the 3rd argument (b) is 1, and all the bits will be 0 when the b is 0. We can print it out to see the memory representation. ^[4]

bit pattern for variable c in bitswap

The bitwise XOR operation will only result in 1 when both of its inputs are different. For each item in the array, we will initialize a temporary variable (line 106). Each bit in this variable will only be set if the corresponding bits of the 3rd argument are set (bitwise and), and the bits of p and q are different (bitwise xor).

Hence t will capture the differences in the bits of p and q. Its bits will be set where the bits of p and q don’t match. Finally, if required, the bitwise XOR with t on lines 107 and 108 will flip the bits of p and q.

Packing

Packing the Field element array back to the byte array is tricky. We start by first ensuring that none of the elements overflow. We do this by calling the carry function three times, as discussed before.

Recall that earlier, we only reduced our calculations by modulo . It means that the input value in the pack function can lie between and . The next step should be to reduce the input value by modulo . Before we do that, notice that only three possible cases exist. The input value can lie within one of these buckets: , , . In the first case, we can pack the input value. In the latter two cases, we must first subtract the input value by and , respectively.

We want a constant time algorithm. It cannot depend on the size of the input (i.e. result of our calculation). To find the correct bucket we will subtract by two times from the result and store our result in an intermediate variable. If the value becomes negative after the first subtraction, we will do the second subtraction anyway but pack the original value (discard the result of the subtract operation). If it becomes negative after the second subtraction, we will use the first subtraction’s result (discarding the second subtraction’s result). Otherwise, we will pack the output of the second subtraction.

Packing the result into a byte array

We can tell a value is negative if the last item of the field element array is wider than 16 bits. Similar to how we calculate carry bits.

Here, we will use the swap function. If the value is negative (carry = 1), we will not swap the intermediate result of the subtraction in temp with the input. If the value becomes negative in first iteration, it will also be negative after the second iteration. We won’t swap again and pack the input. However, if the value is not negative after the subtraction, we will change the original value with the result of the subtraction and proceed with the second subtraction.

Conclusion

With all these functions in place, we have completed the finite field arithmetic required for understanding elliptic curve operations. In subsequent posts, we will use these concepts to implement the Ed25519 signatures.

Notes

1. This book has an excellent account of the impact of using and breaking cipher systems throughout history (see the code book)

2. Check out chapters 4 & 5 from the moon math manual to better understand fields, groups, rings, and elliptic curves.

3. Bitwise & operation is equivalent to taking mod of a power of 2.

4. Check 2s complement binary representation.

CRDTs are shared data structures with superpowers. They enable independent modification and still stay in sync. It means that if two users, Alice and Bob, decide to change a key in a CRDT-based hash map, they don’t need to coordinate beforehand. Alice can change the value of key “foo” to 1, while Bob can make it 3. Eventually, when Alice and Bob sync, it is guaranteed that neither of their changes will be lost. Moreover, the shared data structure will not be corrupted. It will eventually converge to a single value that Alice and Bob agree on.

This post will look at the theoretical underpinning of CRDTs. We will implement two different CRDTs: GCounter (or Grow only counter) and LWW set (Last Writer Wins set). We will also analyze their applications, compare them with the main alternative: Operational Transformations and understand their limitations.

CRDTs Theory

Thanks to order theory, CRDTs are robust to data corruption amid concurrent edits and converge to a canonical value over time. There are a few concepts at work here. First is the notion of order. Second is the concept of a join or merge. The third is semilattice.

Order

An order is a binary relation. For example, we can say, in the set of integers 2 < 3. The less than operator here is the order. An order could be total or partial. A less than operator is a total order over a set of integers.

A totally ordered set of integers

An example of a partial order is the “located-in” relation. Consider a set {USA, NYC, Germany, Berlin}. NYC is located in USA, and Berlin in Germany. We cannot compare USA and Germany with the located-in operator. Neither USA is located in Germany, nor Germany is located in the USA.

In an ordered set, an upper bound is an element greater than or equal to any other element. The join or merge operation on two elements is the least upper bound in such a set.

For example, in a set of integers, the join of 2 and 5 will be 5. There’s no integer smaller than 5 that’s an upper bound on both 2 and 5. 7 is an upper bound on 2 and 5, but it’s not the least upper bound since 5≤7.

Recall that the less than and equal operator is a total order for a set of integers. A curious property of total order is that joining two elements will result in one of the two elements. It is because every element can be directly compared to every other element.

It is not valid for a partial order. Let’s extend our set of locations from above: {North America, USA, NYC, SF, Europe, Germany, Berlin, Munich} Here, the join operation on “NYC” and “SF” cannot produce either “NYC” or “SF” because the two are incomparable. The “USA” and “North America” are the upper bounds for “NYC” and “SF”. “USA” is the least upper bound. It means that the join operation on “SF” and “NYC” results in “USA”.

A partially ordered set of locations

Join

The join or merge operation in our pre-defined set with an order must follow the three algebraic rules. It should be commutative, associative, and idempotent.

Commutativity means it doesn’t matter in which order we join two elements. It is similar to how it doesn’t matter in which order we add or multiply two integers.

Associativity means if we want to join three elements, we can pick any two at a time and then join the result with the remaining element.

Idempotence means that we can join an element with itself any number of times, and we will still end up with the same element.

Semilattice

A set with an order is a join semilattice if we can take any two elements from the set and find a join for them. It is easier to see it in the case of the total order, for example, the set of integers and the less than and equals operator. For any two elements, the least upper bound will be one or the other of those two integers.

It is not straightforward for the partial order. Again, consider our set of locations. We cannot perform a join on “USA” and “Germany” because there is no upper bound. Both of these countries are incomparable. This set and the corresponding located-in order don’t form a join semilattice. However, if we add “Earth” to this set, it will become a join-semilattice.

Here is another example of the difference between total order and partial order. It is more abstract but more familiar. A binary search tree is totally ordered, whereas a binary heap is partially ordered. In a binary search tree, every parent node must be bigger than its left child and smaller than the right child. This property is responsible for the total order in this data structure. In a max heap, every parent element must be bigger than both of its children. However, the children remain relatively unordered. Both data structures are ubiquitous, but each has a different purpose. A binary search tree can quickly find an element. The time it takes is proportional to the logarithm of the number of elements in the tree. On the other hand, a heap is instrumental in quickly finding the extreme (min or max) elements in constant time.

A partially ordered set of locations

According to Wikipedia, state-based CRDTs:

send their full local state to other replicas, where the states are merged by a function which must be commutative, associative, and idempotent. The merge function provides a join for any pair of replica states, so the set of all states forms a semilattice. The update function must monotonically increase the internal state according to the same partial order rules as the semilattice.

It raises a question. How CRDTs deal with a partial order? How can two replicas be merged that cannot be compared? This is, in fact, the most intriguing case when concurrent modification leads to conflicts.

This scenario is analogous to “merge conflicts” in git. When two developers modify the same line of code in different commits, there is a conflict when they merge their code. Generally, developers resolve conflicts offline following a specification.

Similarly, concurrent edits create a conflict in a CRDT. For example, if Alice and Bob are writing a letter to Charlie. They might start at the same time. Alice might type the first line as “Hi Charlie”. Bob, in his editor, may type it as “Hello Charlie”. In this case, the CRDT-based text editing software won’t be able to automatically converge on a value. In such cases, the end-user has to intervene. Alternatively, applications may use a heuristic. For example, the editor can produce the first line as “HiHello Charlie”. Later, either Bob or Alice can fix the word “HiHello” to something sensible.

The values in CRDTs increase monotonically according to some order. CRDTs diverge until they eventually converge. Having briefly looked at order theory, we can see that CRDTs converge mathematically towards the least upper bound of all possible values.

Intuitively we can see that the state is merging. As the nodes sync, they update their local value based on the value they see in the received message. The final shared state is equivalent to the result of some correct sequential execution.

The most fascinating bit is that CRDTs converge to a consensus value without using any consensus algorithm like Raft, Paxos, PBFT, etc. It means that any new state update can execute immediately. Network latency or faults won’t slow down the system.

Types of CRDTs

There are two types of CRDTs: state-based and operation based. Both are theoretically equivalent. However, when it comes to implementation, they offer different trade-offs.

The state-based CRDTs are easier to implement. They do not require any guarantees from the underlying network. The problem is that they are generally inefficient. The entire state of every replica is eventually transmitted over the network.

The operation-based CRDTs only transmit the update operation and hence consume less bandwidth. However, such CRDTs require a reliable underlying network. The operations must be kept from being dropped or duplicated and delivered in a causal order.

CRDT Examples

Some examples of CRDTs are: G counter, PN counter, G set, 2 Phase set, Last Writer Wins set, OR set, etc. For collaborative text editing use cases, generally, sequential CRDTs are used as described in RGA and WOOT, etc. Below are simple implementations of a G counter and an LWW set.

G counter

A grow-only counter is the simplest CRDT. It can only increase in one direction. A use case for such a data structure could be to track the number of “plays” for a video or a song.

A Gcounter implementation in Go

The counter is implemented as an integer-to-integer hash map. The map has an entry for every other replica that this copy knows about. When an actor hears from another, it merges their value into the counter using the merge function.

The Merge function iterates through the received counter and updates the keys if it is not present in the current replica or its value is lower than the received value. The maximum observed value is chosen because that is the least upper bound on a set of integers.

The Value function returns the current value of the counter. The point to note here is that the system’s value is the sum of all observed counters. It makes sense. Ideally, the value should be the total number of increment operations that all the replicas in the system have seen in aggregate.

source: http://jtfmumm.com/blog/2015/11/24/crdt-primer-2-convergent-crdts/

One might find it more intuitive to implement a counter as an integer instead of a hash map, but that design has a flaw. Imagine this scenario. A system has 3 replicas of a counter with values 3, 2, and 1. It means that the system, in total, has seen 6 increments. When node 1 syncs with node 2, both will update their counters to the maximum of each other’s value. Their new shared value will become 3. When node 3 syncs with the other two, it will update its counter value to 3. This is a problem. We are expecting the counter to be 6, not 3.

LWW set

Let’s look at another CRDT, the Last Writer Wins Set. The semantics is as the name suggests. The set allows you to add or remove any item.

Under the hood, such as set maintains two different sets. One for additions and the other for removals (called the tombstone set). An item is a member when the following two conditions hold. One, it is in the “add” set. Second, it is not in the remove set or was added in the remove set before it was added in the “add” set.

LWW set implementation in Go

Such a set can have an “add” or a “remove” bias. When the timestamps are equal, bias resolves the conflict. Note that a timestamp collision is likely to happen after two replicas merge.

Operational Transform

CRDTs offer an approach for achieving eventual consistency in distributed systems. They are designed to handle conflicts between concurrent updates to the same data, ensuring that all replicas eventually reach the same state. It makes CRDTs a handly tool to implement collaborative software such as text editors. However, it is crucial to note that the current state of the art collaborative editors, such as Google Docs, does not use CRDTs. Instead, it uses Operational Transform algorithms. Both Operational Transform and CRDTs-based algorithms strive to converge independent data modifications.

The Operational Transform algorithm, in a nutshell, works by sending the indices at which the edit was made to other nodes. The received index must be transformed if the local copy has that index occupied. Index transformations happen in a central server. The server keeps the authoritative copy of the data that users are working on.

Operation Transforms are battle tested. However, they are less flexible when it comes to network architecture. It relies on a centralized server for sequencing edits to prevent unexpected data races. CRDTs don’t share this limitation because they are a local data structure. Software built on CRDTs can talk to other instances over a p2p connection. No coordination is required.

OT vs CRDT network topology

The central Operational Transform server can become a performance bottleneck when a large number of users use it at the same time. On the other hand, writing p2p applications may be challenging. Also, notice that p2p networking is not required to use CRDTs. It works as well in a client-server architecture.

CRDT Limitations

CRDTs guarantee convergence, but not without cost. One of the main issues with CRDTs is that they could get memory hungry. Based on the examples discussed above, one might have spotted a pattern. These data structures only accrete. For example, a GSet keeps adding new elements without ever deleting anything. A Two-Phase set uses a tombstone set to record retractions.

CRDTs need to store the history of all edits. Garbage collection is not trivial. It is impossible to know when a disconnected node might come back online and wants to merge all the changes to arrive at the current system state.

There are some heuristics for garbage collection, but only with trade-offs. For instance, such GC techniques might require synchronization, which hurts performance. It is hard to implement CRDTs-based algorithms in resource-constrained environments such as edge computers, IoT devices, etc.

In Closing

In this post, we explored Conflict-Free Replicated Data Types (CRDTs), shared data structures that enable independent modifications while remaining in sync. Using order theory concepts, CRDTs ensure convergence to a single value over time without using consensus protocols. We also looked at some simple CRDT implementations, such as GCounter and LWW set, and discussed their applications and limitations. CRDTs offer flexibility in network architecture but can be memory-intensive since they store edit histories.

After seven years of heads down building stuff, I took a break at the end of 2021. I planned a full 1-year sabbatical for the year 2022. This post is a collection of projects that I did during that time.

I didn’t intend to finish most of them or make them production ready. Instead, these half-baked prototypes aim to help me build intuition about complex topics. I develop a deeper understanding when I implement the concepts I am learning.

Gradual Dutch Auctions

Nov 2022

Inspired by the Gradual Dutch Auction spec, I wanted to explore the mechanism. Paradigm’s blog post describes two novel auction mechanisms suitable for fungible and non fungible tokens. I quickly implemented and tested the basic functionality thanks to the original post’s precise specification and reference implementations. The blog had all the details about the pricing function. All that was left was to implement the math equations in solidity.

It led to another rabbit hole. That is to understand how math libraries work in solidity, especially WAD arithmetic. I first started using PRBMath to implement Discrete GDA and continuous GDA contracts. Soon after, I familiarized myself with various techniques to implement mathematical functions in solidity. Finally, I replaced the PRB math library with my custom implementation of muldiv, exp, pow etc., for fixed-point numbers in the format 59x18 (using the last 18 digits for decimals). I still need to optimize these functions for gas usage. This may be a topic for a future project.

CRDTs

Sept 2022

The project’s purpose is to go through the CRDTs theory and implements basic CRDTs as specified here.

Currently, this repo contains the implementation for G Counter, PN Counter, G Set, 2PSet, LWW set, OR Set, and an experimental mix of 2P Set with vector clocks.

This repo also has code for vector clocks with basic functionality such as incrementing and merging different clocks, including benchmarking for these operations.

In the future, this will be a playground to experiment with more complex CRDT-based algorithms such as RGA and WOOT. And even more complex sequential CRDTs such as trees and graphs.

Toy EVM

Aug 2022

The goal of this project is to build the EVM from scratch. This project doesn’t aim to be feature complete, performant, or production ready.

The specification in the Ethereum Yellow Paper is extremely rigorous. At the same time, it is a dry read and difficult to understand.

With this project, I want to create a playground to tinker with EVM primitives. The main purpose is to understand the internals and the core concepts underpinning the EVM.

Solidity Contracts

June 2022

This repo is a collection of solidity contracts. The main purpose of this project is to be a playground for experimenting with different defi primitives on EVM based chains.

The contracts are compiled and tested using hardhat. Typechain is used to generate Typescript types for the solidity contracts, which makes it convenient for writing tests and scripts.

Staking Pool is a contract that allows users to deposit and withdraw ETH or native tokens of other EVM chains. It also allows an admin to add funds to this contract to distribute the rewards to incentivize the depositors. The rewards that an individual depositor receives are proportional to their deposited stake.

The Staking Token contract is similar to Staking Pool contract it but also mints ERC20 tokens for the depositors. Later, the depositors can later redeem their sTokens for the native tokens they deposited plus the rewards accrued.

Damn Vulnerable DeFi CTF

May 2022

Damn Vulnerable DeFi is a set of challenges for practicing the security aspect of smart contracts.

The challenges demonstrate using DeFi primitives like flash loans, price oracles, governance, lending pools, etc. Most puzzles expose a vulnerable protocol design or an implementation issue.

These puzzles are more advanced than the Ethernaut CTFs. My goal in practicing these challenges is to better understand the primitives, common attack vectors, and common solidity mistakes.

While working on these problems, I found that tracing function calls and trying to call the public functions with different values often led to the right idea. In particular, I often followed these steps:

• Understand the problem well enough to be able to draw a diagram to trace all interactions, such as function calls and event emissions.

• Drawing the call stack for successive function calls.

• Deploy the contracts locally and add logs, especially around constraints (require statements).

Ethernaut CTF

Feb 2022

Ethernaut CTF is the collection of “capture the flag” puzzles. These are targeted towards solidity or EVM beginners. Most of the puzzles in this collection are inspired by actual exploits that happened in the wild.

Each puzzle exposes some idiosyncratic behavior in the EVM. Smart Contracts in EVM are a different paradigm; not understanding how EVM works often leads to writing vulnerable contracts.

My goal in solving these puzzles is to familiarize myself with common gotchas in the ethereum virtual machine. I also want to understand common attack vectors in smart contracts.

In this post, we will look at the problem of time measurement and clock synchronization. We will understand why keeping clocks in sync is the paramount issue in distributed systems. We will also look at the challenges of synchronizing the clocks across the network. We will briefly cover the different types of clocks and standards for measuring time and their inconsistencies.

Finally, we will end by taking a look at a couple of widely deployed clock synchronization protocols. How they tackle the clock synchronization problem and the different trade-offs that they make.

Timekeeping

Before we look at how computers tell time, we will briefly examine how time is measured. In general, there are two broad types of clocks: astronomical clocks and atomic clocks.

The stability of a clock means how consistently it measures a unit of time.

Astronomical clocks are instrumental in applications related to navigation and astronomy because they match the earth’s rotation. However, they are less stable. Atomic clocks are more stable but don’t precisely match the earth’s rotation.

More stable atomic clocks tend to be expensive, bulky, and hard to maintain. The usual wall clocks and computers that use quartz crystal oscillations are more prone to drifting. Environmental factors such as room temperature affect how fast quartz crystal oscillates. Quartz clocks can easily drift by 50 microseconds per second (50ppm).

NIST’s ultra-stable ytterbium lattice atomic clock. Credit: Burrus/NIST

Unix Time

Generally, computers keep time using the Unix time standard. It is defined as the number of seconds elapsed since the Unix epoch. The Unix epoch is 00:00:00 UTC on 1 January 1970. According to Unix time, each day contains 86400 seconds. The UTC standard is based on atomic clock reading, but it also periodically adjusts itself to sync with the earth’s rotational period.

As measured in UTC, a day could be longer than 84600 seconds. The speed of the earth’s rotation slows down gradually over time. Hence an astronomical day can have more than 86400 seconds when measured from an atomic clock.

To compensate for this drift, UTC has a standard practice. It occasionally adds a leap second to UTC time at midnight. The leap second was introduced in 1972. Earth’s rotation speed also varies in response to climatic and geological events. That is why UTC leap seconds are irregularly spaced and unpredictable.

When the leap second is added, the UTC time goes from 23:59:59 to 23:59:60 and then finally to 00:00:00. After accounting for a leap second, a UTC day can be 84600 or 84601 seconds long. Another problem is that there is no way to represent the 60th second that UTC introduces in the Unix time. The Unix time simply repeats the last second to work around this limitation. The Unix time goes from 23:59:59 to 23:59:59 and finally to 00:00:00 to account for a leap second.

An alternative to adding leap second is to smear it. This technique recommends slowing down the clock for a longer duration instead of adding a second at one moment, such as midnight. For example, imagine that a leap second is announced for December 2022. In this case, we will slow down the clock at 2022-12-31 12:00:00 UTC and through 2023-01-01 12:00:00 UTC. One advantage of smearing the leap second over a long duration is that it keeps the frequency change small. The change for the smear is about 11.6 ppm. This is within the manufacturing and thermal errors of most machines’ quartz oscillators.

Unix timestamp is essentially a number such as 1664955216. It is amenable to computers because we can represent it as an unsigned integer. However, it is not easily readable by humans.

Another common alternative to Unix time is ISO 8601. It was first proposed in 1988 and has been updated several times. It incorporates the elements of the 24-hour clock, a calendar, and a way of describing the timezone offset. For the notation, ISO 8601 uses Arabic numerals and specifically chosen characters such as “T, W, Z, -, and “: .”Such affordances make this representation more human-friendly. The purpose of ISO 8601 is to provide a straightforward method of representing calendar dates and times for global communication. We can convert between ISO 8601 and Unix time using the Georgian calendar.

Clock synchronization

Most computers don’t have access to expensive atomic clocks. Instead, they rely on their hardware clocks. These clocks generally use humble tools like a quartz crystal for timekeeping. The operating system implements a system clock using the hardware clock. The software clock is set by the hardware when the system boots. The kernel delivers interrupts at dynamic intervals. It keeps the system clock updated.

Clock synchronization is an even more challenging problem for distributed systems. A distributed system is a set of computers that communicates by passing messages to each other and do not have access to a central clock. The individual nodes must obtain some common notion of time. No node will agree on the current time if the local clocks drift freely.

The purpose of designing distributed systems is often to coordinate real-time activities such as running factories, hospitals, and aircraft, recording inventory in databases, maintaining balances in bank accounts, etc. If the system’s clock is not in sync with the real time, none of these systems can function.

Network Time Protocol

NTP or Network Time Protocol is a clock synchronization mechanism. It uses precise GPS receivers and atomic clocks to minimize drift. At this point, NTP is a widely adopted protocol. Most operating system ships with a pre-installed NTP daemon. The computer must connect to a centrally maintained NTP server such as time.apple.com to use it. As per the pre-defined configuration, the NTP daemon polls the time servers regularly to keep their clocks in sync with the network time.

NTP server setting in MacOS

NTP is a UDP-based protocol. It works by relaying timestamps between different time servers across the globe over a packet-switched network like the internet. NTP organizes time servers in layers. Layer 0 only has the devices that keep the most accurate time, such as expensive atomic clocks and GPS receivers. The servers in layer 1 use layer 0 devices to sync their clocks. The servers in layer 2 use the servers in layer 1, and this goes on. NTP’s goal is to reduce the clocks skew as much as possible¹. However, even NTP-enabled servers can skew 100-200 milliseconds in UTC time.

NTP requires a reliable underlying network to function correctly. If the nodes don’t have access to sufficient upstream or downstream bandwidth, the clocks might be off. To work around random network delays, NTP clients usually query multiple servers and poll one server multiple times before resetting their clock.

NTP server architecture

Notice that there is no way to measure the round-trip latency without having access to a global clock. It means that NTP needs to make a reasonable assumption about the network. It assumes that upstream and downstream network latency is the same².

After an NTP client receives the network time, it calculates the skew locally. If the local clock is off by too much, then the client patches its current time. If the difference is insignificant, then the client adjusts the local clock rate to ensure that the local clock can catch up with the network time.

NTP clock skew calculation

Here is another problem. The NTP client can jump to a lower timestamp if the local clock is significantly ahead of the network time. It means that we cannot assume that the timestamps issued in our system are monotonic.

Operating systems generally give access to a wall clock and a monotonic clock. The wall clock can tell the “time of the day” according to the Unix time, but it may jump. The “monotonic clock” doesn’t jump but cannot tell the current time. The monotonic clock returns time passed since an arbitrary point in the past, such as when the machine booted. To calculate the absolute time difference between two instances in a process, use the monotonic clock. The difference between two monotonic clock timestamps on the same device will never be negative³. In summary, the wall clock is for telling time, and the monotonic clock is for measuring time.

Programming languages offer different APIs for monotonic and non-monotonic clocks. For example, in the JVM, the timestamps given by system.currentTimeMillis is not monotonic.

This takes us back to the Cloudflare outage. Before Go version 1.9, the time.Now() function didn’t return a monotonic timestamp. That is why, during a leap second, when Unix time froze for one second, the difference between the two timestamps that were only a few milliseconds apart produced a negative value crashing the application⁴.

Cloudflare DNS outage 2017: root cause

Besides getting access to both astronomical and atomic time, there is another reason to have two different time sources. It makes the system robust. The GPS receivers may have an issue with their antenna, radio interference may block the signal, etc. On the other hand, atomic clocks in the data center may fail in ways uncorrelated to atomic clocks in the GPS satellites and other atomic clocks in different data centers. For example, clocks can significantly drift over long periods due to frequency error.

NTP works well in practice, but it is not perfect. It has its own set of problems. As we saw, NTP can make the clock jump forward and backward. It makes timestamps non-monotonic. Another issue with the NTP protocol is that it is centralized. The “network time” depends on the clocks at the top layer. It makes those clocks in layer 0 a central point of failure. Also, NTP needs a reliable network to function. It cannot reduce the clock drift at a node if it loses the network connection. It may also happen that the NTP client on the node crashes or a firewall accidentally blocks NTP traffic. In such cases, NTP will not work. The system will again suffer from clock skew.

@sadserver on twitter

TrueTime

Google’s TrueTime provides an order of magnitude improvement in clock synchronization compared to NTP. Similar to NTP, TrueTime uses the same time sources. That is a combination of atomic clocks and GPS clocks. However, organized in a different architecture.

TrueTime is built using expensive atomic clocks, GPS receivers, low latency data center connections, and extensive network engineering.

TrueTime timestamps are different. Instead of representing an instance, it represents an interval. This interval is a timestamp with an estimated error margin of calculated real-world time. There is a guarantee that the “actual” time must lie within the given interval.

TrueTime Timestamp

There are two sources of error or uncertainty in the network clock. First, it could be that a particular node knows its clock is drifting and adjusts for the drift as an error. The second is the unpredictable network latency. Notice that sophisticated engineering efforts can reduce the latter to a certain extent.

Google ensures that the network latency is minimized by running TureTime entirely on their private Wide Area Network. That is, it does not rely on other 3rd party internet infrastructure for routing packets because it could be flaky.

Google has atomic clocks and GPS receivers in each data center. To synchronize the clocks on every server within the data center TrueTime classify the servers into two categories. The first type of server has access to high-end timekeeping devices. The other servers do not have direct access to these clocks. They use their own hardware clocks. However, they regularly synchronize their local clocks with the time servers within the same or nearby data centers.

The longer the servers don’t check in with the time servers, the more uncertain the network time becomes. As soon as the nodes synchronize their clocks with the time server, the uncertainty in TrueTime goes down.

Google has managed to keep the uncertainty window low by ensuring non-timekeeping nodes sync with the timekeeping nodes every 30 seconds. The worst-case clock skew in NTP is about 100s of milliseconds, whereas TrueTime promises an uncertainty interval of a maximum of 7ms under normal network conditions.

Source: Martin Kleppmann’s Distributed Systems Lecture Series

It is worth noting that TrueTime doesn’t guarantee the perfect globally synchronized clock. However, timestamps with uncertainty intervals can be as good.

Take the example of Google Spanner. It is a highly distributed ACID-compliant database. The Spanner database was the first substantial use case for TrueTime. Using TrueTime timestamps, Spanner can provide transaction isolation (I in “ACID”) in the following way.

If two timestamps do not overlap, Spanner can be sure that the time on one timestamp is, in fact, earlier than the time on the other timestamp across all nodes in the network. However, if they overlap, Spanner pauses for the entire uncertainty interval. Recall that the maximum uncertainty window is only 7ms. Due to the small uncertainty interval, the database can simply wait before committing a new transaction without considerable performance degradation. It ensures that every node in the Spanner network has observed the first transaction before the second without ambiguity.

A final note about TrueTime is it can suffer from partitions too. Network partitions are not uncommon. During such an event, TrueTime’s uncertainty interval increases. It leads to degraded performance in the systems such as Spanner that rely on TrueTime for clock synchronization. However, these systems can still function correctly with a low throughput until the partition heals⁵.

Conclusion

Computers need access to real physical time for various purposes. Operating systems need to know how long a process has been active for context switching. Databases need to attach a timestamp to the transactions for concurrency control. Applications need access to the current time for logging events and errors for debugging. Applications also need timestamps to record statistics for benchmarking. DNS needs timestamps for renewing the domain mappings. Browsers need to know the current time to validate the TLS certificates. Distributed applications need synchronized time for coordinating real-world activities such as running airplanes, factories, and hospitals.

We started the post by describing the clock synchronization problem in distributed systems. We examined the differences between atomic and astronomical clocks and the inconsistencies in different time measurement standards. We briefly discussed the definition of the most common standard for timekeeping in computers: UTC and Unix Time.

Towards the end, we looked at two clock synchronization protocols: NTP and TrueTime. We compared their performance. We also understood the trade-off space for keeping the clocks consistent across the network.

Notes

1. Clock skew cannot be zero even if there is no drift. It is proved that in realistic networks, even if the clocks all run at the same rate as real time and there are no failures since there is some uncertainty in the message delivery time, it is impossible to synchronize the clocks.

2. If the upstream and downstream latency is the same, then we can calculate the delays as half of the difference between two timestamps (when clients start a request and when it receives a response from the time server).

3. Note that two monotonic timestamps are not comparable if they come from two different devices.

4. The cause of the error was rand.Int63n() function which panics if its argument is negative.

5. This observation is in line with Barbara Liskov’s finding that globally synchronized clocks can improve the performance of a distributed system by reducing communication overhead.

Optimistic Rollup Architecture

Optimistic Rollups sound too good to be true. Without compromising security guarantees, an ORU could scale transactions on a blockchain. It can increase the chain’s throughput by leveraging fraud proofs and off-chain computation.

The supply and demand of blockspace determine the transaction fee for using a permissionless blockchain. Since blockspace is scarce, whales often outbid others. It means that Ethereum could price out many of its most loyal users at a time of high traffic.

By compressing the computation, rollups can fit more transactions in the same number of blocks, reducing congestion and making the blockchain more affordable.

Optimism implements the generic idea of an optimistic rollup on Ethereum. In this post, we will look at basic primitives underlying the Optimistic Rollup design and zoom in on the core components of Optimism’s architecture.

Layer 2 Alternatives

Since the early days of Ethereum, it was clear that 13-second block time is slow. Adjusting parameters such as block size (or block gas limit) only can, at best, be a short-term solution.

Before the Ethereum community aligned itself towards a rollup-centric roadmap, they explored various solutions in parallel. Most notably state channels, side chains, and plasma. Compared to rollups, all of them come with certain limitations.

A state channel is like a bar tab. Two parties can make countless transactions with each other offline. Only the last one, where one party wants to exit, must settle on-chain. It works well for micro-transactions, for example, incrementally paying a storage service to retrieve a large file.

In a State Channel, only a fixed set of parties could transact. New parties cannot be added to the existing channels. Also, for many use cases, it requires high upfront capital.

A side chain is an independent blockchain that routinely checkpoints with the main chain. The side chain is responsible for its own security and hence doesn’t benefit from the already established security faculty of the base chain. This begs the question, how censorship-resistant could a side chain be?

Plasma has a dynamic set of transactors and doesn’t need upfront capital. It is like a side chain that is anchored within Ethereum. Still, it doesn’t inherit the security from the base chain. It is not fully censorship-resistant.

Moreover, channels, side chains, and plasma do not guarantee data availability¹. These solutions put the burden of security in the users’ own hands. Plasma and State Channels require users to store their transaction history. If the system cheats, the user should be able to produce evidence to the main chain to settle the dispute. Side Chains rely on their own validators for security. In a nutshell, state channels are not general-purpose, and plasma and side chains are not secure enough.

Rollups Overview

Rollups are at a sweet spot in the scalability versus security trade-off space. It can inherit the security of the main chain. Intuitively, a rollup is an ordered mempool (an ordered set of transactions) combined with a general-purpose virtual machine. Any computation that EVM can execute, a rollup can run it too.

There are two varieties of Rollups: optimistic rollups and ZK rollups. As the name suggests, ZK rollups are built using Zero-Knowledge magic. Covering them is beyond the scope of this post.

On the other hand, Optimistic Rollups only require implementing the challenge verification game. It builds upon two assumptions.

1. At least one node verifies the computation that the sequencer executes (1:N trust model). In case of a dispute, the verifier generates a fraud-proof.²

2. The underlying blockchain is secure and decentralized. That means it is resistant to censorship and block reorganizations.

Traditional side chains are independent blockchains but commit the block hashes to the main chain. A rollup is a blockchain that is tightly coupled with its main chain. A valid rollup block eventually ends up on the main chain as long as it is not under a Sybil attack.

Rollups publish the block data on-chain, but transactions remain unverified for some time (a trial window, for example, 2 weeks). During this time, blocks are tentatively accepted.

The sequencer “rolls up” the computation required to execute transactions and publishes their results on-chain. The overall throughput increases because every other node on the main chain doesn’t have to re-execute the unrolled transactions to verify it.

A tentative block is up for a challenge by an independent verifier. If the challenger could prove, within the trial window, that a particular block is indeed invalid, it will earn a bounty³. At the same time, the sequencer (block producer) has to pay the penalty.

Unlike the other layer 2 solutions, rollups publish the entire transaction data on-chain to ensure “data availability”. Storing raw data on the blockchain is cheaper than performing computation over that data. Optimistic Rollup implementation such as Optimism and Arbitrum optimally compresses the transaction data, so they don’t have to overpay for L1 gas.

Bridging

SOL tokens in Uniswap on Ethereum

A rollup has a separate blockchain. It is important to note that most users only care about applications and their assets. A rollup is only helpful if it seamlessly interoperates with the underlying chain. An end-user should be able to bring their assets to the rollup chain and withdraw back to the main chain. The rollup and the application deployed on it should support common uses like lending, borrowing, trading assets, voting on proposals, etc.

In the context of blockchains, bridging means locking up an asset on one side and then minting a proxy representation on the other. The proxy token can trade on the other blockchain like any other token and remains redeemable for the original asset. For example, the Wormhole bridge between Ethereum and Solana enables defi users to trade a representation of the Solana token with any other Ethereum based token on Uniswap.

The security of cross-chain bridges is a complicated matter on its own. It is well known that the current bridge architectures are not secure enough relative to the money they carry. Unlike cross-chain L1 to L1 bridging, L1 to Rollup bridging is safe against issues related to consensus because of shared security. However, users are still vulnerable to the risks associated with execution while bridging their assets from L1 to a Rollup.

Million Dollar Bug Bounty

In February 2022, Saurik uncovered a critical bug in Optimism’s execution environment (OVM). Which led to a well-deserved bug bounty of more than $2m! This bug would have allowed a malicious hacker to print an unlimited amount of “bridged” Ethereum tokens (OETH). Fortunately, it was never exploited.

One of the reasons why this bug was lurking in the system was that Optimism’s execution environment was not fully compatible with Ethereum’s. This event motivated the team to double down EVM equivalence. Since then Optimism team has addressed the issue. It will be deployed in their latest bedrock release.

EVM Equivalence

The execution layer is the core of a rollup. Rollups can scale transactions because it relaxes the requirement of every network participant having to run each piece of computation. Instead, it allows a dedicated node (the sequencer) to execute transactions and create a proof. Every node in the network can verify that the block producer is not cheating. Also, a rollup must submit all the transaction data on-chain. If the sequencer disappears, another node can take over.

Notice that Ethereum doesn’t require that rollup’s execution environment comply with the EVM. The only requirement is that the Rollups post the compressed transaction data on the chain. However, the sequencer can execute code in any programming language or VM if it produces the required chunk of bytes.

Fuel is a perfect example. Similar to Optimism and Arbitrum, it is an Optimistic rollup. However, it has its own VM that can execute transactions in parallel. Offers a UTXO-based transaction model similar to bitcoin. Provides a Rust-inspired programming language called Sway that promises a safer contract development method.

With that said, we can now turn to the numerous benefits that EVM equivalence offers. It can prevent compatibility-related bugs like the one described above. Also, EVM equivalence can offer a better developer experience in different ways.

Contract developers can reuse their existing code. They don’t have to re-write their applications from scratch while migrating from Ethereum to Optimism. While doing so, they also gain access to many mature Solidity or Vyper focussed tools, libraries, and frameworks.

In recent years the security community has developed a set of best practices for auditing solidity-based contracts. It is hard to find reputed auditors to review the solidity code. With an unfamiliar language, it is going to be even more complicated.

Optimism Architecture

We don’t need to go into tiny details at this point. In the following paragraphs, we will examine a few essential components of Optimism’s architecture. That should be enough to understand the whole system at a higher level.

Canonical Transaction Chain

The CTC is a contract on Ethereum L1. It contains an append-only log of transactions (the L2 blockchain) ready to be applied to the current L2 state.

It includes two important data structures (queue and batches) and two methods (enqueue and appendSequencerBatch) we should zoom in on.

enqueue:

Canonical Transaction Chain contract

As the name suggests, this method adds transactions to the queue. These transactions are not batched yet. There is no restriction on who calls this it. If the transaction is enqueued, the sequencer is bound to roll it up.

There is a possibility that the sequencer could become malicious. It may start to censor users’ transactions. In such a scenario, a user can force the sequencer to include their transaction by directly adding it to the queue.

However, we should note that when the sequencer adds transactions to the queue, an individual user only pays the amortized gas cost. If an individual directly adds a transaction to the queue, it will be as expensive as an ordinary L1 transaction.

appendSequencerBatch:

appendSequencerBatch method

The appendSequencerBatch method picks up transactions from the queue and rolls them up into a batch. The batches data structure is a ring buffer that defines the order of the transactions. Under the hood, the ChainStorageContainer library implements it.

Only the sequencer can call appendSequencerBatch

Only the sequencer can call this function. Once a transaction is batched, it will eventually be included in the blockchain. However, at this point, it is only tentative. It is not safe to consider it final until the challenge period is over.

Data Transport Layer Service

Data Transport Layer

Data Transport Layer is a typescript service that runs within the Optimism node. Ethereum nodes download new blocks from the peer-to-peer network. In the case of Optimism, the DLT service monitors Optimism’s L1 contracts to download new blocks.

When the DTL detects a new relevant event on the L1 blockchain, it downloads the corresponding transactions. It inspects these transactions to reconstruct the blocks in the standard Ethereum block format.

The L2 blocks are stored in the DTL’s database and exposed via HTTP APIs. sequencers and verifiers can download these blocks and proceed with the execution. After executing, the sequencer proposes the new state by adding it to the SCC contract. If the verifier finds the computation incorrect, it creates a fraud-proof.

SCC

The State Commitment Chain (SCC) is an L1 contract. It contains a list of proposed state roots called batches. Similar to batches in CTC, it is implemented using the ChainStorageContainer.

The batch elements here have a 1:1 correspondence with transactions in the CTC. Each one represents a unique state root. Each state root results from the sequencer applying the canonical transactions one by one over the last batch element.

State roots and Transaction batches

There are three essential methods in SCC.

appendStateBatch:

appendStateBatch method

Anyone can enqueue a transaction in CTC, but only the sequencer can propose a state root in SCC by calling this function. The sequencer asserts that the proposed state root is correct by appending a new state batch.

deleteStateBatch:

deleteStateBatch method

Transactions enqueued in the CTC are valid Ethereum transactions. A sequencer can only cheat by proposing an inconsistent state root (for example, the one where they reward themselves $1B). In that case, a verifier can detect fraud by independently executing the transactions.

If the verifier finds and proves the fault within the challenge window, it can delete the faulty state by calling this function.

The verifier gets a bounty for its service, and the sequencer incurs a penalty for attempting to cheat. The Sequences eventually proposes new batches to replace the deleted state roots.

Notice that the CTC batch doesn’t roll back even after the verifier deletes the corresponding SCC batch. The fault proving-verification game doesn’t alter the order of transactions in CTC.

verifyStateCommitment:

verifyStateCommitment method

Batches in the SCC finalize if they remain unchallenged until the end of the challenge period. This function doesn’t change the state. It only returns true or false. With the given state batch and a Merkle proof, this function verifies the inclusion of a transaction. Any contract on Ethereum can call it to determine the status of any transaction on Optimism.

Standard Bridge

Optimism’s Standard Bridging Protocol

Optimism’s bridge protocol has two central contracts: Layer1StandardBridge and Layer2StandardBridge. The Standard Bridge contracts connect L1 and L2 applications. It is possible to deploy your own bridge. However, Optimism’s bridging protocol provides a standard way of passing messages and assets around. Developers can make asynchronous calls between the two layers. It is worth noting that safely relaying a message from L2 to L1 takes at least 7-14 days (challenge period).

Known Issues, Criticisms, and Caveats

Until now, in this post, we looked at all the benefits that rollups provide. How it can meaningfully increase the throughput of a blockchain without compromising on security. Rollups will play a significant role in making blockchains ready for mainstream adoption. However, in engineering, there are no benefits without inherent cost. This section will look at common issues related to rollups.

The first and most obvious criticism is a high withdrawal time. Withdrawing tokens from an optimistic rollup back to the main chain takes a long time. The 7-14 day withdrawal time is due to the challenge period. The challenge verification game is inherent to the security of an optimistic protocol. Unfortunately, there is no way around it.

Optimistic rollups treat the underlying blockchain as a court. The layer one chain doesn’t have to execute undisputed transactions. It only has to run the disputed ones. The challenge period is crucial because we want network participants to have enough time to gather evidence against the misbehaving sequencer. We want the verifiers to catch a sequencer trying to sneak in an invalid final state.

There also exist cross-rollup bridges. These protocols can significantly reduce users’ withdrawal time without risking their funds. In essence, these bridges use money markets to provide an early withdrawal to a user. In exchange, the user gives the bridge the right to withdraw their original tokens. The bridging protocol takes a small fee for its service. It is worth highlighting that ZK rollups don’t suffer from this limitation. Since ZK rollups rely on validity proofs instead of fraud proofs, there is no challenge verification game and hence no challenge period.

Rollups are not secure against the base chain re-orgs. Such attacks undermine the security of the underlying blockchain. In such an event, there is no guarantee of a canonical transaction ordering. As we saw earlier, rollups (and layer 2 in general) rely on the base layer being secure against chain reorganizations. Economic security is the most crucial feature of a blockchain. Trying to scale an insecure chain is putting more funds at risk.

There are some concerns regarding the centralization of sequencers. Currently, most rollups, including Optimism, Arbitrum, StarkNet, etc., have a centralized sequencer. It is worth emphasizing that a centralized sequencer cannot censor its user. As we saw, users can force the rollup to include their transactions. Liveness remains an issue until rollups move to an architecture with a decentralized sequencer. The centralized sequencer may go down, and a new one may not come online in time. In such cases, the L2 blockchain will stall. A centralized sequencer also gains exclusive access to the MEV. Other network participants may not consider it fair.

A relatively uncommon but relevant concern is rollup token incentive design. Optimism charges its user a gas fee in ETH tokens. However, one can imagine a rollup implementation that uses its own token for execution fee payment. Such an implementation will increase the transaction throughput on the underlying chain and hence deserves revenue.

At the same, its token may compete with the native asset of the layer 1 chain. Such an incentive design could be parasitic to the layer 1 chain. Increased demand for layer 2 tokens in the free market could come at the cost of the L1 native token. This can, in turn, lead to reduced security of a proof of stake-based layer one chain.

There are a couple of UX-related criticisms. For instance, bridging to a different network adds friction to the user flow. Moving tokens to layer 2 could be tedious to users who are used to a smooth 1-click user experience. Also, unlike ZK rollups, optimistic rollups don’t add any feature to enhance transaction privacy. ZK rollups can give privacy as a side effect together with increased throughput.

Conclusion

Gas Fee on Optimism for Uniswap trades and ETH transfers

We started this post by highlighting the supply constraints on the blockspace. We looked at various layer 2 solutions that we have in our toolbox. We briefly went through the security and decentralization properties of each of them. We also analyzed the trade-offs these solutions make to provide higher scalability.

Then we narrowed our focus to Rollups and further narrowed it down to the optimistic variant. We understood the core primitive that optimistic rollups are built on. That is the Fraud proofs and the challenge-verification game between the centralized layer 2 block producer and the independent transaction verifiers.

We briefly looked at what bridging means in the context of blockchains. We also pointed out how Optimism got lucky regarding a critical vulnerability. A compatibility issue between Optimism’s execution environment and the EVM led to a counterfeiting bug. Fortunately, before this vulnerability could be exploited, a whitehat hacker jumped in to save the day. We evaluated Optimism’s decision to pursue EVM equivalence instead of building a new execution environment from scratch.

Towards the end, we zoomed in on a few components in Optimism’s architecture. We saw how different contracts and RPC services interact with each other to implement the complex optimistic rollup design. Finally, we ended the post by looking at some known concerns and caveats related to rollups.

Notes

1. Data Availability is one of the most notorious problems in the blockchain space. It is hard to describe it as a side note. The description deserves an entire post. In summary, it entails preventing attacks where some network participants withhold transaction data to fool participants that don’t keep the whole history of the chain.

2. Verifier’s Dilemma: There is a monetary incentive for independently verifying the rollup computation. However, suppose every network participant thinks that someone else will be vigilant because there is an incentive, and no one actually is. In that case, a malicious sequencer will be able to steal money. Optimistic Rollups are safe if there is at least an independent verifier.

3. A poorly implemented bounty mechanism could cause a security issue. For example, consider this scenario. The sequencer posts a 50 ETH bond upfront to earn the right to produce blocks. Then the sequencer publishes a fraudulent state, and a verifier notices it and publishes a dispute. If successful, the protocol will take away 50ETH from the Sequence’s bond and give it to the verifier. In this case, the sequencer can front run the verifier by publishing a dispute, disputing himself. By doing so, the sequencer can get back its initial capital.

Core Async is a Clojure/Script library for async programming. Tony Hoare’s Communicating Sequential Processes ¹ was influential in its development. It is a tool to model concurrent (not parallel) tasks. Parallel tasks are independent of each other; that is, they do not require coordination. They can be modelled deterministically and executed simultaneously. Concurrent tasks, on the other hand, require coordination among workers ².

Workers are, in the core async context, either producers or consumers. On JVM, these are actual threads or lightweight, cooperatively scheduled threads called go blocks ³. In single-threaded environments such as nodejs or the browser, go blocks are the only choice for workers. Go blocks are scheduled on a fixed-size thread pool. By default, the size of this thread pool is 8.

The golden rule of async programming is: never block the event loop. While working with go blocks, make sure they are free of blocking operations. A blocking operation in a go block will block the underlying executor⁴.

A worker has to wait if the producing or consuming operation is pending. You can choose to implement this wait as blocking or parking. The only way these workers continue execution is when the worker, on the other end, completes a pending operation.

ANATOMY OF A CHANNEL

A channel in core async is like a baggage carousel at the airport. Travellers are the consumer, and airline staff are the producers. If the carousel is full, the staff cannot put any more bags on it. They have to wait until some travellers take their stuff out. If the carousel is empty, the passenger cannot take their baggage. They have to wait for the bags to arrive. Channel, like the carousel, acts as a rendezvous point between the producers and the consumers. Data put on the channel is added to its buffer. Size of the buffer is customizable, but unbounded buffers are not allowed.

Internally a channel maintains three queues. The puts and takes to enqueue all pending put and take operations. And the buffer to hold the data. The put and the take operations use a channel level mutex to make these operations thread-safe. Both put! and take! functions accepts a (callback) handler. It runs when these operations complete. When they complete immediately, then the handler runs on the same thread. Otherwise, it executes asynchronously.

PUT OPERATION

A put operation on the channel completes immediately when there is a pending take operation waiting on the other end or when there is capacity in the buffer. Otherwise, the put operation has to wait in the puts queue. An unbuffered (same as a buffer of size 0) channel can queue only MAX-QUEUE-SIZE put operations. The default value of MAX-QUEUE-SIZE is 1024.

(require '[cljs.core.async :as a]
		  '[cljs.pprint :as pprint])


;; to inspect a channel
(defn inspect-unbuffered-channel
  [ch]
	(pprint/print-table
	  [{:pending-puts  (-> ch .-puts .-length)
		:pending-takes (-> ch .-takes .-length)}]))


;; putting 1025 items in the channel
(let [ch (a/chan)]
  (dotimes [i 1025] ; <== make it 1024 to fix the ERROR
	(a/put! ch i))
	(inspect-unbuffered-channel ch))

Exceeding this limit leads to an exception when the channel uses the default, fixed-size buffer. This Behaviour is, of course, customizable. For example, a Sliding or a Dropping buffers drops the data upon reaching the limit.

For a channel with a buffer of size 1, we can put 1025 values.

;; to inspect channel and its buffer
(defn inspect-buffered-channel
	[ch]
	(pprint/print-table
	 [{:pending-puts  (-> ch .-puts .-length)
	   :pending-takes (-> ch .-takes .-length)
	   :buf-size      (-> ch .-buf .-n)
	   :buf-items     (-> ch .-buf .-buf .-length)}]))


(let [ch-with-buf-1 (a/chan 1)]

  (prn "The first put!")
  (a/put! ch-with-buf-1 :something-useless)
  (inspect-buffered-channel ch-with-buf-1)

  (prn "More puts")
  (dotimes [i 1024] ; <== Make it 1025 to CRASH
	(a/put! ch-with-buf-1 i))
  (inspect-buffered-channel ch-with-buf-1))

The first put! completes immediately filling up the buffer of size 1. The rest 1024 put operations get queued. Adding one more pending put! will lead to an overflow.

A summary of different put! scenarios.

TAKE OPERATION

A take operation completes immediately when the buffer is not empty or when there is a pending put operation on the channel. On completion, the provided handler is run.

In the following example, we can see there is one item in the buffer because put! was completed immediately. Then we take it out of the channel.

;; Channel with a buffer of size 1
(let [ch (a/chan 1)]
  (a/put! ch :the-first-thing)

  ;; print contents of the channel
  (prn "contents of the channel:")
  (inspect-buffered-channel ch)

  (a/take! ch #(prn %)))

When the buffer is empty, in the following example, a take! operation has to wait for the put! operation on the channel to complete. On inspecting the channel, we can see the pending take!. Putting on the channel completes it.

(let [ch (a/chan 1)]
  (a/take! ch #(prn %))

  (prn "contents of the channel:")
  (inspect-unbuffered-channel ch)

  (a/put! ch :an-item))

When a take operation cannot complete, it gets queued up in the takes queue. At max, this queue can accommodate only MAX-QUEUE-SIZE pending take operations. By default, like with puts queue, this value is 1024. Any take operation after that, without a put in the meantime will cause an exception.

(let [ch (a/chan)]
  ;; Take 1024 times on a channel with empty buffer
  (dotimes [i 1024]  ; <== Make it 1023 to fix the ERROR
	(a/take! ch #(prn %)))

  ;; See whats in the channel
  (prn "contents of the channel:")
  (inspect-unbuffered-channel ch)

  ;; This take! will cause an exception
  (a/take! ch #(prn %)))

A summary of different take! scenarios.

CLOSING A CHANNEL

Metamorphically, the close operation is like putting the lid on a container with a faucet. No more stuff can be put into it. But it can be taken out until the container is empty. The close! function closes the channel. After it is closed, it no longer accepts any new put operations.

(def chan (a/chan))

(a/put! chan :foo)

;; closing the channel
(a/close! chan)

;; put! is rejected, returns `false`
(a/put! chan :bar)

Data in the channel after it is closed can be taken until it is drained.

(a/take! chan #(prn "we got back:" %))

A take! operation on a closed channel that is drained completes with nil.

	(a/take! chan #(prn "take completed with:" %))

If there are pending take! operations, on closing the channel, they will complete by receiving nil values.

(let [ch (a/chan)]

  (a/take! ch #(prn "take completed with: " %))

  ;; close the channel without putting anything
  (a/close! ch))

The timeout function returns a channel that closes itself after t milliseconds.

(let [c (a/timeout 1000)] ; <== Make it 3000 & count to 3
  (a/take! c #(prn "Channel has closed returning-> " %)))

The pending take! operation completes after the channel is closed. It receives nil from the channel.

NOTES

1. Communicating Sequential Processes

2. A clear distinction made between concurrency and parallelism in the book Joy of Clojure by Chris Houser and Michael Fogus.

Parallelism refers to partitioning a task into multiple parts, each run at the same time. Typically, parallel tasks work toward an aggregate goal and the result of one doesn’t affect the behaviour of any other parallel task, thus maintaining determinacy. Concurrency refers to the execution of disparate tasks at roughly the same time, each sharing a common resource. The results of concurrent tasks often affect the behaviour of other concurrent tasks, and therefore contain an element of nondeterminism.

1. core.async/go a macro that, through inversion of control, converts synchronous block of code to an async block of code.

2. Go blocks run on a fixed size thread pool. Due to an accidental blocking operation, for example, a blocking HTTP request, inside the go block the thread that runs it will be tied up waiting for this operation to complete. This means this thread is now effectively of the pool. If 8 such blocking operations run at the same time, it can freeze the whole process bringing down the production system (how do I even know that :) )

Let’s take the example of two coin tosses. Let’s label the event of getting heads in the first toss as A and second as B. Then the probability of A and B happening together is defined to be the intersection of A and B:

This can be visualized using a Venn diagram:

Now we reason that since the first coin toss does not impact the second coin toss in any way so the intersection of two probabilities must be the same as their product:

In this case, the probability of getting two heads in a row will be:

So that means:

But this result does not always hold. In fact, it rarely holds outside of trivial textbook examples.

CONDITIONAL PROBABILITY

Imagine you want to analyze the probability for it to rain today, and you already know that it rained yesterday. Intuitively you know that raining a day before makes it more likely to rain on the present day.

It is a conditional analysis. Conditional probability is defined as the probability of event A happening given that event B has already happened. If A and B are two coin tosses as described earlier, then what is the probability of the second coin landing heads if the first has already landed heads? It is straightforward. We know one toss doesn’t affect the other, so it is the same:

But in the case of the rain, as we saw before, raining a day earlier makes it more likely to rain on the present day. That means:

In such a case of dependence, conditional probability is defined as the ratio of the probability of observing both A and B to the probability of observing event B (independent of A).

Represented in the Venn diagram as:

BAYES THEOREM

From set theory, we understand that set intersection is commutative. That is:

In this context, it implies:

This result looks very close to a fundamental theorem of high applicability. Bayes theorem is abstract and counterintuitive. I always found it difficult to grasp it fully. Let alone applying it in real problems. But it can be constructed relatively easily using simple equations as we saw above.

Christopher Alexander, like in The Timeless Way of Building, talks about the design process in the abstract. But the book imparts concrete understanding.

There is an artificial distinction between the two design processes. The selfconscious and the unselfconscious process. Design, be it wireframes, blueprints or software, aims to solve a problem. The solution doesn’t exist in a vacuum. Context is vital. Take an example of the human body. It is a design the evolution came up with. The cells exist and functions within the context of the organisms. And organisms within species.

What is the problem? Here, the definition of the problem is a misfit between the form and its environment. The misfit is not resolved unless the form satisfies its environment and vice versa.

In the unselfconscious process, the designer is just an agent of change. The process is self-organizing. It relies on immediate feedback from the surrounding. Consider how an Eskimo builds an Igloo. If its too warm inside (a misfit!) the designer just digs a hole. He did not do elaborate planning before digging a hole. Contrast this with a Skyscraper. A large team of engineers and architects put a significant effort to deliver a project of such magnificence. We plan it to perfection. In a top-down fashion. And if a fault is detected in the building, we need an expert to deal with it. A layperson has no role here.

Preexisting knowledge and directness of action are essential properties of the unselfconscious process. The Eskimo intuitively knew that he needs to dig a hole. The unselfconscious process of design, unlike architecture or engineering, is not taught academically. The skill is specific to the context. And perhaps, it is passed down as a tradition. Trial and error play an important role here. Repeated trial and error leads to a position of a fit between the artefact and its context. It may take a long time to reach this equilibrium between the environment and the form.

In the unselfconscious culture, change for the sake of change is discouraged. Once the system starts to work, it is left untouched. Until something changes in the context, causing a misfit. A complex system built this way is naturally divided into subsystems such that when one subsystem is modified, there is no cascading effect on others¹. This localizes the error and as a result, make the system, as a whole, more robust.

The process of maintaining and evolving software can be unselfconscious. The project will grow into a robust and functional product if the maintainer fixes bugs and only add features that are absolutely necessary to solve its user’s problems. The user, like the Eskimo from the earlier example, has to be in the loop. Giving the developers feedback, raising flags or better submitting a patch.

Decoupling of subsystems is the most significant point of leverage in software. If done correctly there is a massive gain in maintainability. Also, thoughtful decoupling removes the chance of cascading effects of tweaking the sub-components. Which makes iterations cheap. More iterations in less time imply a dynamic equilibrium can be reached faster between the solution and the context in which the problem exists.

On the other hand, there is the selfconscious process. It is not guided by the immediate feedback from the environment. It is the process of intellectualization. In the book, the author offers an example of designing a kettle. You start by gathering the requirements. Quantitive or qualitative. For example, its capacity, the temperature it heats to, how much it costs, and how should it look etc. A subtle point about requirements: you will only notice when it is not satisfied. Each requirement is an invariant of the problem. It defines the context. The more invariants, the lower the chance of satisfying all of them at once. For a high number of requirements, the likelihood of meeting all of them at once will be extremely low². This makes it challenging to produce forms that work. The designer, when working in the selfconscious paradigm, must rely on rigorous formal methods.

NOTES

1. The tight feedback loop will enforce a more modular design. And modularity will ensure quick modification of subsystems.

2. For n independent requirements we can estimate the likelihood to satisfy all of them at once will be:

Since 2017 WASM ecosystem is maturing at a rapid pace. It is also gaining traction. Autocad is ported to run in the browsers. PSPDFKIT, a company that offers a document conversion SDK, have cross compiled their 500000 lines c++ pdf conversion codebase to target WASM. Products like Google Earth and Figma are using WebAssembly in production. I have seen some Haskell cross compilers out there that target WASM. So, in theory, Pandoc can be cross-compiled to target WASM for general purpose document conversion in the browser.

For the use case that I described above, I am trying to use libwkthmltox, a C library, to convert an HTML document to PDF in the browser. I want to leverage this battle-tested C library with minimal glue code in C to solve my problem. The glue code exposes an API that takes in a pointer to a string (UTF-8 encoded HTML document) and returns another pointer that contains PDF formatted document. JS runtime and WASM runtime share memory indirectly using a linear memory model. We can use this feature to pass pointers around.

Emscripten is a compiler toolchain that does all the heavy lifting. It compiles C code first to LLVM bitcode using Clang then finally output a .wasm file along with boilerplate HTML and js.

Forget about the document conversion for a minute. Then solving this problem becomes simply a matter of string manipulation. Here is a function in C that takes a string concatenate it with another string and returns the new string. EMSCRIPTEN_KEEPALIVE macro comes from emscripten.h header file. It tells Emscripten to preserve this function while tree shaking.

const char* EMSCRIPTEN_KEEPALIVE get_string(char* input_str) {
	char str[] = "Hello, ";
	return strcat(str, input_str);
}

After transpiling C to WASM using Emscripten, from the Browser (boilerplate HTML and js code generated by Emscripten takes care of loading the WASM module in the browser) we can call the get_string function like this:

UTF8ToString(Module._get_string(allocateUTF8("World!")))
// "Hello, World!"

This contrived example is easy to write and run, but real challenge appears when I try to use the libwkthmltox library for actual document conversion. Emscripten documentation says, in the “using libraries” section, that to use 3rd party libraries, you need to build them to LLVM bitcode and then link them. For libraries that have a straightforward build process, it is not hard to do. Here is an excellent example of compiling libwebp, a c library for image compression, to WASM. However, compiling libwkhtmltox is not straightforward. It even has a separate repository just for building for different architectures. It uses a patched version of Qt. During the build, It compiles Qt from the source too.

At this stage, I am stuck.

So here is a new information diet plan that I am trying out. I will expire the unread current edition of the newsletters when the next one reaches my inbox. Expiring weekly newsletter at end of the week and daily newsletters at the end of the day. Newsletters mostly contain “newsy” stuff which is better read fresh. Most news events lose relevance as time passes. Do you pile up the newspapers to read them all on weekends because of FOMO? This should keep unread-list in check. For the links that I find on Twitter. It is better to skim through it once before sending it for later. This way I avoid click baits. Skimming the writeup also gives some context before actual reading. Adding this context to the email is helpful for later. Sometimes while going through my to-read list I find few items that I am no longer interested in. I archive those. Other times I find few items may not be useful immediately I archive them too. A guiding principle I tend to follow, read on ‘on demand basis’, read something that you need right now rather than reading about things that you might need in future.

Books are another source of consumption. Even more demanding in terms of time and attention. Some days I have time other days I don’t. That has to be okay. With many great books out there it is important to be able to pick up good books for yourself. Different books appeal to different people. Here is a great model, borrowed from Patrick Collison:

at every moment you should be reading the best book you know of in the world. I don’t mean ... the absolute best for everyone, but the best book for you. As soon as you discover something that seems more interesting or more important or whatever, you should absolutely discard your current book in favor of that, because any other algorithm necessarily results in you reading “worse stuff over time.

I think this is great advice.

What about the other sources like the Orange Website, Reddit et al? Overtime I am losing my appetite for Hacker News. I don’t read it that often. There were times when Hacker News was the morning routine. Partly because I don’t want to follow newsy stuff, instead I want to chase specific topics. Partly because of general toxicity in the threads and a lot of misinformation in the comments. Sometimes I still open up and follow one of the stories if I find something interesting.

A big question that remains unanswered. What is the goal of reading? Reading cannot be a goal in itself. People are reading all the time. Twitter, blogs, news websites, books and whatnot. What are they looking for? Just fear of missing out doesn’t seem like a reasonable answer too. There has to be something more concrete to be gained from groking all the stuff. Some people say it is curiosity. To me ‘curiosity’ is an abstract term. I don’t know what it means. It may mean different things to different people. Is curiosity a property of humans? Why are some people more curious and others less so? I don’t know the answers to these questions. But there is one pithy aphorism that drives the point home for me.

The universe rewards us for understanding it and punishes us for not understanding it. When we understand the universe, our plans work and we feel good. Conversely, if we try to fly by jumping off a cliff and flapping our arms the universe will kill us.”

-- Jack Cohen & Ian Stewart

If this is true then everyone in the universe is trying to learn about it as much as they can. Reading is probably the second best way to learn. Only after direct experiences. With a huge amount of information readily available ability to filter through them to find illuminating resources is even more valuable.