Neal Charbonneau

Some Good Reads of 2020

I’ve decided to try writing more notes here this year in case others find them useful, as I’ve benefited from lots of other peoples’ writing over the years. Who knows if I’ll stick to it, but to kick things off, I’ll start with an easy topic: some technical content I enjoyed from last year.


A Fast File System for UNIX: I thought this paper, written in 1984, was interesting for a few reasons. It one of the earlier file system papers I’ve come across that describes how to design a file system to take advantage of the characteristics of the underlying storage medium. They define allocation groups and policies with the goal of improving locality of reference and, as a result, performance. The paper has some interesting historical tidbits too. Among other things, they introduced file locking, which was explicitly discarded in original UNIX paper as being “neither necessary nor sufficient”. They also give some background as to why they went with advisory locks rather than mandatory locks.

The Zettabyte File System: This is a well written paper. It’s nice to see such a clear focus on their design principles and the tradeoffs they’re making. The content is good too, ZFS is a really interesting file system. It’s way more advanced than most of its production-grade contemporaries. Even though this was written in the aughts, there is still a lot in here that applies today.

Xen and the Art of Virtualization: This one was published around the same time as the ZFS paper. The technology specifics here are much more dated since virtualization has changed quite a bit since then. What stood out for me was the way they approached the problem. Given two layers (VMM and guest) of a system and a well established abstraction boundary (the virtual hardware interface), you have a tendency to try to optimize the system by implementing optimizations within the abstraction boundaries. Rather than take this approach, they asked, “well, what if we just break the abstraction a little bit”. I think the paper is a good example of challenging assumptions and exploring tradeoffs in system design.

Tales of the Tail: Hardware, OS, and Application-level Sources of Tail Latency: This paper is a nice mix of theory and practice. They show the implications to tail latencies for different theoretical queuing models, then demonstrate that in practice with real software they see much worse tails, and finally show what you can do to get closer to the theoretical lower bound. There are really good lessons both in how to design and structure your system to get better tail latencies and also some specific advice for system tuning. Good demonstration that reducing tail latencies requires making system-level decisions, not just tightly optimized code.


The Hundred-Page Machine Learning Book: It’s a dense 100 pages (well, 136, but who’s counting), but it’s a great intro to both the theory behind the most popular ML algorithms and also the sorts of things you need to think about when applying these techniques in the real world.

Database Internals: I thought this was a good survey-style book. It doesn’t explain much of the “why” or intuition around a lot of the topics, so it’s not great for getting a deep understanding of any one thing, but it’s a nice introduction to a broad field that you can use as a starting point for diving deeper.


Keeping CALM: When Distributed Consistency Is Easy: Coordination in a distributed system is expensive. If you’ve got a distributed algorithm, how do you know when coordination is necessary? If you come up with an algorithm that uses coordination, is that because it fundamentally requires it, or because you haven’t found the more clever solution yet? CALM provides an answer here to what sort of problems fundamentally do and do not require coordination.

sled theoretical performance guide: Lots of good advice for benchmarking and writing performance-critical code.

Files are fraught with peril: File systems are complex, both in implementation and, unfortunately, the interface. Writing robust software on top of a file system has a lot of non-obvious pitfalls. The author gave a talk about this as well that is really entertaining.


Ok, not a “read”, but Crust of Rust is a video series by Jon Gjengset that explain Rust at an intermediate level. With Rust, I found that there can still be a pretty big gap between having a basic grasp of different aspects of the language and writing code that compiles on the first try. These videos are a great way to fill that gap.