Article Review: Containers will not fix your broken culture (and other hard truths)

First things first, if you do not know what is ACM Queue (or even worse, do not know what ACM is), click on the links provided. ACM relatively recently has reformed and now presents articles by industry experts, especially in the Queue magazine (you get an article from Queue with every Communications of the ACM magazine but there is more, much more). (disclaimer – while I am a paying ACM member, I make no profit or have no further affiliation with the organization (i.e. I am not an official Ambassador).
With that out of the way, let’s focus in the article in question. The author is Bridget Kromhout, currently working for Microsoft. The main idea of the article is that solution to difficult, seemingly technical problems, can be best resolved by examining the interactions with others. The main ideas discussed therein are the following

  • Tech is not a panacea
  • Good team interactions: Build, because you can’t buy
  • Tech, like Soylent Green, is made of people
  • Good fences make good neighbors
  • Avoiding sadness-as-a-service

The article is extremely well written. One thing I liked the most is that it includes links to definition you might or might have not heard. The key take away idea of the article is that we tend to think technology and enforce technology rules in an increasingly complex distributed system world, whereas the key is communication between individuals and teams, peers or otherwise. It also coins a phrase that unfortunately will ring true for a lot of people in the audience of this blog “on-call PTSD” and even manages to kill one of my favorite interview questions, and these are only the first two pages. The article also states “we succeed when share responsibility and have agency” – Amen to that, personally I have seen more than a few dysfunctional environments where responsibilities were shrugged off routinely. So to sum it up (and keep this review proportional to the length of the article), Bridget states the value of communication, brings in a ton of references to support her case (making the article simultaneously well research but not falling into the trap of being esoteric) and, at the same time, emphasizes the need of technology. Highly recommended reading!

Book Review: The Practice Of Cloud System Administration Volume 2 – Designing And Operating Large Distributed Systems

Hello everyone with another book review. This time, I will be reviewing a book that I consider a classic. As always, let’s start with the list of contents:
Part I Design: Building it

  • Designing in a distributed world
  • Designing for Operations
  • Selecting a Service Platform
  • Application Architectures
  • Design Patterns for Scaling
  • Design Patterns for Resiliency

Part II Operations: Running it

  • Operations in a Distributed World
  • DevOps Culture
  • Service Delivery: The Build Phase
  • Service Delivery: The Deployment Phase
  • Upgrading Live Services
  • Automation
  • Design Documents
  • Oncall
  • Disaster Preparedness
  • Monitoring Fundamentals
  • Monitoring Architecture and Practice
  • Capacity Planning
  • Creating KPIs
  • Operational Excellence

Part III Appendices

  • Assessments
  • The Origins and Future of Distributed Computing and Clouds
  • Scaling Terminology and Concepts
  • Templates and Examples
  • Recommended Reading

overall a bit over 500 beautifully printed pages (as you would come to expect from Addison-Wesley).
As you can see from the ToC, the breadth of information contained in this book is tremendous, every chapter can easily expand into a book on its own (and indeed, there are volumes that expand on a lot of the topics), however this book achieves to give the astute reader a ton of information, heck it is almost like the information is condensed – just add water. The authors do not fell into the pit of sticking with a particular technology, they maintain a level of abstraction, that in my opinion is about right, not too abstract (that would limit the potential of the book to be applied in real world situations) and, yet, not tied to a particular technology (i.e. this book came before container orchestration frameworks became as popular as they are today but you will not notice) that would instantly severely date the book. The format of the book is similar for all chapters, first an attention-grabbing introduction, then a nice discussion of the topic at hand and finally exercises, so the reader can follow up with what has been discussed – most of them are open ended. After all, large scale distributed systems have a common set of characteristics, no matter what the implementation details are or purpose.
The potential audience of this book are both SREs and their managers. In particular, Part II of the book contains a ton of information relevant to both sides of the equation. If you manage SREs, you’d better be at least acquainted with the material and this book is more than a fine introduction. If you need a book on how to use AWS/Azure/GCP or their specifics, this volume will NOT meet your expectations, as discussed this book is more like a framework.
In case, this is not obvious by now, I consider this book a must-read for anyone dealing with modern distributed systems, be it SRE, SWE or Engineering Manager. I cannot praise this book enough, it is extremely well written, in certain cases it goes against the trends and how can you go wrong with a book that considers a zombie outbreak a valid reason for a datacenter outbreak?
Further resources:
Companion Website
Thomas Limoncelli’s Twitter
PS. A book that everybody is recommending (and asking me about it, in a variety of contexts) is Google’s SRE book. If you have not read this book by now, then you can start by going there to enjoy the book in its entirety. While the Google SRE book is an extremely useful resource, and without wanting to create a false dichotomy, it kind of overshadows this volume, which, in my humble opinion is a better choice in certain regards. Specifically, while both books have an strong Google influence (one is coming from Google, the author of the other was a Google SRE), I find that the “Practice of …” is a more focused volume, something perhaps to be expected given that it is written by “only” three authors. So, do yourself a favour, read both books, there is a wealth of information contained therein.

Book Review: PostgreSQL Replication

So for my series of System Engineering books, I will proceed with a short review of PostgreSQL Replication by Packt. The reason this book came to be a part of my collection is that while there is a lot of information regarding PostgreSQL replication out there, a lot of it is out of date, given the overhaul of the replication system in PostgreSQL 9.X. Without further ado, here is the list of contents of the book.

  • Understanding Replication Concepts
  • Understanding the PostgreSQL Transaction Log
  • Understanding Point-In-Time Recovery
  • Setting up asynchronous replication
  • Setting up synchronous replication
  • Monitoring your setup
  • Understanding Linux High-Availability
  • Working with pgbouncer
  • Working with PgPool
  • Configuring Slony
  • Using Skytools
  • Working with Postgres-XC
  • Scaling with PL/Proxy
    The book gets straight into business with an introduction of replication concepts, and why this is a hard problem that cannot be a one-size fits all solution. Topics such as master-master replication and sharding are addressed as well. After this short introduction, specifics of PostgreSQL are examined, with a heavy focus on XLOG and related internals. The book goes into a nice balanced amount of detail, detailed enough to surpass the trivial level but not overwhelming (and thank $DEITY, we are spared source code excerpts, although a few references would be nice for those that are willing to dig further into implementation details), providing a healthy amount of background information. With that out of the way, a whole chapter is devoted to the topic of Point-In-Time-Recover (PITR for now on). PITR is an invaluable weapon in the arsenal of any DBA and gets a fair and actionable treatise, actionable meaning that you will walk away from this chapter with techniques you can start implementing right away.With the theory and basic vocabulary defined, the book then dives into replication. Concepts are explained, as well as drawbacks of each technique, alongside with specific technical instructions on how to get there, including a Q&A on common issues that you may encounter in the field.
    PostgreSQL has a complex ecosystem and once the actual built-in replication mechanisms are explained, common tools are presented (with the glaring omission of Bucardo unfortunately). This is where the book falters a bit, given the excellent quality of the replication related chapters. The presentation of the tools is not even nor deep in all cases – my gripe is that the Linux-HA chapter stops when it starts to get interesting. Having pointed this out, still these chapters can be better written and more concise than information scattered around in the web. I have paid particular attention to the PgPool chapter, which does not cover PgPool-HA (hint: there is more than one way to do it). These chapters assume no previous exposure to the ecosystem so they serve as a gentle (and again, actionable) introduction to the specific tools but I would have preferred them to be 10-15 pages longer each, providing some additional information, especially on the topic of high-availability. Even as-is, these chapters will save you a lot of time searching and compiling information, filling in a few blanks along the way, so, make no mistake, they are still useful. Bonus points for covering PostgreSQL-XC, which is somewhat of an underdog.
    A small detail is that examples in the book tend to focus on Debian-based systems so if you are administering a Red Hat derivative you should adapt the examples slightly, taking into consideration the differences in the packaging of PostgreSQL. Overall, the book goes for a broad as opposed to deep approach and can server as a more than solid introductory volume. Inevitably, there is an overlap with the official PostgreSQL manuals, which is to be expected given that they are great. The quality of the book is on par with other Packt Publishing titles, making this an easy to read book that will save you a lot of time for certain use cases.