# Peter Alvaro et al. - Automating Failure Testing Research at Internet Scale (Highlights)  ## Metadata **Review**:: [readwise.io](https://readwise.io/bookreview/61307343) **Source**:: #from/readwise #from/zotero **Zettel**:: #zettel/fleeting **Status**:: #x **Authors**:: [[Peter Alvaro]], [[Kolton Andrus]], [[Chris Sanden]], [[Casey Rosenthal]], [[Ali Basiri]], [[Lorin Hochstein]] **Full Title**:: Automating Failure Testing Research at Internet Scale **Category**:: #articles #readwise/articles **Category Icon**:: 📰 **Highlighted**:: [[2026-06-14]] **Created**:: [[2026-06-20]] ## Highlights - Building a production fault injection infrastructure is only the first step towards maintaining fault-tolerant systems. The space of distinct failure scenarios—combinations of faults across a distributed system—that such an infrastructure can test is exponential in the number of potential faults. ([Page 1](zotero://open-pdf/library/items/KX7JUZRR?page=0&annotation=5TKZ8ELB)) ^1025088831 - fault-tolerance is redundancy ([Page 2](zotero://open-pdf/library/items/KX7JUZRR?page=1&annotation=259RP5GU)) ^1025088832 - The second insight is that instead of starting from initial states and exhaustively searching the space of possible executions, a better strategy for quickly identifying fault-tolerance bugs is to start with successful outcomes and reason backwards, from effects to causes, in order to understand whether some combination of faults could have prevented the outcome. ([Page 2](zotero://open-pdf/library/items/KX7JUZRR?page=1&annotation=MXGQWGVC)) ^1025088833 - It begins with a correct outcome, and asks why the system produced it. This recursive process of asking why questions (which we illustrate below) yields a lineage graph that characterizes all of the computations and data that contributed to the outcome. ([Page 2](zotero://open-pdf/library/items/KX7JUZRR?page=1&annotation=CDS93WQ6)) ^1025088834 - Preventing the good outcome, however, would require us to find a combination of failures that invalidates all of the alternative computations. We can encode this as a conjunction of disjunctions of propositional variables (i.e., a formula in conjunctive normal form): ([Page 3](zotero://open-pdf/library/items/KX7JUZRR?page=2&annotation=LUHDPQI7)) ^1025088835 Disjunction of nodes in a path, conjunction to combine paths. - In particular, we are interested in the minimal solutions—those that do not contain other solutions. ([Page 3](zotero://open-pdf/library/items/KX7JUZRR?page=2&annotation=ESF7EKZ2)) ^1025088836 - The system succeeds in producing the expected outcome. This indicates that it has revealed an alternative strategy for obtaining the outcome. The lineage graph from this execution should be merged with the current one, and a new formula should be extracted and solved. ([Page 3](zotero://open-pdf/library/items/KX7JUZRR?page=2&annotation=PDEUUZIS)) ^1025088837 - Netflix leverages an internal tracing system (similar to Dapper [34]) that provides the ability to trace requests through their interactions with distributed services and find all of the injection points along the path. ([Page 4](zotero://open-pdf/library/items/KX7JUZRR?page=3&annotation=WM9CFFQV)) ^1025088838 - Fortunately, the tracing service described in Section 3.2 records service interactions as call graphs. A call graph is shown in Figure 4. At a high level (and at a glance), such a trace resembles a lineage graph. ([Page 5](zotero://open-pdf/library/items/KX7JUZRR?page=4&annotation=ABP796LS)) ^1025088839 - The aforementioned heterogeneity of the applications running in the Netflix ecosystem made it challenging to identify an appropriately general measure of success for individual service interactions. ([Page 5](zotero://open-pdf/library/items/KX7JUZRR?page=4&annotation=CY4GC6Y5)) ^1025088840 - After some debugging, we found that certain injected failures affected not only the request itself, but the mechanism for reporting errors as well. Therefore we treat a lack of device reported metrics as a failure. ([Page 6](zotero://open-pdf/library/items/KX7JUZRR?page=5&annotation=K7PSYL3R)) ^1025088841 - To circumvent this obstacle, we simulate replay by treating all user requests that cause the same back-end behavior as if they were replays of a single, canonical request. ([Page 6](zotero://open-pdf/library/items/KX7JUZRR?page=5&annotation=WCZIYKBS)) ^1025088842