# FoundationDB Authors - Simulation and Testing (Highlights) ![rw-book-cover|256](https://readwise-assets.s3.amazonaws.com/static/images/article2.74d541386bbf.png) ## Metadata **Cover**:: https://readwise-assets.s3.amazonaws.com/static/images/article2.74d541386bbf.png **Source**:: #from/readwise **Zettel**:: #zettel/fleeting **Status**:: #x **Authors**:: [[FoundationDB Authors]] **Full Title**:: Simulation and Testing **Category**:: #articles #readwise/articles **Category Icon**:: 📰 **Document Tags**:: #distributed-system #test **URL**:: [apple.github.io](https://apple.github.io/foundationdb/testing.html) **Host**:: [[apple.github.io]] **Highlighted**:: [[2020-04-16]] **Created**:: [[2022-09-26]] ## Highlights - robust simulation, live performance testing, and hardware-based failure testing. - increased intensity of the failures - a *deterministic* simulation of an entire FoundationDB cluster within a single-threaded process - Determinism is crucial in that it allows perfect repeatability - simulates all physical components - To swizzle-clog, you first pick a random subset of nodes in the cluster. Then, you “clog” (stop) each of their network connections one by one over a few seconds. Finally, you unclog them in a random order, again one by one, until they are all up.