# Siddon Tang - From Chaos to Order — Tools and Techniques for Testing TiDB, a Distributed NewSQL Database (Highlights) ![rw-book-cover|256](https://en.pingcap.com/wp-content/uploads/2018/04/shrodinger-architecture-on-k8s.png) ## Metadata **Review**:: [readwise.io](https://readwise.io/bookreview/61298644) **Source**:: #from/readwise #from/reader **Zettel**:: #zettel/fleeting **Status**:: #x **Authors**:: [[Siddon Tang]] **Full Title**:: From Chaos to Order — Tools and Techniques for Testing TiDB, a Distributed NewSQL Database **Category**:: #articles #readwise/articles **Category Icon**:: 📰 **Document Tags**:: #via/ddia **URL**:: [pingcap.co.jp](https://pingcap.co.jp/blog/chaos-practice-in-tidb/) **Host**:: [[pingcap.co.jp]] **Highlighted**:: [[2026-06-13]] **Created**:: [[2026-06-20]] ## Highlights - Conduct an experiment on the chosen hypothesis by injecting faults and analyzing the result. ([View Highlight](https://read.readwise.io/read/01kv1dsca5bnv5rkj854yh2b94)) ^1024996887 - Send SIGSTOP to hang or SIGCONT to resume the process. ([View Highlight](https://read.readwise.io/read/01kv1dv3e0dmhsdtx9x3sd8qej)) ^1024997234 - Use `renice` to adjust the process priority or use `setpriority` for the threads of the process. ([View Highlight](https://read.readwise.io/read/01kv1dv7rxn6dc0ssbb7an5sjp)) ^1024997257 - Max out the CPU. ([View Highlight](https://read.readwise.io/read/01kv1dvd7b59j6p2jpshvv0s96)) ^1024997287 - Use `iptables` or `tc` to drop or reject the network packages or delay the network packages. ([View Highlight](https://read.readwise.io/read/01kv1dvjybfbt72r3y9rntn5qj)) ^1024997312 - Use `tc` to reorder the network packages and use a proxy to reorder the gRPC requests. ([View Highlight](https://read.readwise.io/read/01kv1dvs69m786t4yntb20daff)) ^1024997318 - Use `iperf` to take all network throughput. ([View Highlight](https://read.readwise.io/read/01kv1dwa3pjynfz8xq377t5wqr)) ^1024997375 - Use `libfuse` to mount a file system and do the I/O fault injection. ([View Highlight](https://read.readwise.io/read/01kv1dw6wmdr7y3mdgna7cbn96)) ^1024997367 - Link `libfiu` to do the I/O fault injection. ([View Highlight](https://read.readwise.io/read/01kv1dwjcrdcry28m2pjedcsy8)) ^1024997390 - Use `rm -rf` forcbily to remove all data. ([View Highlight](https://read.readwise.io/read/01kv1dwqm97z5zfyvs717jcjp9)) ^1024997406 - Use `echo 0 > file` to damage a file. ([View Highlight](https://read.readwise.io/read/01kv1dwt2r847amfyrvwfghsts)) ^1024997419 - Copy a huge file to create the `NoSpace` problem. ([View Highlight](https://read.readwise.io/read/01kv1dww37fahkgba8d2nf8w1k)) ^1024997421 - A popular fault injection tool is the Fault Injection Framework that’s included in the Linux kernel, which developers can use to implement simple fault injection to test device drivers. ([View Highlight](https://read.readwise.io/read/01kv1e0swvgj8gbnw36fm4c5hw)) ^1024997849 - Another way to inject fault is [‘SystemTap’](https://sourceware.org/systemtap/), a scripting language and tool which can assist diagnosis of a performance or functional problem. ([View Highlight](https://read.readwise.io/read/01kv1e53z68vgveghxf7wxs7wm)) ^1024998014 - We can use a mechanism called [`fail`](https://www.freebsd.org/cgi/man.cgi?query=fail&sektion=9&apropos=0&manpath=FreeBSD%2B10.0-RELEASE). Using `fail` we can inject the fault exactly where we want it. In Go, we can use [`gofail`](https://github.com/coreos/gofail) and in Rust, we can use [`fail-rs`](https://github.com/pingcap/fail-rs). ([View Highlight](https://read.readwise.io/read/01kv1e6j7cabqjjn3c4j6zp05c)) ^1024998042 - Another platform is [Jepsen](https://github.com/jepsen-io/jepsen) which is designed to verify the linearizability of the distributed database. Jepsen uses Nemeses to disturb the system, records the client operations, and verifies the linearizability through the operation history. ([View Highlight](https://read.readwise.io/read/01kv1efhv0vg51gpghx6v26af9)) ^1024999033