# SQLite Authors - How SQLite Is Tested (Highlights)  ## Metadata **Review**:: [readwise.io](https://readwise.io/bookreview/26479641) **Source**:: #from/readwise **Zettel**:: #zettel/fleeting **Status**:: #x **Authors**:: [[SQLite Authors]] **Full Title**:: How SQLite Is Tested **Category**:: #articles #readwise/articles **Category Icon**:: 📰 **URL**:: [www.sqlite.org](https://www.sqlite.org/testing.html) **Host**:: [[www.sqlite.org]] **Highlighted**:: [[2023-04-16]] **Created**:: [[2023-04-18]] ## Highlights - Anomaly tests are tests designed to verify the correct behavior of SQLite when something goes wrong. ([View Highlight](https://read.readwise.io/read/01gy3trb6sywfz80w7c83zgwaf)) ^509767798 - The TCL and TH3 test harnesses are both capable of inserting a modified version of malloc() that can be rigged to fail after a certain number of allocations. ([View Highlight](https://read.readwise.io/read/01gy3tt5hkj45er5zswz9ed8hc)) ^509768379 - The concept of fuzz testing has been around for decades, but fuzz testing was not an effective way to find bugs until 2014 when Michal Zalewski invented the first practical profile-guided fuzzer, [American Fuzzy Lop](http://lcamtuf.coredump.cx/afl/) or "AFL". ([View Highlight](https://read.readwise.io/read/01gy40r970jf3vr9n56qbqgbsv)) ^509794534 - Beginning in 2016, a team of engineers at Google started the [OSS Fuzz](https://github.com/google/oss-fuzz) project. OSS Fuzz uses a AFL-style guided fuzzer running on Google's infrastructure. The Fuzzer automatically downloads the latest check-ins for participating projects, fuzzes them, and sends email to the developers reporting any problems. When a fix is checked in, the fuzzer automatically detects this and emails a confirmation to the developers. ([View Highlight](https://read.readwise.io/read/01gy40smw94fd75pc42px16e4z)) ^509794683 - MC/DC testing seems to work well for building code that is robust during normal use, whereas fuzz testing is good for building code that is robust against malicious attack. ([View Highlight](https://read.readwise.io/read/01gy41303kj5g2rp4tn6a7cw30)) ^509795900 - The TCL and TH3 test suites both contains numerous tests that push SQLite right to the edge of its defined limits and verify that it performs correctly for all allowed values. ([View Highlight](https://read.readwise.io/read/01gy416m3mchsxrkck1nn4n61d)) ^509796126 - For testing purposes, the SQLite source code defines macros called ALWAYS() and NEVER(). The ALWAYS() macro surrounds conditions which are expected to always evaluate as true and NEVER() surrounds conditions that are always evaluated to false. These macros serve as comments to indicate that the conditions are defensive code. ([View Highlight](https://read.readwise.io/read/01gy41e5tsan7c0czs46tfq3f4)) ^509797906 > When measuring test coverage, these macros are defined to be constant truth values so that they do not generate assembly language branch instructions - But in a coverage measuring build, the testcase() macro generates code that evaluates the conditional expression in its argument. Then during analysis, a check is made to ensure tests exist that evaluate the conditional to both true and false. Testcase() macros are used, for example, to help verify that boundary values are tested. ([View Highlight](https://read.readwise.io/read/01gy4c317qnpzd3m4kw57536a9)) ^509857006 - Branch coverage in SQLite is currently measured using [gcov](https://gcc.gnu.org/onlinedocs/gcc/Gcov.html) with the "-b" option. First the test program is compiled using options "-g -fprofile-arcs -ftest-coverage" and then the test program is run. Then "gcov -b" is run to generate a coverage report. The coverage report is verbose and inconvenient to read, so the gcov-generated report is processed using some simple scripts to put it into a more human-friendly format. ([View Highlight](https://read.readwise.io/read/01gy4c6hzbbqxebqbzsvkkwyrv)) ^509857443 - Using gcov (or similar) to show that every branch instruction is taken at least once in both directions is good measure of test suite quality. But even better is showing that every branch instruction makes a difference in the output. In other words, we want to show not only that every branch instruction both jumps and falls through but also that every branch is doing useful work and that the test suite is able to detect and verify that work. When a branch is found that does not make a difference in the output, that suggests that the code associated the branch can be removed (reducing the size of the library and perhaps making it run faster) or that the test suite is inadequately testing the feature that the branch implements. ([View Highlight](https://read.readwise.io/read/01gy4c9r7zdtr9gbnrv1k9h3gh)) ^509858095 - SQLite strives to verify that every branch instruction makes a difference using [mutation testing](https://en.wikipedia.org/wiki/Mutation_testing). [A script](https://www.sqlite.org/th3.html#muttest) first compiles the SQLite source code into assembly language (using, for example, the -S option to gcc). Then the script steps through the generated assembly language and, one by one, changes each branch instruction into either an unconditional jump or a no-op, compiles the result, and verifies that the test suite catches the mutation. ([View Highlight](https://read.readwise.io/read/01gy4cadsach2es134b8n581e6)) ^509858425 - To help ensure that SQLite does not make use of undefined or implementation defined behavior, the test suites are rerun using instrumented builds that try to detect undefined behavior. For example, test suites are run using the "-ftrapv" option of GCC. And they are run again using the "-fsanitize=undefined" option on Clang. And again using the "/RTC1" option in MSVC. Then the test suites are rerun using options like "-funsigned-char" and "-fsigned-char" to make sure that implementation differences do not matter either. Tests are then repeated on 32-bit and 64-bit systems and on big-endian and little-endian systems, using a variety of CPU architectures. Furthermore, the test suites are augmented with many test cases that are deliberately designed to provoke undefined behavior. For example: "**SELECT -1*(-9223372036854775808);**". ([View Highlight](https://read.readwise.io/read/01gy4cjavq5m0ejvz7pw1ad7ns)) ^509858900 - One verification technique used on SQLite is to run an entire test suite twice, once with optimizations left on and a second time with optimizations turned off, and verify that the same output is obtained both times. This shows that the optimizations do not introduce errors. ([View Highlight](https://read.readwise.io/read/01gy4ckavq3ymy5f6t5gyxazkx)) ^509858926 - SQLite compiles without warnings on GCC and Clang using the -Wall and -Wextra flags on Linux and Mac and on MSVC on Windows. No valid warnings are generated by the Clang Static Analyzer tool "scan-build" either (though recent versions of clang seem to generate many false-positives.) ([View Highlight](https://read.readwise.io/read/01gy4dchgy5nqbfdttd1pz9f66)) ^509860805