# Linux Crisis Tools ## Metadata **Status**:: #x **Zettel**:: #zettel/fleeting **Source**:: #from/browser **URL**:: [www.brendangregg.com](https://www.brendangregg.com/blog/2024-03-24/linux-crisis-tools.html) **Host**:: [[www.brendangregg.com]] **Created**:: [[2024-03-24]] **Source**:: #from/clipper **Title**:: Linux Crisis Tools **Author**:: [[Brendan Gregg]] ## Synopsis When you have an outage caused by a performance issue, you don't want to lose precious time just to install the tools needed to diagnose it. Here is a list of "crisis tools" I recommend installing on your Linux servers by default (if they aren't already), along with the (Ubuntu) package names that they come from: | Package | Provides | Notes | | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | | procps | ps(1), vmstat(8), uptime(1), top(1) | basic stats | | util-linux | dmesg(1), lsblk(1), lscpu(1) | system log, device info | | sysstat | iostat(1), mpstat(1), pidstat(1), sar(1) | device stats | | iproute2 | ip(8), ss(8), nstat(8), tc(8) | preferred net tools | | numactl | numastat(8) | NUMA stats | | tcpdump | tcpdump(8) | Network sniffer | | linux-tools-common<br>linux-tools-$(uname -r) | perf(1), turbostat(8) | profiler and PMU stats | | bpfcc-tools (bcc) | opensnoop(8), execsnoop(8), runqlat(8), softirqs(8), hardirqs(8), ext4slower(8), ext4dist(8), biotop(8), biosnoop(8), biolatency(8), tcptop(8), tcplife(8), trace(8), argdist(8), funccount(8), profile(8), etc. | canned eBPF tools [^1] | | bpftrace | bpftrace, basic versions of opensnoop(8), execsnoop(8), runqlat(8), biosnoop(8), etc. | eBPF scripting [^1] | | trace-cmd | trace-cmd(1) | Ftrace CLI | | nicstat | nicstat(1) | net device stats | | ethtool | ethtool(8) | net device info | | tiptop | tiptop(1) | PMU/PMC top | | cpuid | cpuid(1) | CPU details | | msr-tools | rdmsr(8), wrmsr(8) | CPU digging | (This is based on Table 4.1 "Linux Crisis Tools" in [SysPerf 2](https://www.brendangregg.com/systems-performance-2nd-edition-book.html).) [^1]: bcc and bpftrace have many overlapping tools: the bcc ones are more capable (e.g., CLI options), and the bpftrace ones can be edited on the fly. But that's not to say that one is better or faster than the other: They emit the same BPF bytecode and are equally fast once running. Also note that bcc is evolving and migrating tools from Python to libbpf C (with CO-RE and BTF) but we haven't reworked the package yet. In the future "bpfcc-tools" should get replaced with a much smaller "libbpf-tools" package that's just tool binaries.