# Haryadi S. Gunawi et al. - What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems (Highlights)  ## Metadata **Review**:: [readwise.io](https://readwise.io/bookreview/61162320) **Source**:: #from/readwise #from/zotero **Zettel**:: #zettel/fleeting **Status**:: #x **Authors**:: [[Haryadi S. Gunawi]], [[Mingzhe Hao]], [[Tanakorn Leesatapornwongsa]], [[Tiratat Patana-anake]], [[Thanh Do]], [[Jef...]] **Full Title**:: What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems **Category**:: #articles #readwise/articles **Category Icon**:: 📰 **Document Tags**:: #distributed-system #from/zotero #paper #testing **Highlighted**:: [[2026-06-09]] **Created**:: [[2026-06-13]] ## Highlights - In addition to this, we find “new” classes of bugs unique to cloud systems: data consistency (5%), scalability (2%), and topology (1%) bugs. Cloud dependability tools should evolve to capture these new problems. ([Page 2](zotero://open-pdf/library/items/HC3P3ZFC?page=1&annotation=NB2LJ94L)) ^1023731835 - Besides the main read/write protocols, many other operational protocols (e.g., bootstrap, cloning, fsck) touch and modify data, and bugs within them can cause data inconsistency. ([Page 4](zotero://open-pdf/library/items/HC3P3ZFC?page=3&annotation=VWX7EBRN)) ^1023731836 Bugs in cold code path because they are less frequent ran. - Operational protocols modify data replicas, but they tend to be less tested than the main protocols, and thus often carry data inconsistency bugs. ([Page 5](zotero://open-pdf/library/items/HC3P3ZFC?page=4&annotation=IUH9Q4JD)) ^1023731837 - Positive feedback loop: This is the case where failures happen, then recovery starts, but the recovery introduces more load and hence more failures [29, 32]. ([Page 6](zotero://open-pdf/library/items/HC3P3ZFC?page=5&annotation=4LRVY3S8)) ^1023731838 - Buggy failover: A key to no-SPoF is to detect failure and perform a failover. But, such guarantee breaks if the failover code itself is buggy. ([Page 7](zotero://open-pdf/library/items/HC3P3ZFC?page=6&annotation=DAYLZ573)) ^1023731839 - Put simply, failover in failover is brittle [25]. ([Page 7](zotero://open-pdf/library/items/HC3P3ZFC?page=6&annotation=K3RRHVLX)) ^1023731840 - Another key to no-SPoF is that after a successful failover, the system should be able to resume the previously failed operation. This is true if the cause was a machine failure, but not true for a software bug. In other words, if after a failover the system must run the same buggy logic again, then the whole process will repeat and the entire cluster will eventually die. ([Page 7](zotero://open-pdf/library/items/HC3P3ZFC?page=6&annotation=PM75XK2Z)) ^1023731841 - To reduce the severity of these killer bugs, cloud systems must distinguish between hardware failures and software logic bugs. In the latter case, it is better to stop the failover rather than killing the entire cluster. ([Page 7](zotero://open-pdf/library/items/HC3P3ZFC?page=6&annotation=8UQSM6SH)) ^1023731842 - Simple testing of error handling can uncover many flaws [55]. ([Page 9](zotero://open-pdf/library/items/HC3P3ZFC?page=8&annotation=YBR7IKG9)) ^1023731843 - Overall, we find cloud systems code lacks of specifications of what error handling should do, but in most discussions the developers know what the code ideally should perform. This “specification gap” between systems code and developers needs to be narrowed. ([Page 9](zotero://open-pdf/library/items/HC3P3ZFC?page=8&annotation=KW4NFATK)) ^1023731844 - This hints that the relationship between heartbeat and actual progress can have a loophole. ([Page 10](zotero://open-pdf/library/items/HC3P3ZFC?page=9&annotation=2TWCMKVM)) ^1023731845 - In many cases, cleanup procedures are still manually done by administrators. If not done in timely fashion, tight space can cause performance issues or downtimes. ([Page 10](zotero://open-pdf/library/items/HC3P3ZFC?page=9&annotation=7ZK8RF6U)) ^1023731846 - Many times, cloud systems try to serve all requests even if the load is beyond their limit. This can cause problems such as backlogs and OOM, which then make the system die and cannot serve any requests. It seems better for cloud systems to know their limits and reject requests when overloaded. ([Page 11](zotero://open-pdf/library/items/HC3P3ZFC?page=10&annotation=2LYUCJ5Y)) ^1023731847 - operational protocols should be load tested as well ([Page 11](zotero://open-pdf/library/items/HC3P3ZFC?page=10&annotation=BMDYK76T)) ^1023731848