# James Hamilton - On Designing and Deploying Internet-Scale Services (Highlights)  ## Metadata **Review**:: [readwise.io](https://readwise.io/bookreview/62293460) **Source**:: #from/readwise #from/reader **Zettel**:: #zettel/fleeting **Status**:: #x **Authors**:: [[James Hamilton]] **Full Title**:: On Designing and Deploying Internet-Scale Services **Category**:: #articles #readwise/articles **Category Icon**:: 📰 **URL**:: [www.usenix.org](https://www.usenix.org/legacy/events/lisa07/tech/full_papers/hamilton/hamilton_html/) **Host**:: [[www.usenix.org]] **Highlighted**:: [[2026-07-26]] **Created**:: [[2026-08-01]] ## Highlights - Expect failures. A component may crash or be stopped at any time. Dependent components might fail or be stopped at any time. There will be network failures. Disks will run out of space. Handle all failures gracefully. ([View Highlight](https://read.readwise.io/read/01kyf3d4y6gh3vyz3p7q2py92w)) ^1037616917 - Keep things simple. Complexity breeds problems. Simple things are easier to get right. Avoid unnecessary dependencies. Installation should be simple. Failures on one server should have no impact on the rest of the data center. ([View Highlight](https://read.readwise.io/read/01kyf3dbmt0pmxbxdnxssd6m2q)) ^1037616918 - Automate everything. People make mistakes. People need sleep. People forget things. Automated processes are testable, fixable, and therefore ultimately much more reliable. Automate wherever possible. ([View Highlight](https://read.readwise.io/read/01kyf3ddqkg410q1wqf7f4pkm3)) ^1037616921 - If a hardware failure requires any immediate administrative action, the service simply won't scale cost-effectively and reliably. ([View Highlight](https://read.readwise.io/read/01kyf3nmqbwnhfn4dkgv51yr4w)) ^1037619260 - Failure recovery must be a very simple path and that path must be tested frequently. ([View Highlight](https://read.readwise.io/read/01kyf3nzg2kek6yjgc8kvfdxnr)) ^1037619590 - The acid test for full compliance with this design principle is the following: is the operations team willing and able to bring down any server in the service at any time without draining the work load first? ([View Highlight](https://read.readwise.io/read/01kyf3rm2mt96nykrfkny88ef8)) ^1037619889 - What isn't tested in production won't work, so periodically the operations team should conduct a "fire drill" using these tools. ([View Highlight](https://read.readwise.io/read/01kyf451574bmnxb5qftr8xj5b)) ^1037620928 - Our general rule is that optimizations that bring an order of magnitude improvement are worth considering, but percentage or even small factor gains aren't worth it. ([View Highlight](https://read.readwise.io/read/01kyg6651qhkc5nkaazdfman3p)) ^1037807076 - When planning new features, always consider what load they are going to put on the backend store. ([View Highlight](https://read.readwise.io/read/01kyg6kfjv70nz3zkk6nqwmms0)) ^1037808788 - A best practice is to build it into the specification with a section such as, "What impacts will this feature have on the rest of the infrastructure?" Then measure and validate the feature for load when it goes live. ([View Highlight](https://read.readwise.io/read/01kyg6qq3wsm63r6k3hxer0x91)) ^1037809645 - As a general rule, dependence on small components or services doesn't save enough to justify the complexity of managing them. ([View Highlight](https://read.readwise.io/read/01kyg765yrwy0k3x9adjw2kgk5)) ^1037810459 - If the service is overloading a dependent service, the depending service needs to know and, if it can't back-off automatically, alerts need to be sent. ([View Highlight](https://read.readwise.io/read/01kyg7c6yvc9q9vhk47y68tcge)) ^1037813280 - Testing in production is a reality and needs to be part of the quality assurance approach used by all internet-scale services. ([View Highlight](https://read.readwise.io/read/01kyg7kza4w9x2ntj6xmmett8y)) ^1037818635 - **Minimize false positives**. People stop paying attention very quickly when the data is incorrect. It's important to not over-alert or operations staff will learn to ignore them. This is so important that hiding real problems as collateral damage is often acceptable. ([View Highlight](https://read.readwise.io/read/01kyg7wpd2mpmhqwxxaqmzb06t)) ^1037818886 - **Monitor continuously**. It bears noting that people must be looking at all the data every day. Everyone should do this, but make it the explicit job of a subset of the team to do this. ([View Highlight](https://read.readwise.io/read/01kyg7xvxcge6813rekyr1kevm)) ^1037818906 - Too often, organizations grow operations to deal with scale and never take the time to engineer a scalable, reliable architecture. Services that don't think big to start with will be scrambling to catch up later. ([View Highlight](https://read.readwise.io/read/01kyg7yh7s0x3ngspdhsmp9dgh)) ^1037818927 - **Stress test for load**. Run some tiny subset of the production systems at twice (or more) the load to ensure that system behavior at higher than expected load is understood and that the systems don't melt down as the load goes up. ([View Highlight](https://read.readwise.io/read/01kyg7z45zt8exxxgytdacnee3)) ^1037818945 - Abstract the network and naming as far as possible, using DNS and CNAMEs. Always, always use a CNAME. ([View Highlight](https://read.readwise.io/read/01kyg86q55555jxh0ah73h26pw)) ^1037819712 - If development is frequently called in the middle of the night, automation is the likely outcome. If operations is frequently called, the usual reaction is to grow the operations team. ([View Highlight](https://read.readwise.io/read/01kyg8se32eeynqhchm33amp47)) ^1037825786 - Keep a rolling two week (or more) history of all changes to help recover from software or administrative errors. ([View Highlight](https://read.readwise.io/read/01kyg8t0qjy4s0p3n6grfjvvw1)) ^1037825796 - Anything that has any chance of needing to be changed in production should be made configurable and tunable in production without a code change. ([View Highlight](https://read.readwise.io/read/01kyg93cvagtgcg5d1sjg3hmdj)) ^1037825999 - Generally, a "big red switch" is a designed and tested action that can be taken when the service is no longer able to meet its SLA, or when that is imminent. ([View Highlight](https://read.readwise.io/read/01kyg9sb80ysxgzk8xs9x96t11)) ^1037826702