# Ben Maurer - Systems at Facebook Scale (Highlights) ![rw-book-cover|256](https://i.ytimg.com/vi/dlixGkelP9U/hqdefault.jpg) ## Metadata **Cover**:: https://i.ytimg.com/vi/dlixGkelP9U/hqdefault.jpg **Source**:: #from/readwise **Zettel**:: #zettel/fleeting **Status**:: #x **Authors**:: [[Ben Maurer]] **Full Title**:: Systems at Facebook Scale **Category**:: #podcasts #readwise/podcasts **Category Icon**:: 🎙 **URL**:: [www.youtube.com](https://www.youtube.com/watch?v=dlixGkelP9U) **Host**:: [[www.youtube.com]] **Highlighted**:: [[2021-02-15]] **Created**:: [[2022-09-26]] ## Highlights - Solve problems only once: code reuse ([Time 0:06:01](https://reclipped.com/a/Bzdbtm9sEeuylmsc6J8XWQ#361)) - Issues of `pthread_cond_t` ([Time 0:08:03](https://reclipped.com/a/Bzdbtm9sEeuylmsc6J8XWQ#483)) - ## Issues of pthread semphore 08:59 Potential context switches because of the loop in `pthread_cond_wait`. 10:20 The job scheduling in pthread is FIFO. It's bad for memory usage and CPU cache. 13:05 LifoSem, a LIFO semphore in folly ([source code](https://github.com/facebook/folly/blob/master/folly/synchronization/LifoSem.h)). It's 13x faster, and has 12x fewer context switches. 13:49 There are also issues in other pthread primitives. - `pthread_mutex_t` can cause contention in futex spinlock. A different futex api could fix this. I cannot find the link in the slides, but this article [Rethinking the futex API](https://lwn.net/Articles/823513/) may help. - `pthread_rwlock_t` uses a mutex, consider [RWSpinLock](https://github.com/facebook/folly/blob/master/folly/synchronization/RWSpinLock.h) in folly. ([Time 0:08:04](https://reclipped.com/a/-X_srm-dEeuy3Au6qSPQcQ#484)) - Potential context switches because of the loop in `pthread_cond_wait`. ([Time 0:08:59](https://reclipped.com/a/-X_srm-dEeuy3Au6qSPQcQ#539)) - The job scheduling in pthread is FIFO. It's bad for memory usage and CPU cache. ([Time 0:10:20](https://reclipped.com/a/-X_srm-dEeuy3Au6qSPQcQ#620)) - LifoSem, a LIFO semphore in folly ([source code](https://github.com/facebook/folly/blob/master/folly/synchronization/LifoSem.h)). It's 13x faster, and has 12x fewer context switches. ([Time 0:13:05](https://reclipped.com/a/-X_srm-dEeuy3Au6qSPQcQ#785)) - There are also issues in other pthread primitives. - `pthread_mutex_t` can cause contention in futex spinlock. A different futex api could fix this. I cannot find the link in the slides, but this article [Rethinking the futex API](https://lwn.net/Articles/823513/) may help. - `pthread_rwlock_t` uses a mutex, consider [RWSpinLock](https://github.com/facebook/folly/blob/master/folly/synchronization/RWSpinLock.h) in folly. ([Time 0:13:49](https://reclipped.com/a/-X_srm-dEeuy3Au6qSPQcQ#829)) - Overload handling philosophy ([Time 0:17:20](https://reclipped.com/a/Bzdbtm9sEeuylmsc6J8XWQ#1040)) - ## Overload Handling Philosophy * If you're going to fail, fail quickly — prevents delaying the overall request. * Servers should signal overload. * Order doesn't matter (Adaptive LIFO). * Clients should defend themselves (Concurrency Control) * Complex knobs are tuned poorly. ([Time 0:17:21](https://reclipped.com/a/PqOjzG-hEeuy4Ful1dyEuw#1041)) - Controlled Delay ([Time 0:20:29](https://reclipped.com/a/Bzdbtm9sEeuylmsc6J8XWQ#1229)) - ## Controlled Delay Use a very short time out when the queue is unable to empty the jobs for a long time. ([Time 0:20:30](https://reclipped.com/a/AUya6m-jEeuy4d_0JknF_w#1230)) - Adaptive LIFO ([Time 0:22:29](https://reclipped.com/a/Bzdbtm9sEeuylmsc6J8XWQ#1349)) - ## Adaptive FIFO Use FIFO as the default mode. Switch to LIFO on high load. Adaptive LIFO and CoDel play nicely together. ([Time 0:22:30](https://reclipped.com/a/Rvzmsm-jEeuy4p_l5_Z-wA#1350)) - Concurrency Control ([Time 0:24:49](https://reclipped.com/a/Bzdbtm9sEeuylmsc6J8XWQ#1489)) - ## Concurrency Control Each client keeps track of the number of outstanding outbound requests on a per-service basis. When new requests are sent, if the number of outstanding requests to that service exceeds a configurable number, the request is immediately marked as an error. ([Time 0:24:59](https://reclipped.com/a/fjxrUm-jEeuy42Mj-gMXEA#1499)) - ## TCP Listen Queue Overflow The kernel drops the packet and the peer will retransmit it. A setting you can fix today: ``` sysctl -w net.ipv4.tcp_abort_on_overflow=1 ``` ([Time 0:26:39](https://reclipped.com/a/rYKddG9wEeuymK-QT0l0Tw#1599)) - TCP listen queue overflow ([Time 0:26:39](https://reclipped.com/a/Bzdbtm9sEeuylmsc6J8XWQ#1599)) - ## DERP * Detection. How was the issue detected—alarms, dashboards, user reports? * Escalation. Did the right people get involved quickly? Could these people have been brought in via alarms rather than manually? * Remediation. What steps were taken to fix the issue? Can these steps be automated? * Prevention. What improvements could remove the risk of this type of failure happening again? How could you have failed gracefully, or failed faster to reduce the impact of this failure? ([Time 0:38:58](https://reclipped.com/a/ahgLqG-kEeuy5IMn6XCpLw#2338)) - DERP: detection, escalation, remediation, prevention ([Time 0:38:58](https://reclipped.com/a/Bzdbtm9sEeuylmsc6J8XWQ#2338))