--- title: OSDI 2024 Digest venue: OSDI year: 2024 date: '2024-07-10' tags: - llm-serving - distributed-systems - verification - memory - networking - storage draft: false paper_count: 11 --- 11 papers selected. --- ### DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving *Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu *et al.** **TL;DR** — Separates the compute-heavy prefill phase from the memory-bound decoding phase onto different GPU pools, eliminating head-of-line blocking and significantly improving LLM serving throughput. **Why notable** — Became one of the most influential LLM systems papers of 2024; the prefill–decode disaggregation insight is now widely adopted in production inference stacks (vLLM, SGLang, etc.). [→ Read paper](https://www.usenix.org/conference/osdi24/presentation/zhong-yinmin) --- ### Fairness in Serving Large Language Models *Ying Sheng 0007, Shiyi Cao, Dacheng Li, Banghua Zhu *et al.** **TL;DR** — Introduces VTC, a token-count-weighted fair scheduling policy that prevents long-prompt users from monopolising GPU capacity in multi-tenant LLM services. **Why notable** — First paper to formally study multi-tenant fairness in LLM serving; directly influenced subsequent work on SLA-aware serving and resource allocation in shared inference clusters. [→ Read paper](https://www.usenix.org/conference/osdi24/presentation/sheng) --- ### Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve *Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan *et al.** **TL;DR** — Introduces chunked prefill and stall-free scheduling to decouple throughput and latency goals, letting the same serving system meet both SLOs simultaneously. **Why notable** — Elegant framing of the throughput–latency tension; chunked prefill became a standard technique in open-source inference engines within months of publication. [→ Read paper](https://www.usenix.org/conference/osdi24/presentation/agrawal) --- ### Llumnix: Dynamic Scheduling for Large Language Model Serving *Biao Sun 0002, Ziming Huang, Hanyu Zhao, Wencong Xiao *et al.** **TL;DR** — Treats in-flight LLM requests as migratable units, enabling load balancing and SLO recovery by live-migrating KV-cache state across GPU instances. **Why notable** — Request migration for LLM serving was considered impractical due to KV-cache size; this paper shows it is feasible and impactful, opening a new design dimension for inference schedulers. [→ Read paper](https://www.usenix.org/conference/osdi24/presentation/sun-biao) --- ### MAST: Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale *Arnab Choudhury, Yang Wang 0009, Tuomas Pelkonen, Kutta Srinivasan *et al.** **TL;DR** — Describes Google's production system for scheduling ML training jobs across geographically distributed datacenters, balancing GPU utilisation, job deadlines, and cross-datacenter bandwidth costs. **Why notable** — Rare large-scale production paper on global ML scheduling; the insights on heterogeneous cluster management and placement constraints are directly useful for anyone operating multi-site GPU infrastructure. [→ Read paper](https://www.usenix.org/conference/osdi24/presentation/choudhury) --- ### SquirrelFS: using the Rust compiler to check file-system crash consistency *Hayley LeBlanc, Nathan Taylor, James Bornholt, Vijay Chidambaram* **TL;DR** — Encodes crash-consistency invariants in Rust's type system so that a file system that compiles is guaranteed not to leave the storage in an inconsistent state after a crash. **Why notable** — A clean demonstration that language-level type checking can replace runtime or proof-assistant-based verification for an important systems property; the approach is general and practically viable. [→ Read paper](https://www.usenix.org/conference/osdi24/presentation/leblanc) --- ### Anvil: Verifying Liveness of Cluster Management Controllers *Xudong Sun 0013, Wenjie Ma, Jiawei Tyler Gu, Zicheng Ma *et al.** **TL;DR** — Presents the first framework for mechanically verifying liveness (eventual progress) of Kubernetes-style reconciliation controllers, with proofs for real controllers including ZooKeeper and RabbitMQ operators. **Why notable** — Liveness proofs for real-world cloud controllers were previously out of reach; Anvil's methodology closes a critical gap in the formal verification of cloud infrastructure. [→ Read paper](https://www.usenix.org/conference/osdi24/presentation/sun-xudong) --- ### DRust: Language-Guided Distributed Shared Memory with Fine Granularity, Full Transparency, and Ultra Efficiency *Haoran Ma, Yifan Qiao 0002, Shi Liu, Shan Yu *et al.** **TL;DR** — Exploits Rust's ownership model to implement distributed shared memory at cache-line granularity, achieving near-local performance with no programmer annotations. **Why notable** — Prior DSM systems required explicit data placement or suffered high coherence overhead; DRust shows that a language's ownership semantics can serve as a zero-overhead coherence protocol. [→ Read paper](https://www.usenix.org/conference/osdi24/presentation/ma-haoran) --- ### Nomad: Non-Exclusive Memory Tiering via Transactional Page Migration *Lingfeng Xiang, Zhen Lin, Weishu Deng, Hui Lu 0001 *et al.** **TL;DR** — Enables tiered-memory systems to migrate pages concurrently with ongoing accesses using a transactional protocol, eliminating the stop-the-world pauses of existing page-migration approaches. **Why notable** — CXL-based memory tiering is becoming essential for cost-effective cloud deployments; Nomad's non-exclusive migration is a key enabling mechanism for practical tiering at scale. [→ Read paper](https://www.usenix.org/conference/osdi24/presentation/xiang) --- ### Fast and Scalable In-network Lock Management Using Lock Fission *Hanze Zhang, Ke Cheng, Rong Chen 0001, Haibo Chen 0001* **TL;DR** — Splits a distributed lock into independent sub-locks held in programmable switches, allowing lock acquisition to complete in a single network round-trip without touching any server CPU. **Why notable** — Achieves latencies previously only possible with RDMA using commodity programmable switching hardware; the lock-fission abstraction generalises cleanly to other in-network coordination primitives. [→ Read paper](https://www.usenix.org/conference/osdi24/presentation/zhang-hanze) --- ### Chop Chop: Byzantine Atomic Broadcast to the Network Limit *Martina Camaioni, Rachid Guerraoui, Matteo Monti, Pierre-Louis Roman *et al.** **TL;DR** — Achieves Byzantine fault-tolerant atomic broadcast at near-network-bandwidth rates by batching, pipelining, and carefully overlapping cryptographic operations with network I/O. **Why notable** — Closes the gap between the theoretical throughput of BFT protocols and what commodity hardware can actually deliver; relevant baseline for any production BFT system design. [→ Read paper](https://www.usenix.org/conference/osdi24/presentation/camaioni)