venue: OSDI
year: 2024
date: "2024-07-10"
tags: [llm-serving, distributed-systems, verification, memory, networking, storage]

selected:
  - dblp_key: "conf/osdi/ZhongLCHZL0024"
    title: "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving"
    tldr: "Separates the compute-heavy prefill phase from the memory-bound decoding phase onto different GPU pools, eliminating head-of-line blocking and significantly improving LLM serving throughput."
    why_notable: "Became one of the most influential LLM systems papers of 2024; the prefill–decode disaggregation insight is now widely adopted in production inference stacks (vLLM, SGLang, etc.)."

  - dblp_key: "conf/osdi/0007CLZ0ZGS24"
    title: "Fairness in Serving Large Language Models"
    tldr: "Introduces VTC, a token-count-weighted fair scheduling policy that prevents long-prompt users from monopolising GPU capacity in multi-tenant LLM services."
    why_notable: "First paper to formally study multi-tenant fairness in LLM serving; directly influenced subsequent work on SLA-aware serving and resource allocation in shared inference clusters."

  - dblp_key: "conf/osdi/AgrawalKPMKGTR24"
    title: "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve"
    tldr: "Introduces chunked prefill and stall-free scheduling to decouple throughput and latency goals, letting the same serving system meet both SLOs simultaneously."
    why_notable: "Elegant framing of the throughput–latency tension; chunked prefill became a standard technique in open-source inference engines within months of publication."

  - dblp_key: "conf/osdi/SunHZXZL024"
    title: "Llumnix: Dynamic Scheduling for Large Language Model Serving"
    tldr: "Treats in-flight LLM requests as migratable units, enabling load balancing and SLO recovery by live-migrating KV-cache state across GPU instances."
    why_notable: "Request migration for LLM serving was considered impractical due to KV-cache size; this paper shows it is feasible and impactful, opening a new design dimension for inference schedulers."

  - dblp_key: "conf/osdi/Choudhury0PSJLD24"
    title: "MAST: Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale"
    tldr: "Describes Google's production system for scheduling ML training jobs across geographically distributed datacenters, balancing GPU utilisation, job deadlines, and cross-datacenter bandwidth costs."
    why_notable: "Rare large-scale production paper on global ML scheduling; the insights on heterogeneous cluster management and placement constraints are directly useful for anyone operating multi-site GPU infrastructure."

  - dblp_key: "conf/osdi/LeBlancTBC24"
    title: "SquirrelFS: using the Rust compiler to check file-system crash consistency"
    tldr: "Encodes crash-consistency invariants in Rust's type system so that a file system that compiles is guaranteed not to leave the storage in an inconsistent state after a crash."
    why_notable: "A clean demonstration that language-level type checking can replace runtime or proof-assistant-based verification for an important systems property; the approach is general and practically viable."

  - dblp_key: "conf/osdi/0013MGMCH0PSSX24"
    title: "Anvil: Verifying Liveness of Cluster Management Controllers"
    tldr: "Presents the first framework for mechanically verifying liveness (eventual progress) of Kubernetes-style reconciliation controllers, with proofs for real controllers including ZooKeeper and RabbitMQ operators."
    why_notable: "Liveness proofs for real-world cloud controllers were previously out of reach; Anvil's methodology closes a critical gap in the formal verification of cloud infrastructure."

  - dblp_key: "conf/osdi/MaQLYNLW0K024"
    title: "DRust: Language-Guided Distributed Shared Memory with Fine Granularity, Full Transparency, and Ultra Efficiency"
    tldr: "Exploits Rust's ownership model to implement distributed shared memory at cache-line granularity, achieving near-local performance with no programmer annotations."
    why_notable: "Prior DSM systems required explicit data placement or suffered high coherence overhead; DRust shows that a language's ownership semantics can serve as a zero-overhead coherence protocol."

  - dblp_key: "conf/osdi/XiangLD0RY024"
    title: "Nomad: Non-Exclusive Memory Tiering via Transactional Page Migration"
    tldr: "Enables tiered-memory systems to migrate pages concurrently with ongoing accesses using a transactional protocol, eliminating the stop-the-world pauses of existing page-migration approaches."
    why_notable: "CXL-based memory tiering is becoming essential for cost-effective cloud deployments; Nomad's non-exclusive migration is a key enabling mechanism for practical tiering at scale."

  - dblp_key: "conf/osdi/ZhangC0024"
    title: "Fast and Scalable In-network Lock Management Using Lock Fission"
    tldr: "Splits a distributed lock into independent sub-locks held in programmable switches, allowing lock acquisition to complete in a single network round-trip without touching any server CPU."
    why_notable: "Achieves latencies previously only possible with RDMA using commodity programmable switching hardware; the lock-fission abstraction generalises cleanly to other in-network coordination primitives."

  - dblp_key: "conf/osdi/CamaioniGMRVV24"
    title: "Chop Chop: Byzantine Atomic Broadcast to the Network Limit"
    tldr: "Achieves Byzantine fault-tolerant atomic broadcast at near-network-bandwidth rates by batching, pipelining, and carefully overlapping cryptographic operations with network I/O."
    why_notable: "Closes the gap between the theoretical throughput of BFT protocols and what commodity hardware can actually deliver; relevant baseline for any production BFT system design."