--- title: SoCC 2024 Digest venue: SoCC year: 2024 date: '2024-11-01' tags: - cloud-computing - distributed-systems paper_count: 12 draft: false --- 12 papers selected. --- ### Queue Management for SLO-Oriented Large Language Model Serving *Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu *et al.** **TL;DR** — A queue management framework that enforces latency SLOs for LLM serving by dynamically routing and prioritizing requests across heterogeneous inference capacity. **Why notable** — As LLM deployments move into production clouds, meeting strict time-to-first-token and total latency SLOs becomes critical; this work directly addresses that gap with a practical, deployable solution. It is one of the first papers to treat LLM serving as a cloud SLO-management problem rather than a pure model-optimization problem. [→ Read paper](https://doi.org/10.1145/3698038.3698523) --- ### Kale: Elastic GPU Scheduling for Online DL Model Training *Ziyang Liu, Renyu Yang, Jin Ouyang, Weihan Jiang *et al.** **TL;DR** — Kale elastically resizes GPU allocations for online DL training jobs in response to real-time resource pressure, improving cluster utilization without violating training progress guarantees. **Why notable** — Elastic GPU scheduling is an unsolved pain point in shared ML clusters; Kale's ability to dynamically shrink and expand jobs without checkpointing overhead is directly applicable to production training infrastructure at hyperscalers. [→ Read paper](https://doi.org/10.1145/3698038.3698532) --- ### Hops: Fine-grained heterogeneous sensing, efficient and fair Deep Learning cluster scheduling system *Qinghe Wang, Futian Wang, Xinwei Zheng* **TL;DR** — Hops uses fine-grained, heterogeneity-aware GPU sensing to make scheduling decisions that are simultaneously efficient and fair across diverse DL workloads. **Why notable** — Hardware heterogeneity in GPU clusters is the norm, not the exception; Hops provides a principled framework for exploiting that diversity, making it immediately relevant to operators of mixed-generation GPU fleets. [→ Read paper](https://doi.org/10.1145/3698038.3698515) --- ### Process-as-a-Service: Unifying Elastic and Stateful Clouds with Serverless Processes *Marcin Copik, Alexandru Calotoiu, Gyorgy Réthy, Roman Böhringer *et al.** **TL;DR** — PraaS introduces a long-lived, stateful serverless process abstraction that bridges the gap between ephemeral FaaS functions and persistent cloud VMs. **Why notable** — Statelessness is the central limitation of today's FaaS platforms; this paper proposes a well-grounded new programming model that could reshape how developers think about serverless, backed by implementation and evaluation at scale. [→ Read paper](https://doi.org/10.1145/3698038.3698567) --- ### FaPES: Enabling Efficient Elastic Scaling for Serverless Machine Learning Platforms *Xiaoyang Zhao 0005, Siran Yang, Jiamang Wang, Lansong Diao *et al.** **TL;DR** — FaPES achieves fast, fine-grained vertical and horizontal scaling of serverless ML serving pods by decoupling memory provisioning from compute allocation. **Why notable** — Elastic scaling for ML inference is a key cost driver in cloud ML platforms; FaPES demonstrates sub-second scaling decisions that reduce both cold-start overhead and resource waste, with results from a production deployment. [→ Read paper](https://doi.org/10.1145/3698038.3698548) --- ### Faascale: Scaling MicroVM Vertically for Serverless Computing with Memory Elasticity *Xinmin Zhang, Qiang He 0001, Hao Fan 0006, Song Wu 0001* **TL;DR** — Faascale enables runtime vertical memory scaling of Firecracker microVMs for serverless functions, eliminating the need to restart or pre-provision fixed memory sizes. **Why notable** — Memory over-provisioning is a major cost inefficiency in serverless platforms; Faascale's live memory elasticity directly reduces waste while maintaining the isolation guarantees of microVM-based FaaS. [→ Read paper](https://doi.org/10.1145/3698038.3698512) --- ### AutoBurst: Autoscaling Burstable Instances for Cost-effective Latency SLOs *Rubaba Hasan, Timothy Zhu, Bhuvan Urgaonkar* **TL;DR** — AutoBurst exploits burstable cloud instance types and their CPU credit mechanics to autoscale services at lower cost while still meeting tail-latency SLOs. **Why notable** — Burstable instances are widely available on all major clouds yet poorly understood for SLO-sensitive workloads; this paper provides a rigorous autoscaling policy that unlocks significant cost savings without sacrificing latency guarantees. [→ Read paper](https://doi.org/10.1145/3698038.3698530) --- ### Dynamic Idle Resource Leasing To Safely Oversubscribe Capacity At Meta *Nishant Gupta, Iyswarya Narayanan, Shivam Handa, Sayak Chakraborti *et al.** **TL;DR** — Meta's production system dynamically lends idle reserved-capacity to opportunistic workloads, recovering stranded compute while ensuring low-latency eviction when owners reclaim resources. **Why notable** — This industry paper provides rare visibility into hyperscale capacity management at Meta's scale, demonstrating that safe oversubscription can recover tens of percent of otherwise idle datacenter capacity. [→ Read paper](https://doi.org/10.1145/3698038.3698537) --- ### Forecasting Algorithms for Intelligent Resource Scaling: An Experimental Analysis *Yanlei Diao, Dominik Horn, Andreas Kipf, Oleksandr Shchur *et al.** **TL;DR** — A comprehensive empirical study comparing classical and learned forecasting algorithms for cloud autoscaling, yielding concrete guidelines on when each approach wins. **Why notable** — Autoscaling relies heavily on workload forecasting, yet practitioners lack principled guidance on algorithm choice; this work from the MIT/AWS group fills that gap with rigorous experimentation across real-world cloud traces. [→ Read paper](https://doi.org/10.1145/3698038.3698564) --- ### Vista: Machine Learning based Database Performance Troubleshooting Framework in Amazon RDS *Vikramank Y. Singh, Zhao Song 0001, Balakrishnan (Murali) Narayanaswamy, Kapil Eknath Vaidya *et al.** **TL;DR** — Vista is a production ML framework deployed in Amazon RDS that automatically diagnoses performance regressions by correlating database metrics with causal performance models. **Why notable** — Database performance debugging at cloud scale is labor-intensive and error-prone; Vista's deployment in RDS demonstrates how ML-driven root-cause analysis can reduce mean-time-to-resolution for thousands of customer instances. [→ Read paper](https://doi.org/10.1145/3698038.3698519) --- ### Inshrinkerator: Compressing Deep Learning Training Checkpoints via Dynamic Quantization *Amey Agrawal, Sameer Reddy, Satwik Bhattamishra, Venkata Prabhakara Sarath Nookala *et al.** **TL;DR** — Inshrinkerator applies dynamic quantization to DL training checkpoints at save time, reducing checkpoint sizes by up to 4x with negligible impact on training convergence. **Why notable** — Checkpoint storage and I/O are significant costs in large-scale distributed training; this work provides a transparent, easy-to-adopt compression layer that can be retrofitted into existing training pipelines. [→ Read paper](https://doi.org/10.1145/3698038.3698553) --- ### The Sunk Carbon Fallacy: Rethinking Carbon Footprint Metrics for Effective Carbon-Aware Scheduling *Noman Bashir, Varun Gohil, Anagha Belavadi Subramanya, Mohammad Shahrad *et al.** **TL;DR** — The paper argues that conventional carbon metrics misattribute embodied (manufacturing) carbon as a fixed sunk cost, and proposes revised metrics that make carbon-aware scheduling decisions more accurate and actionable. **Why notable** — Carbon-aware cloud scheduling is an emerging priority, but flawed metrics can lead to counterproductive decisions; this work from the Delimitrou and Irwin groups provides a conceptual correction with broad implications for green cloud policy and tooling. [→ Read paper](https://doi.org/10.1145/3698038.3698542)