Vincent Lannurien
d822cdaa6a
7.9 KiB
title, venue, year, date, tags, paper_count, draft
| title | venue | year | date | tags | paper_count | draft | ||
|---|---|---|---|---|---|---|---|---|
| SoCC 2024 Digest | SoCC | 2024 | 2024-11-01 |
|
12 | false |
12 papers selected.
Queue Management for SLO-Oriented Large Language Model Serving
Archit Patke, Dhemath Reddy, Saurabh Jha, Haoran Qiu et al.
TL;DR — A queue management framework that enforces latency SLOs for LLM serving by dynamically routing and prioritizing requests across heterogeneous inference capacity.
Why notable — As LLM deployments move into production clouds, meeting strict time-to-first-token and total latency SLOs becomes critical; this work directly addresses that gap with a practical, deployable solution. It is one of the first papers to treat LLM serving as a cloud SLO-management problem rather than a pure model-optimization problem.
Kale: Elastic GPU Scheduling for Online DL Model Training
Ziyang Liu, Renyu Yang, Jin Ouyang, Weihan Jiang et al.
TL;DR — Kale elastically resizes GPU allocations for online DL training jobs in response to real-time resource pressure, improving cluster utilization without violating training progress guarantees.
Why notable — Elastic GPU scheduling is an unsolved pain point in shared ML clusters; Kale's ability to dynamically shrink and expand jobs without checkpointing overhead is directly applicable to production training infrastructure at hyperscalers.
Hops: Fine-grained heterogeneous sensing, efficient and fair Deep Learning cluster scheduling system
Qinghe Wang, Futian Wang, Xinwei Zheng
TL;DR — Hops uses fine-grained, heterogeneity-aware GPU sensing to make scheduling decisions that are simultaneously efficient and fair across diverse DL workloads.
Why notable — Hardware heterogeneity in GPU clusters is the norm, not the exception; Hops provides a principled framework for exploiting that diversity, making it immediately relevant to operators of mixed-generation GPU fleets.
Process-as-a-Service: Unifying Elastic and Stateful Clouds with Serverless Processes
Marcin Copik, Alexandru Calotoiu, Gyorgy Réthy, Roman Böhringer et al.
TL;DR — PraaS introduces a long-lived, stateful serverless process abstraction that bridges the gap between ephemeral FaaS functions and persistent cloud VMs.
Why notable — Statelessness is the central limitation of today's FaaS platforms; this paper proposes a well-grounded new programming model that could reshape how developers think about serverless, backed by implementation and evaluation at scale.
FaPES: Enabling Efficient Elastic Scaling for Serverless Machine Learning Platforms
Xiaoyang Zhao 0005, Siran Yang, Jiamang Wang, Lansong Diao et al.
TL;DR — FaPES achieves fast, fine-grained vertical and horizontal scaling of serverless ML serving pods by decoupling memory provisioning from compute allocation.
Why notable — Elastic scaling for ML inference is a key cost driver in cloud ML platforms; FaPES demonstrates sub-second scaling decisions that reduce both cold-start overhead and resource waste, with results from a production deployment.
Faascale: Scaling MicroVM Vertically for Serverless Computing with Memory Elasticity
Xinmin Zhang, Qiang He 0001, Hao Fan 0006, Song Wu 0001
TL;DR — Faascale enables runtime vertical memory scaling of Firecracker microVMs for serverless functions, eliminating the need to restart or pre-provision fixed memory sizes.
Why notable — Memory over-provisioning is a major cost inefficiency in serverless platforms; Faascale's live memory elasticity directly reduces waste while maintaining the isolation guarantees of microVM-based FaaS.
AutoBurst: Autoscaling Burstable Instances for Cost-effective Latency SLOs
Rubaba Hasan, Timothy Zhu, Bhuvan Urgaonkar
TL;DR — AutoBurst exploits burstable cloud instance types and their CPU credit mechanics to autoscale services at lower cost while still meeting tail-latency SLOs.
Why notable — Burstable instances are widely available on all major clouds yet poorly understood for SLO-sensitive workloads; this paper provides a rigorous autoscaling policy that unlocks significant cost savings without sacrificing latency guarantees.
Dynamic Idle Resource Leasing To Safely Oversubscribe Capacity At Meta
Nishant Gupta, Iyswarya Narayanan, Shivam Handa, Sayak Chakraborti et al.
TL;DR — Meta's production system dynamically lends idle reserved-capacity to opportunistic workloads, recovering stranded compute while ensuring low-latency eviction when owners reclaim resources.
Why notable — This industry paper provides rare visibility into hyperscale capacity management at Meta's scale, demonstrating that safe oversubscription can recover tens of percent of otherwise idle datacenter capacity.
Forecasting Algorithms for Intelligent Resource Scaling: An Experimental Analysis
Yanlei Diao, Dominik Horn, Andreas Kipf, Oleksandr Shchur et al.
TL;DR — A comprehensive empirical study comparing classical and learned forecasting algorithms for cloud autoscaling, yielding concrete guidelines on when each approach wins.
Why notable — Autoscaling relies heavily on workload forecasting, yet practitioners lack principled guidance on algorithm choice; this work from the MIT/AWS group fills that gap with rigorous experimentation across real-world cloud traces.
Vista: Machine Learning based Database Performance Troubleshooting Framework in Amazon RDS
Vikramank Y. Singh, Zhao Song 0001, Balakrishnan (Murali) Narayanaswamy, Kapil Eknath Vaidya et al.
TL;DR — Vista is a production ML framework deployed in Amazon RDS that automatically diagnoses performance regressions by correlating database metrics with causal performance models.
Why notable — Database performance debugging at cloud scale is labor-intensive and error-prone; Vista's deployment in RDS demonstrates how ML-driven root-cause analysis can reduce mean-time-to-resolution for thousands of customer instances.
Inshrinkerator: Compressing Deep Learning Training Checkpoints via Dynamic Quantization
Amey Agrawal, Sameer Reddy, Satwik Bhattamishra, Venkata Prabhakara Sarath Nookala et al.
TL;DR — Inshrinkerator applies dynamic quantization to DL training checkpoints at save time, reducing checkpoint sizes by up to 4x with negligible impact on training convergence.
Why notable — Checkpoint storage and I/O are significant costs in large-scale distributed training; this work provides a transparent, easy-to-adopt compression layer that can be retrofitted into existing training pipelines.
The Sunk Carbon Fallacy: Rethinking Carbon Footprint Metrics for Effective Carbon-Aware Scheduling
Noman Bashir, Varun Gohil, Anagha Belavadi Subramanya, Mohammad Shahrad et al.
TL;DR — The paper argues that conventional carbon metrics misattribute embodied (manufacturing) carbon as a fixed sunk cost, and proposes revised metrics that make carbon-aware scheduling decisions more accurate and actionable.
Why notable — Carbon-aware cloud scheduling is an emerging priority, but flawed metrics can lead to counterproductive decisions; this work from the Delimitrou and Irwin groups provides a conceptual correction with broad implications for green cloud policy and tooling.