Harnessing the Power of Serverless Runtimes for Large-Scale Optimization

  • 2019-01-10 13:57:06
  • Arda Aytekin, Mikael Johansson
  • 2

Abstract

The event-driven and elastic nature of serverless runtimes makes them a veryefficient and cost-effective alternative for scaling up computations. So far,they have mostly been used for stateless, data parallel and ephemeralcomputations. In this work, we propose using serverless runtimes to solvegeneric, large-scale optimization problems. Specifically, we build amaster-worker setup using AWS Lambda as the source of our workers, implement aparallel optimization algorithm to solve a regularized logistic regressionproblem, and show that relative speedups up to 256 workers and efficienciesabove 70% up to 64 workers can be expected. We also identify possiblealgorithmic and system-level bottlenecks, propose improvements, and discuss thelimitations and challenges in realizing these improvements.

 

Quick Read (beta)

loading the full paper ...