Abstract

The event-driven and elastic nature of serverless runtimes makes them a veryefficient and cost-effective alternative for scaling up computations. So far,they have mostly been used for stateless, data parallel and ephemeralcomputations. In this work, we propose using serverless runtimes to solvegeneric, large-scale optimization problems. Specifically, we build amaster-worker setup using AWS Lambda as the source of our workers, implement aparallel optimization algorithm to solve a regularized logistic regressionproblem, and show that relative speedups up to 256 workers and efficienciesabove 70% up to 64 workers can be expected. We also identify possiblealgorithmic and system-level bottlenecks, propose improvements, and discuss thelimitations and challenges in realizing these improvements.

 

Introduction (beta)

None

 

Conclusion (beta)

None