We describe TF-Replicator, a framework for distributed machine learningdesigned for DeepMind researchers and implemented as an abstraction overTensorFlow. TF-Replicator simplifies writing data-parallel and model-parallelresearch code. The same models can be effortlessly deployed to differentcluster architectures (i.e. one or many machines containing CPUs, GPUs or TPUaccelerators) using synchronous or asynchronous training regimes. Todemonstrate the generality and scalability of TF-Replicator, we implement andbenchmark three very different models: (1) A ResNet-50 for ImageNetclassification, (2) a SN-GAN for class-conditional ImageNet image generation,and (3) a D4PG reinforcement learning agent for continuous control. Our resultsshow strong scalability performance without demanding any distributed systemsexpertise of the user. The TF-Replicator programming model will be open-sourcedas part of TensorFlow 2.0 (seehttps://github.com/tensorflow/community/pull/25).
Quick Read (beta)
We describe TF-Replicator, a framework for distributed machine learning designed for DeepMind researchers and implemented as an abstraction over TensorFlow. TF-Replicator simplifies writing data-parallel and model-parallel research code. The same models can be effortlessly deployed to different cluster architectures (i.e. one or many machines containing CPUs, GPUs or TPU accelerators) using synchronous or asynchronous training regimes. To demonstrate the generality and scalability of TF-Replicator, we implement and benchmark three very different models: (1) A ResNet-50 for ImageNet classification, (2) a SN-GAN for class-conditional ImageNet image generation, and (3) a D4PG reinforcement learning agent for continuous control. Our results show strong scalability performance without demanding any distributed systems expertise of the user. The TF-Replicator programming model will be open-sourced as part of TensorFlow 2.0 (see https://github.com/tensorflow/community/pull/25).
oddsidemargin has been altered.
marginparsep has been altered.
topmargin has been altered.
marginparwidth has been altered.
marginparpush has been altered.
paperheight has been altered.
The page layout violates the ICML style. Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you. We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again.
TF-Replicator: Distributed machine learning for researchers
Peter Buchlovsky* 0 David Budden* 0 Dominik Grewe* 0 Chris Jones* 0 John Aslanides 0 Frederic Besse 0 Andy Brock 0 Aidan Clark 0 Sergio Gómez Colmenarejo 0 Aedan Pope 0 Fabio Viola 0 Dan Belov 0
* Denotes equal contribution \@xsect
(Amodei & Hernandez, 2018) demonstrated that the amount of compute used for the largest machine learning training runs has increased exponentially with a 3.5 month-doubling period since 2012 (compared to 18 months historically for Moore’s Law). Although there have been noteworthy improvements in the vertical scalability of machine learning systems (e.g. by leveraging GPUs, and recently the introduction of custom accelerators such as Google’s TPU (Jouppi et al., 2017)), researchers have been forced to scale horizontally across massively distributed compute clusters to conduct experiments of this magnitude.
Researchers faced with building a distributed system are currently required to make a steep trade-off between simplicity and generality. High-level abstractions have been developed to simplify common use-cases, e.g. the parameter server model for synchronous SGD as supported by TensorFlow’s combination of replica_device_setter and SyncReplicasOptimizer. Horovod (Sergeev & Del Balso, 2018) supports similar applications by spawning multiple data-parallel TensorFlow graphs, using MPI or NCCL all_reduce implementations for efficient communication of gradients (NVIDIA, 2017). The TensorFlow Estimator was developed to address certain models (e.g. generative adversarial networks (GANs) (Goodfellow et al., 2014)) in a device-agnostic manner, but deny researchers the freedom of naturally defining their own run loops.
The number of models not addressed by these abstractions is growing rapidly. In the original TensorFlow paper, the authors cited multi-loss methods and reinforcement learning (RL) agents as poorly suited to the parameter server-based model of its DistBelief predecessor (Dean et al., 2012). Since TensorFlow’s release, multi-loss methods have proliferated, including proximal gradient TD-learning (Liu et al., 2016), multi-level optimization (Pfau & Vinyals, 2016), synthetic gradients (Jaderberg et al., 2016) and RL agents that interact with multiple losses in increasingly complex ways (e.g. hierarchical RL (Vezhnevets et al., 2017)). Many popular machine learning frameworks provide low-level primitives that can in principle be used to implement such systems, but their usage requires a deep understanding of distributed system architectures.
In this paper we present TF-Replicator, designed and developed in close collaboration with DeepMind researchers across several machine learning disciplines. In Section id1, we provide an overview of the low-level mechanisms and common patterns for distributed TensorFlow, which are recurring concepts throughout the remainder of the paper. In Section id1 we provide an overview of TF-Replicator. Starting with the API used to define a replica, we demonstrate painless generalization across different system architectures and devices, and show optional arguments to configure more complex use-cases (e.g. combining data and model parallelism). Section id1 provides an overview of TF-Replicator’s implementation details. The remainder of the paper demonstrates how TF-Replicator can be applied to achieve compelling, scalable performance for various machine learning models in both GPU and TPU clusters.
We start with a brief description of the underlying framework. TensorFlow is a computation-graph based machine learning framework, wherein a user builds a graph consisting of tensors, representing multidimensional arrays of data; and operations (ops), which consume and produce tensors. A user can request to see the value of a tensor, entailing the execution of any ancestor operations.
Each machine in a distributed TensorFlow cluster launches a TensorFlow server, communicating over gRPC. The server exports two RPC services: the master, which coordinates and provides access to a set of distributed machines; and the worker, responsible for executing subgraphs using its local devices. On one or more machines, a client binary (usually a Python script) defines a TensorFlow graph. Subgraphs are dispatched by the master to the workers for execution. One master can dispatch work to multiple workers, and one worker can perform work for multiple masters (e.g. the parameter server model (Li et al., 2014), whereby parameter server workers store and update variable weights accessible from many parallel replicas).
During graph construction, device strings are used to identify devices (e.g. a specific GPU) in a cluster. When device “A” requires the result of an operation with device string “B”, its master sends a gRPC to device “B” asking its worker to run the operation. During preprocessing, TensorFlow identifies all tensors produced on one device but consumed by an operation on another, and inserts send and recv nodes to transfer data between devices. The details depend on the devices in question, for example cudaMemcpyAsync for local CPU GPU and gRPC over TCP in the distributed setting (Abadi et al., 2016). Relative device strings are mapped to physical network addresses in a ClusterSpec dictionary provided to each server at initialization.
So that two clients can share state, TensorFlow defines resources (e.g. variables), objects with a unique name in addition to their device string. If two clients define a resource with identical device and name, only a single resource is allocated on the relevant device, and referring operations from either client will share the underlying data.
TensorFlow provides a versatile platform for distributed computing with a high degree of freedom, but little abstraction. To capture common use cases, we shall use the key concept of a replica: a computation designed to be run in parallel across many devices (e.g. one step of SGD), with different input to each device, and some shared state accessible to all replicas (e.g. model parameters). Many machine learning techniques (e.g. Hogwild! (Recht et al., 2011), Downpour SGD (Dean et al., 2012) and A3C (Mnih et al., 2016)) can be seen as computations involving many replicas with differing synchronicity of the shared parameters. There are many ways to construct a system with replicas, but two common patterns are in-graph replication and between-graph replication (illustrated in Figure 1).
For in-graph replication (Figure 1(a)), a single client binary constructs one replica on each device (all contained in a massive multi-device graph), each sharing a set of parameters (resources) placed on the client’s own device. That client’s master service then coordinates synchronous execution of all replicas: feeding data and fetching outputs.
For between-graph replication (Figure 1(b)), multiple machines run a client binary, each creating a graph with a single replica on a local device, alongside resources on a shared device (i.e. the parameter server model). Because this relies on TensorFlow’s name resolution to share state, care must be taken to ensure each client creates resources identically.
Though TensorFlow supports near-arbitrary replication, implementation is often difficult. Handling state is non-trivial, with incorrect behavior often leading to silent error. Coordinating heterogeneous behavior among replicas (not yet described), necessary for many methods in reinforcement learning, adds even more complexity.
Google recently unveiled its TPU v2 devices, each with 180 teraflops and 64 GB of high bandwidth memory across four chips of two cores. These devices can be connected in a 2-dimensional toroidal mesh network to form a 11.5 petaflop TPU “Pod” (Google, 2018). Leveraging the power of these devices demands a specific replication pattern, and requires the user to manually handle asynchronous data transfer using InfeedQueue and OutfeedQueue ops. Building an efficient system within these constraints adds further complexity to distributed TensorFlow, and was a key influence in our decision to build a new abstraction for replicated machine learning computation.
TF-Replicator trivializes the process of building distributed machine learning systems by allowing researchers to naturally define their model and run loop as per the single-machine setting. This is achieved by providing a simple API for defining a (potentially model-parallel) replica, and a set of in-built Replicators that support deployment across different system architectures (in-graph versus between-graph), training regimes (synchronous versus asynchronous) and hardware (GPU and TPU). Unlike TensorFlow Estimator, it makes few assumptions about model structure and is therefore readily extensible to novel use cases.
In TF-Replicator, a replica is defined by two functions: input_fn for describing how a replica receives its input, and step_fn for describing the computation performed by the replica (detailed in Section id1). To deploy a replica, the user builds one of the in-built Replicator implementations detailed in Table 1. Any resource constructed within the Replicator context is itself replicated, e.g. the model and optimizer dataflow graphs. The Replicator wrap_optimizer function coordinates variable updates by all_reduce averaging of the computed gradients. Note that TF-Replicator exposes several of these MPI-style primitives for scenarios requiring inter-replica communication, as detailed in Section id1. Bringing the features together, a user might deploy an ImageNet classifier on a single TPU v2 (8 cores across 4 chips) as follows:
A summary of currently supported replica implementations is provided in Table 1. Maximum workers refers to the number of separate machines available to the distributed system, e.g. a server with many GPUs or a TPU device constituting 8 cores split across 4 chips. Devices per worker refers to the number of physical devices available on each of these workers, e.g. each GPU on a server. Note that for the MultiGpuReplicator, MultiWorkerReplicator and TpuReplicator, the in-graph replication pattern is adopted for synchronous training.
Machine learning workloads using asynchronous SGD are still common-place, as popularized by Hogwild! (Recht et al., 2011). In addition to the performance benefits inherent to lock-free distributed communication, asynchronous systems adopting the between-graph replication pattern are more fault tolerant, as any worker can fail while the rest continue unaffected. They also exhibit a reduced memory footprint, as the replica subgraph is not replicated on the master service for each device. This is of particular importance for very large systems or models, as TensorFlow graphs serialized as protocol buffers have a maximum size of 2GB. To address these scenarios we include the MultiWorkerAsyncReplicator, which implements the standard between-graph parameter server architecture.
To define a replica, users define an input_fn (input pipeline for the replica) and a step_fn (work performed by the replica) in their TensorFlow client.
The input_fn is a Python function that builds the input pipeline for a single model step, and can return either a tf.data.Dataset or a callable that returns an arbitrary nested structure of tensors.
TF-Replicator supports two types of input pipeline replication: PER_WORKER and PER_REPLICA. In the former, the datasets or callables produced by a single input_fn instance are split across replicas according to an input_split_fn, which defaults to partitioning on the leading (batch) dimension. An example of where this function is useful to override is for batch-parallelism over time-major sequences to be used with recurrent networks. In the latter, each replica (e.g. GPU or TPU core) maintains its own copy of the input pipeline.
The step_fn is a Python function that builds the dataflow graph for one step of one replica. It ingests the data emitted by the input_fn and is executed on the targeted accelerator device (e.g. GPU or TPU). The step_fn output can be any nested structure of tensors and is more general than existing solutions as it allows the execution of arbitrary TensorFlow, e.g. multiple losses and optimizers:
|Implementation||Maximum workers||Max devices per worker||Synchronous||Parameter Servers|
|TpuReplicator||TPU Pod||8 TPU cores||Yes||No|
One motivation of TF-Replicator is to provide general support over both data and model parallel workloads. To accomplish this, we extend the notion of a replica to support dataflow graphs that themselves span multiple devices, e.g. as necessary for models with very large memory footprints (Krizhevsky et al., 2012). This is achieved by introducing a logical_device function responsible for mapping logical device IDs within the replica step_fn to global device contexts. Such model parallelism can be combined with the data parallel features described above, such as this example of deploying 4 data-parallel instances of a 2-device model-parallel replica:
Although replicated computation lends itself to most of our users’ machine learning workflows, there are use cases where communication between replicas is necessary. To address these scenarios, TF-Replicator supports MPI-style primitives (similar to torch.distributed) that can be used to implement bespoke cross-replica reductions. Specifically, TF-Replicator implements: all_reduce, all_sum, all_gather, broadcast, map_reduce and map_gather. The most obvious example requiring such cross-replica communication is the aggregation of gradients, as implemented in the wrap_optimizer function. We achieve this in a similar manner to Horovod’s DistributedOptimizer (Sergeev & Del Balso, 2018), by leveraging TF-Replicator’s all_sum primitive within the optimizer’s apply_gradients function:
All communication primitives are exposed to the user, allowing for the implementation of models that require cross-replica statistics. For example, the all_sum primitive can be used to perform cross-replica batch normalization:
Note that not all primitives are supported by the MultiWorkerAsyncReplicator.
TF-Replicator users describe computation in terms of the work performed by a single replica, including any communication between replicas (e.g. gradient accumulation). The implementation of such communication in TensorFlow is a non-trivial problem, as we allow the user to call these primitives anywhere within their Python step_fn. If these primitives were only allowed before or after the step_fn was called, i.e. while there is a global view of all replica’s inputs or outputs, then cross-replica communication could be trivially stitched together. Instead, one replica’s step_fn can call a primitive mid graph construction. Because TensorFlow constructs dataflow graphs in a feedforward manner, this requires referring to data coming from another replica that itself is yet to be built.
The details of how computation is distributed across devices, and how communication primitives are implemented, depends on the specific Replicator instance (see Table 1). The simplest case is MultiWorkerAsyncReplicator, which only supports gradient accumulation between replicas but no other communication primitives. Each worker builds its own graph according to the user-provided step_fn. Asynchronous gradient accumulation is implemented by placing trainable variables on parameter servers that are shared across workers, as per the standard parameter server model (Li et al., 2014). This corresponds to the between-graph replication pattern described in Section id1.
The MultiGpuReplicator and MultiWorkerReplicator classes implement the in-graph replication pattern. We therefore need to build a TensorFlow graph containing multiple instances of the user-provided step_fn (i.e. one per replica) and handle cross-replica communication in the dataflow graph itself. One solution to the problem of feedforward graph construction in TensorFlow is to build all replicas in parallel in separate Python threads, inserting barriers where cross-replica communication is required (i.e. as implemented in TensorFlow’s DistributionStrategy). Instead, we chose to insert placeholder ops into each graph at construction time where cross-replica communication is required. This allows each replica to be constructed independently and without the complexities of TensorFlow thread safety, as the placeholders can be rewritten once all replica subgraphs are finalized.
The TpuReplicator also uses the in-graph replication across multiple TPU devices, and relies on Google’s Accelerated Linear Algebra (XLA) compiler (Google, 2017) to implement cross-replica communication (i.e. using the tf.contrib.tpu.cross_replica_sum op).
Since the celebrated victory of AlexNet at the 2012 Large Scale Visual Recognition Challenge (ILSVRC) (Krizhevsky et al., 2012), ImageNet classification has become a standard benchmark for evaluating deep convolutional neural networks (ConvNets). Successive ILSVRC winners such as GoogLeNet (Szegedy et al., 2015) and VGG (Simonyan & Zisserman, 2014) dramatically improved classification performance, reducing Top-5 error rate from 15% to 5%. The ILSVRC15-winning Residual Neural Network (ResNet (He et al., 2016)) introduced the concept of skip connections, allowing ConvNets many hundreds or thousands of layers deep to be trained successfully. A 152-layer ResNet exceeded human-level ImageNet performance with a Top-5 error rate of 3.57%, and the slightly shallower ResNet-50 architecture has since emerged as the gold standard for image classification problems.
Instead of attempting to improve classification accuracy, many recent papers have focused on reducing the time taken to achieve some performance threshold (typically 75% Top-1 accuracy). Aside from the growing body of literature on low-precision training (De Sa et al., 2018; Micikevicius et al., 2017), these studies have typically leveraged weak scaling to train in fewer steps with very large batches. (Goyal et al., 2017) trained a ResNet-50 in 1 hour with batch size of 8K on 256 GPUs. Subsequent studies (Codreanu et al., 2017; Smith et al., 2017; You et al., 2018) reduced this time to 15 minutes through a combination of larger batches and more GPUs (e.g. batch size 32K with 1024 Tesla P100s in (Akiba et al., 2017)), along with algorithmic tricks such as batch normalization and learning rate scheduling. (Baidu, 2017) introduced the idea of a ring-based all-reduce for faster cross-replica gradient updates, and similar methods have been applied successfully by (Sergeev & Del Balso, 2018) and (Jia et al., 2018), who reduced training time to 6.6 minutes using 2048 Tesla P40 GPUs.
In this Section we demonstrate how TF-Replicator can be used to obtain competitive performance on these ImageNet benchmarks, using many GPUs or TPUs in an in-graph replicated system to weak-scale a ResNet-50 ConvNet.
We use TF-Replicator to implement a ResNetv1-50 model matching that benchmarked in the Cloud TPU documentation (Google, 2018; Goyal et al., 2017). Each TPU core or GPU receives a batch size of 64 images (32 for the largest TPU configuration, 32x TPUv2). We use a Nesterov momentum optimizer (Sutskever et al., 2013) with a learning rate that “warms up” linearly over 6 epochs to a maximum of , followed by 90% decay at epochs 30, 60 and 80. Note that this differs from the Cloud TPU example due to our use of cross-replica gradient averaging (versus gradient summing in the TensorFlow TPUEstimator). Importantly, our GPU and TPU implementations use the exact same code except for the targeting of the MultiGpuReplicator and TpuReplicator respectively, each with a PER_WORKER-replicated input pipeline (note that there is one TPU “host” per device, i.e. per 4 chips / 8 cores).
Our ImageNet classification results are presented in Figure 2, in terms of the standard log loss, Top-1 and Top-5 accuracy measures. The final Top-1 accuracy and training time for 90 epochs is additionally presented in Table 2. Consistent with previous studies, weak-scaling from 1 to many GPU or TPU accelerators provides dramatic improvement in training time. TF-Replicator additionally allows us to deploy this model on Cloud TPU hardware, and on 32 TPUv2 devices we are able to match the published 75.3% Top-1 accuracy in less than 30 minutes of training. Note that these results are obtained using the standard TF-Replicator implementation, without any systems optimization specific to ImageNet classification.
|Device||Top-1 Accuracy||Time (min)|
|32x GPU (4x8)||76.46%||330|
|64x GPU (8x8)||76.14%||182|
Generative models can be broadly characterized as models which learn to produce samples from a target distribution. The ability to directly model and sample from complex, high-dimensional distributions is an important task in machine learning, and permits applications ranging from speech synthesis (Oord et al., 2016a) to image super-resolution (Ledig et al., 2017). For modeling images, likelihood-based methods (Dinh et al., 2016; Kingma & Welling, 2013; Oord et al., 2016b) and implicit models (Goodfellow et al., 2014; Li et al., 2015; Mohamed & Lakshminarayanan, 2016) are the dominant approaches.
Of particular recent interest are Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) which frame the training process as a two-player minmax game wherein one network, the generator, attempts to produce realistic samples that fool the other network, the discriminator, which must learn to distinguish real and fake samples. Formally, the vanilla GAN objective, , is defined as:
where typically , and and are ConvNet trained through alternating gradient descent.
GANs are notoriously difficult to train, but with a recent torrent of research into stabilizing the process (some focused on modifying the objective (Arjovsky et al., 2017), others focused on improving the network conditioning (Gulrajani et al., 2017)) the community has settled on a reasonably robust setup, enabling training on even complex, multimodal datasets such as ImageNet. A particularly promising variant is Spectral Normalization GAN (SN-GAN) (Miyato et al., 2018), which stabilizes training by normalizing each discriminator parameter by its top singular value, ensuring Lipschitz continuity and promoting convergence.
In this Section, we benchmark a class-conditional SN-GAN trained on ImageNet. We leverage TF-Replicator to train on much larger batches than can fit on a single GPU, and find that this leads to sizable gains in sample quality (as measured by the Inception Score (Salimans et al., 2016) and Fréchet Inception Distance (Heusel et al., 2017)). Without any adaptations such as changing the learning rate, simply increasing the batch size from 64 (as in (Miyato et al., 2018)) to 512 improves the Inception Score by a relative 50%.
We implement SN-GAN using TF-Replicator using the same hyperparameters as (Zhang et al., 2018). Class conditioning is provided via class-conditional BatchNorm (Dumoulin et al., 2017) and Projection Discrimination (Miyato & Koyama, 2018). As TF-Replicator is agnostic to the number of losses defined for a model, the alternating generator and discriminator optimization steps required no additional engineering effort. We adopt the weak-scaling approach used for ImageNet classification, scaling large batches across multiple GPU and TPU devices. Our implementation further benefits from the application of cross-replica BatchNorm (implemented with TF-Replicator’s all_sum primitive), yielding a further performance boost of 5-10%.
Our SN-GAN sample quality results are presented in Figure 3. By leveraging 8 GPUs (or a set of TPUv2 devices) we are able to scale to larger batches than can be fit on a single GPU, yielding substantial improvement in sample quality.
In a standard reinforcement learning (RL) setup, an agent seeks to maximize the return (-discounted sum of future rewards) accumulated by interacting with an environment in discrete time. At each timestep the agent makes observation , takes actions and receives rewards . The behavior of the agent is controlled by a policy , which maps each observation to an action. The state-action value function, describes the expected return conditioned on first taking an action from state and following the policy thereafter:
where , , and . In the scenario of discrete actions (e.g. Atari (Mnih et al., 2015)), the optimal policy can be derived by maximizing with respect to , where is learned via fixed-point iteration of a Bellman operator.
RL is complicated by continuous actions (e.g. robotics), with many successful algorithms instead learning a parameterized policy directly by optimizing . In the case of the deterministic policy gradient (DPG) agent (Lillicrap et al., 2015; Silver et al., 2014), the gradient of this objective is approximated as:
where an actor network parameterizes the policy, , and a critic network approximates the true value of the current policy, . Note that similar to our GAN example, this use case requires us to support optimization over multiple losses within the step_fn. Additionally, we do not require the state-visitation distribution, , to be associated with the current policy. This allows us to decouple the process of acting (following a stochastic policy to encourage exploration) from learning the optimal deterministic policy. We accomplish this using a replay buffer of experiences, as introduced in DQN (Mnih et al., 2015).
In this Section we describe and benchmark an implementation of the D4PG agent (Barth-Maron et al., 2018) on the DeepMind Control Suite (Tassa et al., 2018), a standardized set of physically-realistic control tasks of varying difficulty. D4PG is an off-policy actor-critic algorithm that extends DPG with a set of recent algorithmic advancements, e.g. a distributional Bellman operator. Importantly, unlike the benchmarks provided in the D4PG paper, the scalability of TF-Replicator allows us to quickly solve these tasks purely from pixel observations rather than relying on a low-dimensional representation of joint positions and velocities.
Decoupling the processes of acting and learning is key to distributed RL agents, as both components can be independently and horizontally scaled. First, many independent actor processes can be deployed in parallel, each following an exploration policy to generate observation and reward statistics that can be added asynchronously to a global memory store (e.g. a replay buffer in the off-policy case). Previous RL agents built on distributed TensorFlow (e.g. IMPALA (Espeholt et al., 2018) and ApeX (Horgan et al., 2018)) achieve this within a monolithic TensorFlow graph, with careful application of device scopes and variable pinning to enforce the intended behavior. Second, the learner process can be scaled in much the same way as a standard supervised learning workflow by performing SGD updates over larger batches of sampled actor data.
Our implementation of D4PG differs from the published version. Instead of an in-graph replay buffer, our replay is implemented out-of-graph as a memcached-style key-value store. Many independent actor processes interact with their own instance of the Control Suite environment to asynchronously generate transitions and write them to a replay buffer. The learner is implemented as a TF-Replicator replica, with additional processes responsible for sampling batches from replay and exposing a callable compatible with the replica input_fn. This allows us to easily data-parallelize batches using any of the architectures or Replicators described in previous sections. The learner periodically serves out-of-graph variable update requests from RPC clients maintained by each actor, ensuring that they follow a policy centred on the most recent model parameters.
Unlike the ResNet-50 and SN-GAN examples above, D4PG is more difficult to scale as larger batch sizes do not necessarily lead to better performance (in fact for many tasks in the Control Suite, we observe that larger batches have a negative impact). We instead use this opportunity to assess TF-Replicator’s performance in scenarios requiring strong-scalability, i.e. a fixed total batch (256, the largest we could fit on a single GPU) split across multiple devices.
Consistent with the pixel benchmarking described in (Tassa et al., 2018), we consider 84x84 RGB observations representing the default camera for each task. Noting that previous studies have learnt from low-dimensional proprioceptive state that captures both the position and velocity of each actuator, we additionally frame-stack frames to allow the inference of velocity information. Our agent uses the same actor and critic network architectures described in (Barth-Maron et al., 2018), and the input to each is the length-64 embedding produced by a shared 9-layer ResNet over the frame-stacked pixel observations. For each experiment we use 32 actor and 8 sampler processes per learner replica to maintain a roughly equivalent acting/learning ratio (to prevent over-fitting to stale data). All other hyperparameters are as per (Barth-Maron et al., 2018).
Figure 4 presents our D4PG agent performance (mean episode reward) as a function of wall-clock training time. It is evident that in this strong-scalability regime and with a modest batch size of 256, TF-Replicator allows for substantial acceleration of agent training time. Consistent with our ResNet-50 and SN-GAN weak-scalability results, we observe that a single TPUv2 device (8 cores across 4 chips) provides competitive performance compared to 8 NVLink-connected Tesla V100 GPUs.
Early systems for distributed machine learning built upon the popular MapReduce batch dataflow architecture (Dean & Ghemawat, 2008). MapReduce adopts the functional programming and primitives, and provides a fault tolerant architecture that could scale to many thousand machines. Examples of MapReduce-based systems for machine learning include Mahout (Apache, 2012), based on Hadoop (Shvachko et al., 2010), and MLI (Sparks et al., 2013), based on Spark (Zaharia et al., 2012). Although these systems provided early success when scaled to tens of machines, the MapReduce paradigm is relatively ill-suited to scaling the iterative computation of deep neural networks (Abadi et al., 2016). For example, the SparkNet system (based on Spark) requires 20 seconds to broadcast weights and collect updates from just 5 workers, despite extending earlier frameworks to store intermediate results in memory (Moritz et al., 2015).
Subsequent systems have extended the parameter server model, introduced by (Smola & Narayanamurthy, 2010) and later popularized by (Li et al., 2013; 2014) for machine learning. This architecture separates tasks into two jobs: stateless workers, which perform model computations; and stateful parameter servers, which store sharded network weights. The parameter server itself was inspired largely by memcached (Fitzpatrick, 2004), exposing push(k,v) and pull(k) functions that can be used for applying gradient-based updates to shared neural network weights. Examples of parameter server-based systems for machine learning include DistBelief (Dean et al., 2012), GeePS (Cui et al., 2016) and Project Adam (Chilimbi et al., 2014). A particularly noteworthy example is MXNet (Chen et al., 2015), which uses dataflow graphs to represent worker computation in a similar manner to TensorFlow (Abadi et al., 2016).
More recent frameworks include Mesh-TensorFlow (Shazeer et al., 2018), which defines a new language that simplifies data and model-parallelism by allowing the user to explicitly map sub-computations to mesh-connected hardware; and Ray (Moritz et al., 2018), which unifies simulation, training and serving for distributed reinforcement learning models. Both frameworks have demonstrated impressive scalability performance, e.g. Mesh-TensorFlow achieves state-of-the-art English-to-French translation by training a 5-billion parameter Transformer model on 512 TPU cores. TF-Replicator instead aims to promote rapid iteration of research ideas, allowing researchers to scale their existing, single-machine models to a distributed system without the need to learn new tools or reason explicitly about network topology.
We have presented TF-Replicator, and demonstrated that it is a useful abstraction that simplifies the process of scaling general machine learning models. Without any distributed systems expertise, the user is able to leverage the power of clusters of GPUs or TPUs to achieve results competitive with complex hand-crafted systems. We believe that massive scalability will continue to be an important ingredient of machine learning research, and that TF-Replicator will be a useful tool for enabling impactful outcomes in this domain.
Contributions TF-Replicator design: F.B., P.B., D.G., C.J., F.V.; TF-Replicator implementation: P.B., C.J.; Experimental design and paper writing: D.B.; ImageNet experiments: S.G.C.; D4PG experiments: J.A., D.B., A.C.; GAN experiments: A.B.; General support: D.B., D.G., A.P.
Acknowledgements Thanks to Tim Harley, Tom Hennigan, Karen Simonyan, Koray Kavukcuoglu and the entire DeepMind research community.
- Abadi et al. (2016) Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. Tensorflow: a system for large-scale machine learning. In OSDI, volume 16, pp. 265–283, 2016.
- Akiba et al. (2017) Akiba, T., Suzuki, S., and Fukuda, K. Extremely large minibatch SGD: Training ResNet-50 on ImageNet in 15 minutes. arXiv preprint arXiv:1711.04325, 2017.
- Amodei & Hernandez (2018) Amodei, D. and Hernandez, D. AI and Compute. https://blog.openai.com/ai-and-compute/, 2018.
- Apache (2012) Apache. Mahout Project. http://mahout.apache.org/, 2012.
- Arjovsky et al. (2017) Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein GAN. arXiv preprint arXiv:1701.07875, 2017.
- Baidu (2017) Baidu. Bringing HPC techniques to deep learning. http://research.baidu.com/bringing-hpc-techniques-deep-learning, 2017.
- Barth-Maron et al. (2018) Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., Muldal, A., Heess, N., and Lillicrap, T. Distributed distributional deterministic policy gradients. International Conference on Learning Representations (ICLR), 2018.
- Chen et al. (2015) Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., and Zhang, Z. MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274, 2015.
- Chilimbi et al. (2014) Chilimbi, T. M., Suzue, Y., Apacible, J., and Kalyanaraman, K. Project Adam: Building an efficient and scalable deep learning training system. In OSDI, volume 14, pp. 571–582, 2014.
- Codreanu et al. (2017) Codreanu, V., Podareanu, D., and Saletore, V. Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train. arXiv preprint arXiv:1711.04291, 2017.
- Cui et al. (2016) Cui, H., Zhang, H., Ganger, G. R., Gibbons, P. B., and Xing, E. P. GeePS: Scalable deep learning on distributed GPUs with a GPU-specialized parameter server. In Proceedings of the Eleventh European Conference on Computer Systems, pp. 4. ACM, 2016.
- De Sa et al. (2018) De Sa, C., Leszczynski, M., Zhang, J., Marzoev, A., Aberger, C. R., Olukotun, K., and Ré, C. High-accuracy low-precision training. arXiv preprint arXiv:1803.03383, 2018.
- Dean & Ghemawat (2008) Dean, J. and Ghemawat, S. MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1):107–113, 2008.
- Dean et al. (2012) Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Senior, A., Tucker, P., Yang, K., Le, Q. V., et al. Large scale distributed deep networks. In Advances in neural information processing systems, pp. 1223–1231, 2012.
- Dinh et al. (2016) Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density estimation using Real NVP. arXiv preprint arXiv:1605.08803, 2016.
- Dumoulin et al. (2017) Dumoulin, V., Shlens, J., and Kudlur, M. A learned representation for artistic style. International Conference on Learning Representations (ICLR), 2017.
- Espeholt et al. (2018) Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. IMPALA: Scalable distributed Deep-RL with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018.
- Fitzpatrick (2004) Fitzpatrick, B. Distributed caching with memcached. Linux journal, 2004(124):5, 2004.
- Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680, 2014.
- Google (2017) Google. XLA (Accelerated Linear Algebra). https://www.tensorflow.org/performance/xla/, 2017.
- Google (2018) Google. Cloud TPU. https://cloud.google.com/tpu/, 2018.
- Goyal et al. (2017) Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch SGD: training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
- Gulrajani et al. (2017) Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pp. 5767–5777, 2017.
- He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
- Heusel et al. (2017) Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626–6637, 2017.
- Horgan et al. (2018) Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., Van Hasselt, H., and Silver, D. Distributed prioritized experience replay. International Conference on Learning Representations (ICLR), 2018.
- Jaderberg et al. (2016) Jaderberg, M., Czarnecki, W. M., Osindero, S., Vinyals, O., Graves, A., Silver, D., and Kavukcuoglu, K. Decoupled neural interfaces using synthetic gradients. arXiv preprint arXiv:1608.05343, 2016.
- Jia et al. (2018) Jia, X., Song, S., He, W., Wang, Y., Rong, H., Zhou, F., Xie, L., Guo, Z., Yang, Y., Yu, L., et al. Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes. arXiv preprint arXiv:1807.11205, 2018.
- Jouppi et al. (2017) Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. In-datacenter performance analysis of a tensor processing unit. In Computer Architecture (ISCA), 2017 ACM/IEEE 44th Annual International Symposium on, pp. 1–12, 2017.
- Kingma & Welling (2013) Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
- Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
- Ledig et al. (2017) Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A. P., Tejani, A., Totz, J., Wang, Z., et al. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR, 2017.
- Li et al. (2013) Li, M., Zhou, L., Yang, Z., Li, A., Xia, F., Andersen, D. G., and Smola, A. Parameter server for distributed machine learning. In Big Learning NIPS Workshop, volume 6, pp. 2, 2013.
- Li et al. (2014) Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A., Josifovski, V., Long, J., Shekita, E. J., and Su, B.-Y. Scaling distributed machine learning with the parameter server. In OSDI, volume 14, pp. 583–598, 2014.
- Li et al. (2015) Li, Y., Swersky, K., and Zemel, R. Generative moment matching networks. In International Conference on Machine Learning, pp. 1718–1727, 2015.
- Lillicrap et al. (2015) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
- Liu et al. (2016) Liu, B., Liu, J., Ghavamzadeh, M., Mahadevan, S., and Petrik, M. Proximal gradient temporal difference learning algorithms. In IJCAI, pp. 4195–4199, 2016.
- Micikevicius et al. (2017) Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaev, O., Venkatesh, G., et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017.
- Miyato & Koyama (2018) Miyato, T. and Koyama, M. cGANs with projection discriminator. arXiv preprint arXiv:1802.05637, 2018.
- Miyato et al. (2018) Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
- Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
- Mnih et al. (2016) Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928–1937, 2016.
- Mohamed & Lakshminarayanan (2016) Mohamed, S. and Lakshminarayanan, B. Learning in implicit generative models. arXiv preprint arXiv:1610.03483, 2016.
- Moritz et al. (2015) Moritz, P., Nishihara, R., Stoica, I., and Jordan, M. I. Sparknet: Training deep networks in spark. arXiv preprint arXiv:1511.06051, 2015.
- Moritz et al. (2018) Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., et al. Ray: A distributed framework for emerging AI applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pp. 561–577, 2018.
- NVIDIA (2017) NVIDIA. NVIDIA Collective Communications Library (NCCL). https://developer.nvidia.com/nccl, 2017.
- Oord et al. (2016a) Oord, A. v. d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016a.
- Oord et al. (2016b) Oord, A. v. d., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 2016b.
- Pfau & Vinyals (2016) Pfau, D. and Vinyals, O. Connecting generative adversarial networks and actor-critic methods. arXiv preprint arXiv:1610.01945, 2016.
- Recht et al. (2011) Recht, B., Re, C., Wright, S., and Niu, F. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in neural information processing systems, pp. 693–701, 2011.
- Salimans et al. (2016) Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp. 2234–2242, 2016.
- Sergeev & Del Balso (2018) Sergeev, A. and Del Balso, M. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv:1802.05799, 2018.
- Shazeer et al. (2018) Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young, C., et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pp. 10435–10444, 2018.
- Shvachko et al. (2010) Shvachko, K., Kuang, H., Radia, S., and Chansler, R. The hadoop distributed file system. In Mass storage systems and technologies (MSST), 2010 IEEE 26th symposium on, pp. 1–10. IEEE, 2010.
- Silver et al. (2014) Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In ICML, 2014.
- Simonyan & Zisserman (2014) Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
- Smith et al. (2017) Smith, S. L., Kindermans, P.-J., and Le, Q. V. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.
- Smola & Narayanamurthy (2010) Smola, A. and Narayanamurthy, S. An architecture for parallel topic models. Proceedings of the VLDB Endowment, 3(1-2):703–710, 2010.
- Sparks et al. (2013) Sparks, E. R., Talwalkar, A., Smith, V., Kottalam, J., Pan, X., Gonzalez, J., Franklin, M. J., Jordan, M. I., and Kraska, T. MLI: An API for distributed machine learning. In Data Mining (ICDM), 2013 IEEE 13th International Conference on, pp. 1187–1192. IEEE, 2013.
- Sutskever et al. (2013) Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139–1147, 2013.
- Szegedy et al. (2015) Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9, 2015.
- Tassa et al. (2018) Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
- Vezhnevets et al. (2017) Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., and Kavukcuoglu, K. Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv:1703.01161, 2017.
- You et al. (2018) You, Y., Zhang, Z., Hsieh, C.-J., Demmel, J., and Keutzer, K. ImageNet training in minutes. In Proceedings of the 47th International Conference on Parallel Processing, pp. 1. ACM, 2018.
- Zaharia et al. (2012) Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., Mccauley, M., Franklin, M., Shenker, S., and Stoica, I. Fast and interactive analytics over hadoop data with Spark. Usenix Login, 37(4):45–51, 2012.
- Zhang et al. (2018) Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A. Self-attention generative adversarial networks. arXiv preprint arXiv:1805.08318, 2018.