Optimizing Quantum Error Correction Codes with Reinforcement Learning

  • 2019-08-08 12:33:10
  • Hendrik Poulsen Nautrup, Nicolas Delfosse, Vedran Dunjko, Hans J. Briegel, Nicolai Friis
  • 0


Quantum error correction is widely thought to be the key to fault-tolerantquantum computation. However, determining the most suited encoding for unknownerror channels or specific laboratory setups is highly challenging. Here, wepresent a reinforcement learning framework for optimizing and fault-tolerantlyadapting quantum error correction codes. We consider a reinforcement learningagent tasked with modifying a quantum memory until a desired logical error rateis reached. Using efficient simulations of a surface code quantum memory withabout 70 data qubits with arbitrary connectivity, we demonstrate that such areinforcement learning agent can determine near-optimal solutions, in terms ofthe number of data qubits, for various error models of interest. Moreover, weshow that agents trained on one setting are able to successfully transfer theirexperience to different settings. This ability for transfer learning showcasesthe inherent strengths of reinforcement learning and the applicability of ourapproach for optimization from off-line simulations to on-line laboratorysettings.


Quick Read (beta)

Optimizing Quantum Error Correction Codes with Reinforcement Learning

Hendrik Poulsen Nautrup [email protected] Institute for Theoretical Physics, University of Innsbruck, Technikerstr. 21a, A-6020 Innsbruck, Austria    Nicolas Delfosse Station Q Quantum Architectures and Computation Group, Microsoft Research, Redmond, WA 98052, USA    Vedran Dunjko LIACS, Leiden University, Niels Bohrweg 1, 2333 CA Leiden, The Netherlands    Hans J. Briegel Institute for Theoretical Physics, University of Innsbruck, Technikerstr. 21a, A-6020 Innsbruck, Austria Department of Philosophy, University of Konstanz, Konstanz 78457, Germany    Nicolai Friis Institute for Quantum Optics and Quantum Information, Austrian Academy of Sciences, Boltzmanngasse 3, 1090 Vienna, Austria Institute for Theoretical Physics, University of Innsbruck, Technikerstr. 21a, A-6020 Innsbruck, Austria
August 07, 2019

Quantum error correction is widely thought to be the key to fault-tolerant quantum computation. However, determining the most suited encoding for unknown error channels or specific laboratory setups is highly challenging. Here, we present a reinforcement learning framework for optimizing and fault-tolerantly adapting quantum error correction codes. We consider a reinforcement learning agent tasked with modifying a quantum memory until a desired logical error rate is reached. Using efficient simulations of a surface code quantum memory with about 70 data qubits with arbitrary connectivity, we demonstrate that such a reinforcement learning agent can determine near-optimal solutions, in terms of the number of data qubits, for various error models of interest. Moreover, we show that agents trained on one setting are able to successfully transfer their experience to different settings. This ability for transfer learning showcases the inherent strengths of reinforcement learning and the applicability of our approach for optimization from off-line simulations to on-line laboratory settings.


[innerlinewidth=0.5pt, roundcorner=4pt,linecolor=mycolor,innerleftmargin=6pt, innerrightmargin=6pt,innertopmargin=6pt,innerbottommargin=6pt]mybox \tcbsetcolback=mycolor!5,colframe=mycolor, fonttitle=, float=htb \newtcolorboxsmallbox[3][] float=ht!,lower separated=false, blend before title=colon hang, title=#2, label= fig:#3 ,#1

1 Introduction

Quantum computers hold the promise to provide advantages over their classical counterparts for certain classes of problems [56, 30, 20]. Yet, such advantages may be fragile and can quickly disappear in the presence of noise, losses, and decoherence. Provided the noise is below a certain threshold, these difficulties can in principle be overcome by means of fault-tolerant quantum computation [56, 60]. There, the approach to protect quantum information from detrimental effects is to encode each logical qubit into a number of data qubits. This is done in such a way that physical-level errors can be detected, and corrected, without affecting the logical level, provided they are sufficiently infrequent [36]. Quantum error correction (QEC) codes thus allow for devices —usually referred to as quantum memories [68] —that can potentially store quantum information for arbitrarily long times if sufficiently many physical qubits are available. However, physical qubits will be a scarce resource in near-term quantum devices. It is hence desirable to make use of QEC codes that are resource-efficient given a targeted logical error rate. Yet, while some types of errors can be straightforwardly identified and corrected, determining the most suitable QEC strategy for arbitrary noise is a complicated optimization problem. Nevertheless, solutions to this complex problem may offer significant advantages, not only in terms of resource efficiency but also error thresholds [73, 35].

Here, we consider a scenario where certain QEC codes can be implemented on a quantum memory that is subject to arbitrary noise. Given the capacity to estimate the logical error rate, our objective is to provide an automated scheme that determines the most economical code that achieves a rate below a desired threshold. A key contributing factor to the complexity of this task is the diversity of the encountered environmental noise and the corresponding error models. That is, noise may not be independent and identically distributed, may be highly correlated or even utterly unknown in specific realistic settings [52, 63]. Besides possible correlated errors, the error model might change over time, or some qubits in the architecture might be more prone to errors than others. Moreover, even for a given noise model, the optimal choice of QEC code still depends on many parameters such as the minimum distance, the targeted block error rate, or the computational cost of the decoder [56]. Determining these parameters requires considerable computational resources, e.g., the computation of the minimum distance of a linear code is NP-hard [74]. At the same time, nascent quantum computing devices are extremely sensitive to noise while having only very few qubits available to correct errors.

For the problem of finding optimized QEC strategies for near-term quantum devices, adaptive machine learning [29] approaches may succeed where brute force searches fail. In fact, machine learning has already been applied to a wide range of decoding problems in QEC [72, 10, 21, 67, 9, 54, 46, 44, 6]. Efficient decoding is of central interest in any fault-tolerant scheme. However, only a limited improvement can be obtained by optimizing the decoding procedure alone since decoders are fundamentally limited by the underlying code structure. Moreover, a number of efficient decoders already exist for topological codes [23, 24, 25]. This paper deals with an entirely different question: Instead of considering modifications of an algorithm run on a classical computer to support a quantum computation, we ask how the structure of the quantum mechanical system itself can be changed, both a priori and during a computation. More specifically, we present a reinforcement learning (RL) [66] framework for adapting and optimizing QEC codes. Through an adaptive code selection procedure, our approach enables tailoring QEC codes to realistic noise models for state-of-the-art quantum devices and beyond. The proposed scheme can be employed both for off-line simulations with specified noise models and for on-line optimization for arbitrary, unknown noise, provided that potential hardware restrictions are taken into account. In particular, we demonstrate how the presented learning algorithm trained on a simulated environment is able to transfer its experience to different, physical setups. This transfer learning skill is both remarkable and extremely useful: Maintaining the option to switch from one scenario to another, we can train a learning algorithm on fast simulations with modeled environments before optimizing the QEC code under real conditions. Transfer learning thus offers the ability to bootstrap the optimization of the actual quantum device via simulations. These simulations are comparably cheap since resources are much more limited under laboratory conditions. We show that our scheme can easily handle simulations with up to 68 qubits and is hence sufficiently efficient for setups expected to be available in the near-future [34, 77, 14]. In addition, these methods are parallelizable and hence expected to perform well also for larger system sizes. Our results thus suggest that RL is an effective tool for adaptively optimizing QEC codes.

2 Framework & Overview

We consider a situation where a learning algorithm —referred to as an ‘agent’ —interacts with an ‘environment’ by modifying a given QEC code based on feedback from the environment. As illustrated in Fig. 1, this environment consists of a topological quantum memory [68] (subject to noise) and its classical control system guiding the QEC. In each round of interaction, the agent receives perceptual input (‘percepts’) from the environment, that is, some information about the current code structure is provided. The agent is tasked with modifying the code to achieve a logical error rate below a desired threshold. To this end, the agent is able to perform certain actions in the form of fault-tolerant local deformations [15, 17] of the code. The agent is rewarded if the logical qubits have been successfully protected, i.e., if the logical error rate drops below the specified target. The problem of adapting QEC codes thus naturally fits within the structure of RL. Here it is important to note that the agent optimizing the quantum memory is oblivious to the details of the environment. In particular, the agent cannot discern whether the environment is an actual experiment or a simulation.

Figure 1: Adapting quantum memories via reinforcement learning: In the RL paradigm, an agent interacts with an environment through actions (red) and percepts (blue). The environment consists of a surface code quantum memory subject to noise, and a classical control system that guides the quantum error correction (QEC) on the memory and estimates logical error rates. The topological quantum memory embedded on a torus can be adapted through code deformations as instructed by the agent. In turn, the agent receives perceptual input in form of the current code structure and a reward which is issued by the classical control if the logical error rate is estimated to be below a desired threshold.

The quantum memory we consider is based on the surface code [27, 17], one of the most promising candidates for practical QEC [32]. In particular, this versatile code is easily adaptable to compensate for a wide range of different types of errors [35]. Accordingly, we consider independent and identically distributed (i.i.d.) as well as non-i.i.d. errors on data qubits. We assume that a logical qubit is initially encoded in an 18-qubit surface code that can be extended by the agent by adding up to 50 additional data qubits via fault-tolerant local deformations [17]. Our simulations assume arbitrary qubit connectivity which allows the most flexibility to illustrate our approach, but the method can be adapted to different topologies that one may encounter in a specific hardware, as we discuss in Sec. 3.3. At the heart of the classical control system is the SQUAB algorithm [23, 24] simulating an optimal decoder in linear-time. SQUAB returns an estimation of the logical error rate in the (simulated) QEC procedure. As a specific learning model we employ Projective Simulation (PS) [19], which has been shown to perform well in standard RL problems [47, 49, 48], in advanced robotics applications [38], and recently PS has been used to design new quantum experiments [50].


Summary of main resultsfidel Reinforcement Learning framework

  • for optimizing and adapting QEC codes, using

  • arbitrary topological QEC codes,

  • arbitrary decoders & noise models,

  • adaptable to any optimizer (RL paradigm)

  • implementable on arbitrary platforms,

  • applicable off-line (simulation) & in-situ

Simulations using

  • surface code quantum memory

  • up to 68 fully connected data qubits,

  • optimal linear-time decoder (SQUAB)

  • Projective Simulation model for RL

Simulations demonstrate agent’s ability

  • to determine optimal QEC codes

  • for simple standard noise channels

  • as well as non-isotropic noise, and for

  • transfer learning in changing environments

Within this framework, we demonstrate that agents can learn to adapt the QEC code based on limited resources to achieve logical error rates below a desired value for a variety of different error models. Moreover, we show that agents trained in such a way perform well also for modified requirements (e.g., lower thresholds) or changing noise models, and are thus able to transfer their experience to different circumstances [69, 75]. In particular, we find that beneficial strategies learned by agents in a simulated environment with a simplified error model that allows for fast simulation of QEC [26] are also successful for more realistic noise models. This showcases that agents trained on simulations off-line could be employed to ‘jump start’ the optimization of physical quantum memories on-line in real-time. One reason for this adaptivity is that the agent obtains no information about the specifics of the noise, i.e., the latter appears to the agent as a black box environment. This further implies that our approach is hardware-agnostic, i.e., it can be applied to the same problem on different physical platforms, including trapped ions [63], superconducting qubits [11], or topological qubits [40]. Depending on the particular platform, on-line optimization may be limited by specific hardware constraints (e.g., nearest-neighbour interactions), restricting, e.g., the possible code deformations (see Sec. 3.3). Nevertheless, off-line optimization still remains a viable option even for architectures with severe restrictions. Finally, note that the presented adaptive QEC framework in Fig. 1 goes beyond the particular code, decoder, noise model, and RL method used here, and hence offers potential for extensions in all of these regards.

The remainder of this paper is structured as follows. In Sec. 3, we briefly review the surface code in the stabilizer formalism [36] with emphasis on its inherent adaptability. Sec. 4 introduces the concept of RL and the specific algorithm used. In Sec. 5 various different scenarios within our framework are investigated. In Sec. 6, we discuss the efficiency of our simulations and analyze the prospect of using results obtained in a simulated environment for more realistic, experimental settings. We conclude with a summary and outlook in Sec. 7.

3 Adaptable Quantum Memory

3.1 Surface Code Quantum Memory

The purpose of a quantum memory is to encode logical information in such a way that small enough perturbations do not change the stored information. This is achieved through QEC, which can, in parts, be understood within the stabilizer formalism [36]. Although a detailed background in QEC is not necessary to understand the basic principle of our approach, which can be viewed as the problem of restructuring a lattice (see Sec. 3.2), let us briefly illustrate the stabilizer formalism for the example of a surface code quantum memory [17].

Figure 2: Standard surface code on a 3×3 square lattice embedded on a torus. Larger, blue dots represent data qubits, smaller black and grey dots represent syndrome qubits. Lines connecting blue dots with vertices or plaquettes represent Z- and X-type stabilizers respectively. The black dotted line represents the dual lattice. The thick green line depicts a possible path for a ZL string and the thick red line a path for a XL string. This is also the initial code considered in Sec. 5.

The surface code can be defined on any lattice embedded on a 2D manifold where data qubits are located on lattice edges (see Fig. 2). These codes are called surface codes because each code in the class can be associated with a tessellation of a 2D manifold or surface [27]. For brevity, we consider lattices embedded on tori which define a subclass of surface codes, called toric codes [27]. However, the methods introduced in this paper can be easily extended to other surfaces or lattices with boundaries [32], and we hence just refer to surface codes in the following. Now, let us define products of Pauli operators for each vertex v and plaquette p as follows,

Av=i𝒩(v)Xi, (1)
Bp=i𝒩(p)Zi. (2)

Here, 𝒩(x) is the set of edges adjacent to a vertex x=v or plaquette x=p and Xi, Zi are Pauli-operators acting on the ith qubit (indices enumerate data qubits located on the edges of the lattice). We define the surface code’s stabilizer group 𝒮 as the group generated by all Av and Bp under multiplication. This group is Abelian, since all generating operators commute and have common eigenvectors. The +1 eigenspace defines the codespace 𝒞 of the surface code. That is, a vector |ψ𝒞 iff S|ψ=|ψ S𝒮. Elements of 𝒮 are called stabilizers. Measuring the observables associated to these stabilizers allows checking whether the system state is still within the codespace. For a surface code defined on a torus, the code space is isomorphic to (2)2. It encodes 2 logical qubits into n data qubits where n is the number of edges.

To each logical qubit one further associates a set of logical Pauli operators which define the centralizer of 𝒮, i.e., operators that commute with 𝒮. Trivial logical operators are stabilizers since they act as the identity on the logical subspace. In the surface code, nontrivial logical operators are strings of tensor products of Pauli operators along topologically nontrivial cycles, i.e., noncontractible paths, of the torus. Each logical qubit is defined by a logical Z-operator along a noncontractible path of the lattice and a logical X running along a topologically distinct noncontractible path of the dual lattice (see Fig. 2). Since such paths necessarily cross an odd number of times, they anticommute.

Logical operators change the encoded information. Thus, it is desirable to detect and correct errors before they accumulate to realize a logical operation. Errors which are neither stabilizers nor logical operators can be detected by stabilizer measurements since they anticommute with some stabilizers. In order to perform stabilizer measurements, one first associates so-called syndrome qubits with vertices and faces. Then, the syndromes, i.e., the eigenvalues of stabilizers, are obtained by entangling syndrome qubits with adjacent data qubits and subsequent measurements [32]. In a fixed architecture the required entangling gates between syndrome and data qubits may be represented by the edges of the lattice (for X-stabilizers) and its dual (for Z-stabilizers).

The length of the shortest path of any noncontractible loop on the torus is the distance d of the surface code. That is, the distance is the minimal number of physical errors that need to occur in order to realize a logical operator. Any number of errors below d can be detected but not necessarily corrected and less than d/2 errors can always be corrected in the stabilizer formalism [36].

3.2 Adaptable Surface Code

So far, we have assumed the standard notion of a surface code defined on a square lattice on a torus (see Fig. 2). However, surface codes can be defined on arbitrary lattices. Since the surface code stabilizers in Eqs. (1) and (2) are identified with vertices and plaquettes of the lattice, adapting the lattice can change the underlying code and its performance dramatically [35]. This feature makes surface codes particularly useful for biased noise models, e.g., noise models where X- and Z-errors occur with different probability. Even the loss tolerance is affected by the underlying lattice.

In our framework, we will be exploring the space of lattices to make use of the adaptability of the surface code. To search this space in a structured fashion, we introduce a set of elementary lattice modifications.

Specifically, we consider modifications that can be performed with a set of basic moves which were first illustrated in Ref. [17]. The two basic moves are exemplified in Fig. 3: (i) As illustrated in Fig. 3(a), addressing the primal lattice, two non-neighbouring vertices can be connected across a plaquette which is effectively split by the additional edge. That is, the number of Z-stabilizers is increased while an additional data qubit is added. (ii) Conversely, the same operation can be performed on the dual lattice producing an additional vertex (i.e., X-stabilizer) as illustrated in Fig. 3(b). Note that the reverse operation, i.e., deleting an edge by merging two vertices or two faces, respectively, is also possible. While the inclusion of qubit deletion greatly increases the number of possible moves (actions) in each step, the set of reachable codes is the same as if one had started from a single data qubit using only the operations in Fig. 3. Here, we have chosen not to explicitly include such deletions to simplify the simulations. All of these moves can be implemented fault-tolerantly 11 1 Here, fault-tolerance requires changes to be local w.r.t. the lattice topology. More specifically, the circuit that implements the changes has to act on a small number of data qubits. In this way, errors can spread only to a limited number of qubits even if each gate in the circuit fails. and map a surface code to a surface code while changing the underlying stabilizer group [17]. This can be considered as simple version of code deformation [15] which is employed here to explore the space of surface codes. Indeed, code deformation can in principle be employed in a much more general way to map topological stabilizer codes to other topological stabilizer codes. However, for our needs the two basic moves explained above already allow for sufficiently diverse modifications. In particular, the number of available moves increases with the lattice size.

Figure 3: An illustration of the basic moves that fault-tolerantly map surface codes to surface codes while changing the underlying lattice and hence, the stabilizer group. (a) Two vertices (here chosen at the top and bottom in red) are connected across a plaquette (central black dot) which is effectively split into two by a new edge. (b) Two plaquettes (red dots on top left and bottom right) are connected across a vertex (central grey dot) which is effectively split into two by a new edge. In both cases the number of data qubits (blue dots) is increased by one (central red) qubit. Note that move (b) can be understood as a move of type (a) on the dual lattice and vice versa.

3.3 Hardware constraints

Before we further go into detail on our numerical study focussing on setups with the full flexibility required to implement the moves in Fig. 3, let us briefly consider the implications of the variations in hardware of currently available setups. For instance, limitations on how freely qubits that are physically present but not yet part of a specific code structure can be connected to the latter vary between different platforms such as ion trap architectures [63, 4, 16] or superconducting qubit devices [11]. While our method can be used directly when all-to-all connectivity between qubits is given, e.g., in many current ion traps [63, 4, 16], modifications can be made to the set of allowed code deformations when this is not the case, provided that some minimal requirements are met. That is, for any fixed architecture to be able to compensate for biased and potentially correlated noise it is required that (i) the connectivity of the lattice can be changed, (ii) data qubits can be removed and (iii) syndrome qubits can be removed. While changing the connectivity of the lattice enables one to adapt to biased error channels in accordance with Ref. [35], removing qubits allows the adaptability to correlated noise sources arising, e.g., from manufacturing defects. In very restricted scenarios, for example, when only nearest neighbour interactions are possible, these requirements are not met. In such cases, one may still perform code optimization off-line using an estimated noise model in order to produce a blueprint for manufacturing or initialization. However, already a 2-nearest neighbour architecture (featuring connections between syndrome qubits and next nearest data qubits) provides the required features, as can be seen from Fig. 4.

At this point, it should also be mentioned that the flexibility provided by higher connectivity in terms of code adaptability is not the only relevant factor: Higher connectivity might be subject to a trade-off with lower fidelities due to higher technical demands. At the same time, allowing for flexible code deformations and adaptive on-line error correction is certainly not the only motivation for non-nearest neighbour connections. In particular, nearest neighbour architectures may suffer from limitations on fault-tolerant implementations of logical gates [18, 57]. Indeed, the first successful experiments with a non-nearest neighbour architecture of superconducting qubits have been successfully performed [61], suggesting that (variations of) our methods are applicable across a range of platforms currently in use or under development.

Figure 4: A selection of possible lattice configurations for a 2-nearest neighbour surface code architecture with boundaries illustrating the available flexibility. Blue dots represent data qubits, black and grey dots represent syndrome qubits. Connecting lines visualize potential vertex and plaquette stabilizers. (a) A 2-nearest neighbour surface code on a square lattice with additional connections shown. For each vertex and plaquette the possible connections (for syndrome readouts) extend within a square of size two as displayed for one vertex and one plaquette. (b) Exemplary lattice with changed connectivity. (c) Lattice configuration where two data qubits (red dots) and two syndrome qubits (white dots) have been removed. (d) In a 2-nearest neighbour setup a syndrome qubit and its connections can be replaced by a neighbouring syndrome qubit. Dotted lines represent previous connections to removed qubits that are no longer used.

4 Reinforcement Learning and Projective Simulation

In generic RL settings [66], one considers an agent that is interacting with an environment (see Fig. 1). At each time step the agent obtains information from the environment in terms of perceptual input – percepts – and responds through certain actions. Specific behavior, i.e., sequences and combinations of percepts and actions, trigger reinforcement signals – rewards. Typically, the agent adapts its behavior to maximize the received reward per time step over the course of many rounds of interactions. The RL method as we use it makes no assumptions about the environment and applies even if the environment is a black box. In our case, the environment is a surface code memory subject to noise and its classical control system, including the means to estimate logical error rates. Here, knowledge of the environment is also restricted: The agent is never directly provided with information about the noise model. This is because, in practice, acquiring knowledge about the noise requires involved process tomography. Since the agent does not aim to characterize the noise process, but rather to alleviate its effects, a complicated noise estimation procedure is not necessary.

The specific learning agent that is considered in this paper is based on the Projective Simulation (PS) model for RL. PS is a physics-motivated framework for artificial intelligence developed in Ref. [19]. The core component of a PS agent is its clip network which is comprised of units of episodic memory called clips (see Fig. 5). There are two sets of clips constituting the basic network, percept and action clips. In an interactive RL scenario, an agent perceives the environment through the activation of a percept clip siP and responds with an action. The latter, in turn, is triggered by an action clip ajA. Percept clips can be regarded as representations of the possible states of the environment, as perceived by the agent. Similarly, action clips can be seen as internal representations of operations an agent can perform on the environment. A two-layered clip network as in Fig. 5 can be represented by a directed, bipartite graph where the two disjoint sets comprise the percepts P and actions A, respectively. In this network each percept (clip) siP, i[1,N(t)] (where N(t) is the number of percepts at time step t) is connected to an action (clip) ajA, j[1,Mi] (with Mi being the number of actions available for a percept si) via a directed edge (i,j) which represents the possibility of taking action aj given the situation si with probability pij:=p(aj|si). The agent’s policy governing its behavior in this RL setting is defined by the transition probabilities in the episodic memory. Learning is manifested through the adaption of the clip network via the creation of new clips and the update of transition probabilities. Each time a new percept is encountered, it is included into the set P. A time-dependent weight, called h-value, hij(t) is associated with each edge (i,j). The transition probability from percept si to action aj is given by the so-called softmax function [66] of the weight σβ(hij(t)),

pij(t)=eβhij(t)k=1Mieβhik(t), (3)

where β>0 is the softmax parameter.

When a percept is triggered for the first time all h-values are set to 1 such that the transition probabilities are initially uniform. That is, the agent’s behavior is fully random in the beginning. Since random behavior is rarely optimal, changes in the transition probabilities are required. The environment reinforces such changes by issuing nonnegative rewards λ(t) in response to an action of the agent. Then, the agent must ask the question, given a percept si which is the action aj that will maximize the received reward. Therefore, the transition probabilities are updated in accordance to the environment’s feedback such that the chance of the agent to receive a reward in the future is increased. In other words, the environment’s feedback λ(t) controls the update of the h-matrix with entries hij. However, there are many other contributions to the update independent of the environment’s immediate feedback. Particularly noteworthy are contributions that reinforce exploratory over exploitative behavior. A detailed description of how the feedback is processed in the agent’s memory is given in Appendix A.

Once an agent has learned to optimize the reward per interaction step, the clip network in Fig. 5 is a

Figure 5: Two-layered clip network of a PS agent. Percept clips in the first (upper) layer are colored blue and represent surface codes. Action clips are colored red and represent code deformations. Edges (i,j) carry weights that can be associated with transition probabilities pij in accordance with Eq. (3). If an action, say a3, is performed upon seeing a percept, say s2, the code associated with s2 is adapted according to a3. This results in a new code, say s4, that is perceived by the agent. Note that not every percept is necessarily connected to every action since the set of available moves in Fig. 3 depends on the lattice, i.e. the percept.

representation of the agent’s policy: Nonuniform transition probabilities pij mark a systematic decision-making aimed at maximizing the reward per time step while accounting for exploratory behavior. Hence, the network can also be interpreted as an agent’s memory encoding knowledge of the reward landscape of the environment.

5 Optimizing Quantum Memories - a reinforcement learning problem

In this section, we show that RL can be employed within the framework of Fig. 1 to successfully optimize QEC codes w.r.t. their resource efficiency in settings with up to 68 qubits. In our framework for adaptive quantum error correction, we consider a PS agent that interacts with a specific, simulated environment – the surface code quantum memory subject to noise and its classical control managing the QEC procedure. We use the term environment in the sense it is used in RL rather than referring to the source of noise as is common in, say, open-system quantum physics.

To be more precise, we start each trial by initializing a distance-three surface code quantum memory with 18 qubits (edges) defined on a square lattice Λ embedded on a torus (see Fig. 2). Note that the choice of initial code is ad hoc and could be replaced by any other small-scale surface code designed to suit a given architecture. The code is subject to an unknown Pauli noise channel which may change in time and may differ for different data qubits. The classical control simulates the QEC procedure under this noise model and estimates the logical error rate PL. Having a basic set of moves at its disposal (see Fig. 3), the agent is tasked with growing the lattice until the logical error rate is below a target PLrew or at most 50 additional qubits have been added. This choice of an upper limit should be viewed in light of recent progress in the development of quantum devices with more than 50 qubits [77, 14]. Note the difficulty of the simulation task. A single trial requires to simulate the perfomance of up to 50 QEC codes. The basic set of moves is rather generic and could be restricted further to suite the requirements of a given hardware. For instance, actions could be restricted to the boundary of a system. Once the desired logical error rate is reached, the agent receives a reward λ=1, the trial ends and the quantum memory is reset to Λ. If the desired logical error rate is not reached before 50 qubits have been introduced, the code is reset without reward. The details of the algorithm, specifically the environment, are presented in Appendix B and the agent’s clip network, i.e., episodic memory, is visualized in Fig. 5. Note that this scenario – although restricted to surface codes – is already extremely complex and versatile. In Appendix C, we analyze the difficulty of this problem and show that there is no hope of solving it by random search.

Figure 6: The figure shows the results for two simple tasks as considered in Sec. 5.1 which confirm that RL learning can be applied to the problem of optimizing quantum memories. The area shaded in light blue is the standard deviation over 60 agents. Given the available set of actions, shown in Fig. 3, the PS agent can add, one-by-one, qubits to the initial code (see Fig. 2) until the logical error rate is below PLrew=0.001. (a) The memory is subjected to the noise channel in Eq. (4) with p=0.1. The initial logical error rate is PLinit0.006. The number of qubits to reach the desired error rate PLrew per trial is depicted. (b) The same task is shown with the error channel of Eq. (5) such that PLinit0.005. Different learning parameters (see Appendix A) are reflected in different learning times, standard deviations and convergence behavior. Generally, there is a trade off between learning time and standard deviation as can be seen from the comparison between (a) and (b). Shaded areas represent the standard deviation. Details about parameters are given in Appendix D.

In the following, we verify that the task outlined above is indeed an RL problem where learning is beneficial. To this end, we start by considering some instructive cases where the solutions are known before moving on to a more complicated scenario with unknown optimal strategy. The variables that are being changed in between scenarios are the error channel or the rewarded logical error rate PLrew. However, if not stated otherwise, we set PLrew=0.001. For illustrative reasons, the error models that appear in the following are represented by single-qubit Pauli error channels. In fact, the simulated quantum memory will be subject to a so-called erasure channel [13, 37] which models the behavior of the actual Pauli channel. This particular choice of error model is motivated by the availability of an optimal decoder for erasure noise on a surface code [26]. In Sec. 6.1, this simplified channel is analyzed in more detail and we verify that it is suitable to model actual error channels.

5.1 QEC codes for i.i.d. error channels

As a first simple task, we consider two straightforward scenarios with known good solutions as they will nonetheless be crucial in the evaluation of the behavior of the RL agent. First, let us consider an error channel that can only cause Z-errors with probability p=0.1 on each individual, data qubit independently. Formally, we can write this as a quantum channel acting on each single-qubit state ρ as,

(ρ)=pZρZ+(1-p)ρ. (4)

Assuming the initial logical error rate is above the target value PLrew, the RL agent is then tasked with modifying the 18-qubit surface code by increasing the number of data qubits according to the rules described in Fig. 3. At the beginning of each trial, an agent has a reservoir of 50 additional qubits at its disposal to reduce the logical error rate below PLrew=0.001. Fig. 6 shows that the agent indeed learns a strategy, i.e., a policy, which adds – on average – 5 data qubits to the initial code in order to reduce the logical error rate as desired. Fig. 6 can be understood as follows. In the very beginning, the agent has no knowledge about the environment and performs random modifications of the surface code. For the specific error channel in Eq. (4), a random strategy requires on average 20 additional data qubits to reach the desired logical error rate PLrew. What follows are many trials of exploring the space of surface codes. During each trial, the agent’s episodic memory is reshaped and modified as described in Sec. 4 and Appendix A. This learning process effectively increases the probability for the agent to select sequences of actions which lead to a reward quickly. As can be seen from Fig. 6 the length of a rewarded sequence of actions is gradually reduced with each trial. Ideally, the agent’s behavior converges to a policy that requires a minimum number of additional qubits. In Fig. 6 we observe that agents converge towards a policy which requires, on average, 5 additional qubits. Let us now confirm that the strategy found by the RL agents agrees with the best known strategies for this problem. The error channel in Eq. (4) can only cause logical Z-errors. That is, we are looking for the surface code with the maximum number of X-stabilizers and minimum number of Z-stabilizers [35]. Starting from the surface code in Fig. 2 we must therefore ask the question how to increase the number of X-stabilizers given the available actions in Fig. 3. Since X-stabilizers are identified with vertices in the graph, repeatedly applying an action as displayed in Fig. 3 will increase the number of X-stabilizers. We hence expect a sequence of these actions to provide good strategies in this case, resulting in a surface code on a lattice with low connectivity. Indeed, we can confirm this by looking at surface codes constructed by agents that have been successfully applied to this task. In Fig. 7, a particularly interesting example solution is shown: It can be seen that the distance of the code w.r.t. Z-errors along one cycle has been increased from 3 to 4 by inserting 4 new qubits at very specific points in the lattice. In other words, any logical Z-operator crossing the lattice from left to right in Fig. 7 is a product of at least 4 single-qubit Z-operators. Just looking at the initial lattice in Fig. 2 it is not obvious that this can be done with only 4 actions. This is already a nontrivial result for this, at first glance, simple problem.

Figure 7: The best agents only require 4 additional qubits to solve the task described in Sec. 5.1 with the error channel in Eq. (4). Here, an exemplary solution is shown, i.e., a surface code with a logical error rate below PLrew=0.001. Red circles represent qubits added to the code. Removing all red qubits, we recover the initial code (see Fig. 2). As expected, all actions increase the number of X-stabilizers in order to protect against Z-errors. The thick green line depicts a possible path for a ZL string crossing 4 qubits.

Now, let us consider the well-understood scenario of a i.i.d. error channel. That is, each qubit in the quantum computer is subject to a quantum channel of the form

(ρ)=pXXρX+pZZρZ+(1-pX-pZ)ρ, (5)

where we choose pX=pZ=0.09 such that either X- or Z-errors can happen with probability 0.09 everywhere. Otherwise the task remains the same as before. We can see from Fig. 6 that the agent again learns to improve the logical error rate while optimizing over the number of required qubits.

The optimal surface code to protect against depolarizing noise is defined on a square (self-dual) lattice [35]. That is, the best known strategy to grow the surface code using the actions available (see Fig. 3) is to retain the same connectivity both on the primal and dual lattice such that there are the same number of X- and Z-stabilizers. Indeed, looking at some examples from agents trained on this task, we can confirm that the agents also learn this strategy: The most successful agents end up creating surface codes where the number of X- and Z-stabilizers differ by at most 1. On average, the ratio between number of X- and Z-stabilizers is 1.06 with standard deviation of 0.15. The corresponding primal and dual lattices tend to have similar average connectivities, too. The average ratio between primal and dual connectivity is 1.06 with standard deviation of 0.15.

5.2 QEC codes for correlated error channels

Next, let us tackle a complicated, practical scenario where the optimal strategy is unknown: We consider spatial dependencies on top of an i.i.d. error channel. This particular situation is motivated by the common problem of manufacturing defects. Correlated errors arising e.g., from manufacturing defects can be present in any fault-tolerant architecture because active error correction requires multi-qubit operations such as stabilizer measurements. Most architectures using topological codes have a spatial layout which can be associated with the lattice of the surface code [11, 4] such that correlated errors will most likely be local. Consider for instance an ion trap quantum computer using an arrangement of multi-zone Paul trap arrays [16]. Dephasing is the prevalent noise in ion traps [63], so we would already have to consider an asymmetric error channel. Moreover, due to e.g., a manufacturing defect, two Paul traps in this arrangement could fail and produce X-errors on internal qubits.

To be precise, consider the error channel in Eq. (5), with pX=0.02 and pZ=0.1. This is similar to the simplest case in Sec. 5.1 where pX=0. In our construction, a correlated error as motivated above is modeled by an increased X-error rate on edges in the neighbourhood 𝒩(i) of a vertex or plaquette i. Here, we choose two neighbouring plaquettes i,j and modify the error channel as follows,

k(ρ)=pX,kXρX+pZ,kZρZ+(1-pX,k-pZ,k)ρ, (6)

where k labels the qubits. We further assume that all qubits have base error rates of pX,k=0.02 and pZ,k=0.1. In addition, the base probability pX,k of an X-error on qubit k is increased by 0.5 if k𝒩(i)𝒩(j) and pX,k=1 if k𝒩(i)𝒩(j). This serves two purposes. First, we can evaluate the behavior of the agent with regard to the two plaquettes i,j. How is the lattice structure changed in and around these plaquettes? Second, we can understand how the agent handles a deterministic error on the edge neighbouring both i,j. Will this edge be treated differently? In fact, it is far from clear what the optimal strategy is given the available actions displayed in Fig. 3. Nevertheless, we observe that the PS agent learns to protect the surface code against this error channel while optimizing the used resources, see Fig. 8.

Now, let us evaluate successful strategies employed by the best agents in this scenario. In Fig. 9 the relevant part of the final dual lattice as constructed by a successful agent is depicted. We can see that the agent increases the number of Z-stabilizers in the immediate vicinity of the flawed plaquettes, decreasing the connectivity of affected plaquettes. Interestingly, the agent also finds a way of moving the flawed plaquettes apart given the available actions, thereby removing any deterministic error appearing on the edge between these plaquettes. At the same time, due to the prevalent Z-errors, connectivity throughout the lattice is balanced between vertices and plaquettes: the average connectivity of the dual lattice is 3.9 and 4.1 for the primal lattice. Similarly, the ratio between X- and Z-stabilizer is 14/15.

Figure 8: Learning curve for the task specified in Sec. 5.2 averaged over 60 agents. The shaded area shows the standard deviation. Here the quantum memory is subject to a noise channel with spatial dependencies as in Eq. (6). The initial logical error rate is as high as PLinit0.28. Details about parameters are given in Appendix D.

5.3 QEC codes with changing requirements

So far, we have kept the rewarded logical error rate fixed. However, it is in general also desirable to be able to adapt to stricter target thresholds if required. We therefore consider a scenario where the error channel is initially fixed to that of Eq. (4) with p=0.1. Then, after having learned the optimal strategy from before, the agent is tasked with further decreasing the logical error rate to a quarter of the initial value. As one can observe from Fig. 10, the agent can indeed draw on its knowledge from the first stage of the task to find a solution for the second stage.

In the particular scenario of varying target thresholds, the performance of the agent can potentially be improved by issuing a reward which is proportional to the inverse logical error rate. This provides more feedback about the reward landscape in form of a gradient which can be followed. Such a modified reward landscape can also be exploited by simpler optimization algorithms such as Simulated Annealing [41]. However, our main focus here lies on exploring the observed ability to capitalize on previously obtained knowledge. As we will see in Sec. 6, this ability can be viewed as part of a more general transfer learning [69, 75] skill that represents a main strength of RL for QEC.

Figure 9: Part of the final dual lattice constructed by one of the most successful agents in the task described in Sec. 5.2 where noise has spatial dependencies (see Eq. (6)). Qubits that are affected by high X-error probabilities are colored in red and surround flawed plaquettes colored in orange.
Figure 10: Learning curve for the task specified in Sec. 5.3 averaged over 60 agents. The shaded area shows the standard deviation. In this task the agent is tasked to reduce the logical error rate below PLrew=0.001 while the code is subject to the error channel in Eq. (4) with p=0.1. After 6,000 trials the same agent is tasked to reduce the error rate even further below PLrew=0.00025. Details about parameters are given in Appendix D.

6 Simulation vs. Experiment

In the previous sections, the main focus has been to determine whether RL agents in our framework are able to adapt and optimize a surface code quantum memory to various types of noise with different requirements. We have found that this is indeed the case, showcasing the prospect of RL in on-line optimization of quantum memories. Although we have evaluated our procedures only via simulations, our results suggest that such approaches can be successful also in practice in future laboratory experiments. This is because our framework for optimizing QEC codes in Fig. 1 is independent of whether the environment is a real or simulated quantum device. In either case, the interface between environment and agent remains unchanged. For instance, we estimate the logical error rate of the quantum memory using a Monte Carlo simulation. In a real device, this can be done by preparing a quantum state, actively error correcting for a fixed number of cycles, and then measuring the logical operators by measuring all data qubits in a fixed basis. The logical error rate should then be interpreted as the probability of a logical error per QEC cycle. Repeating this measurement provides an estimation of the lifetime of the quantum memory. Moreover, the code deformations which constitute the actions available to an agent are designed with a physical device in mind [17, 15].

Figure 11: Comparison of logical error rates estimated by (a) SQUAB [26] (using an approximated Pauli error model) and (b) the Union Find decoder [25] (using a standard Pauli error model). Each node in the tree corresponds to an individual surface code on a torus and edges imply that two codes are connected by an action as in Fig. 3. The central node is the 2×2 surface code on a square lattice. We explore all codes up to a distance 3 from the root. The color shading indicates the logical error rate. (a) The logical error rate is estimated using SQUAB with 10,000 trials and erasure rates peZ=0.1, peX=0. (b) The logical error rate is estimated using the Union-Find decoder with 10,000 trials and a Z-error rate pZ=0.1. While the logical error rates differ in both cases, the behavior is qualitatively the same for a code subject to an asymmetric erasure channel and the same code affected by Pauli noise. That is, codes found to provide lower logical error rates w.r.t. other codes for one noise model also provide lower logical error rates than other codes for the other error channel.

Ultimately, one is of course interested in performing optimization on actual quantum devices. However, as long as these devices are sufficiently small-scale to allow for simulations that are (much) faster than the timescale of operating the actual device, it is advantageous to perform the bulk of optimizations off-line in simulations before transferring the results to the actual device for further optimizations. In other words, to fully capitalize on the important features of transfer-learning and pre-training in quantum-applied machine learning, it is crucial to ensure that the simulations are as efficient as possible, and that the chosen RL model is able to transfer experience from simulations to real environments. In the following, we therefore discuss the efficiency and practicality of our simulations and show that the RL agents we consider are capable of transfer learning.

6.1 Simulation Efficiency of QEC

To provide sufficiently efficient off-line simulation, the individual components of our QEC simulation have been carefully selected. For instance, note that stabilizer QEC can be simulated efficiently classically [22, 1, 28, 5]. However, estimating the logical error rate, which requires a large number of samples from QEC simulations, remains computationally expensive. In our simulations, we hence make use of a simplified error model to allow for faster estimations of logical error rates. The simplified error model is based on the quantum erasure channel [13] since there exists a linear-time maximum likelihood decoder for surface codes over this channel which is optimal both in performance and speed [26]. The use of this software, SQUAB [23, 24], within our RL framework is hence providing the means for sufficiently fast exploration of the space of surface codes. In essence, the erasure error channel is very similar to a Pauli error channel where the location of an error is known exactly. More specifically, we introduce an asymmetric erasure channel where we choose two erasure probabilities peX,peZ that can have spatial and temporal dependencies. Since X and Z stabilizers are separated in the surface code, error correction of X and Z errors can also be treated independently. In the simulation of X errors over the erasure channel, we erase each qubit ρ in the surface code with probability peX, and replace it by a mixed state of the form ρ=12(ρ+XρX). The set of erased qubits is known. The simulation proceeds analogously for Z errors.

Since our simulations cover setups with up to 68 qubits, we have indeed good reason to believe that our simulations are sufficiently efficient to optimize QEC codes for near-term quantum devices without the need of experimental data. However, one may argue that our software only provides a simulation for a simplified noise model, the erasure channel. In order to prove that the results are relevant in practice, we compare logical error rates of a set of small surface codes subject to erasure errors to the rates obtained from simulating standard Pauli noise on the same codes. To this end, we assume that each qubit is affected independently by a Z-type Pauli error and we perform error correction with both SQUAB, and the Union-Find decoder introduced in Ref. [25]. We report the average logical error rate for each code after 10,000 trials with an erasure rate peZ=0.1 (SQUAB) and Z-error rate pZ=0.1 (Union Find). In Fig. 11 we observe qualitatively the same behavior: codes that are considered to perform well according to the estimation using SQUAB are also considered to perform well using the Union-Find decoder. The difference between the two error channels lies predominantly in the magnitude of the logical error rates. It is therefore better to select codes using SQUAB, allowing for a faster and thus more precise exploration of the space of topological codes.

6.2 Transfer learning in QEC

Figure 12: This figure shows the results for two tasks as considered in Sec. 6.2 which explore the potential of transfer learning. The agent is tasked to optimize a QEC code w.r.t. its resource efficiency. Resource efficiency is reflected in the number of qubits added to the code before a desired logical error rate PLrew is reached. The shaded area shows the standard deviation for 60 agents. (a) At first, the code is subject to an error channel as in Eq. (4) with p=0.14 and PLrew=0.001. The initial logical error rate is PLinit0.019. At trial 4,000 the error probability is increased to p=0.16. The blue curve depicts the results in this scenario averaged over 60 agents. As comparison, the green line is the average over 40 agents that are directly tasked to protect against an error channel with p=0.16 without having been trained on the simpler noise model. These agents encounter an initial logical error rate of PLinit0.028. (b) A transfer learning scenario where 60 agents (blue curve) are trained on a simple, unrealistic error model [see Eq. (4)] before being subjected to either one of the more realistic noise channels ~1,~2 described in Eq. (7) and (8), respectively. For the second stage we have randomly selected one of the most successful agents from the first stage. As comparison, the green and pale rose curve depict an average over 60 agents with the same parameters, subjected to the channels ~1,~2, respectively, but without pre-training. Details about parameters are given in Appendix D.

The usefulness of off-line simulations for QEC code optimization emerges from the application of the results to on-line settings. In this transition, deviations of the error model used in the simulation from the actual noise, or dynamical changes of the latter can lead to non-optimal performance if no further action is taken. Here, machine learning comes into play. That is, a central agenda of machine learning is to develop learning models that are capable of successfully applying previously obtained knowledge in different environments. This possibility, called transfer learning [69], is an active area of research in general AI [75]. Here we should note that the ability to transfer knowledge is indeed much desired but not manifestly present in all machine learning models. At the same time, there is the risk of confusion with the more generally encountered ability for generalization [62, 49]. Let us quickly illustrate the difference: On the one hand, generalization is the ability of a learning agent to be effective across a range of inputs and knowing what to do in similar situations. Transfer learning, on the other hand, is the ability to attain knowledge in one scenario, and then being able to use this knowledge in a different (new but related) learning setting. One of the objectives of transfer learning is to jump start the learning process in a new, but similar scenario, and is fundamentally linked to the AI problem of learning how to learn [58, 69].

Transfer learning can provide a significant improvement in performance as compared to optimization approaches without learning mechanisms, as well as considerable reductions of learning times in comparison to untrained RL agents. Put simply, agents capable of transfer learning do not have to start from scratch every time the environment or task changes slightly [71, 70, 8, 50]. As we will discuss in this section, within the RL setting we consider here, agents trained on the original simulations may transfer their experience from simulations to practical applications. The usefulness of this approach will of course depend on how similar the simulated error model is to the real error channel.

In order to demonstrate the potential of transfer learning in the context of optimizing and adapting quantum memories, we consider a scenario where the agent is first trained on one error model , and then encounters a different error model ~. This change in noise could occur not only because the agents are switched from a simulated training scenario to an actual experiment but also, e.g., due to environmental changes, equipment malfunctions or malicious attacks. Generally, under the assumption that the noise can only vary slightly, we can expect learning to be beneficial since strategies that helped in one scenario should still be favorable in protecting against similar error channels. Then, an RL algorithm can further optimize over a given policy and re-adjust to the new environment. This scenario capitalizes on the strength of RL since it requires a long-term memory that can be exploited between tasks. If exploration of the search space is encouraged over exploitation of rewards, the agent’s memory contains global information about the environment. In other words, given some sub-optimal QEC code, such an agent knows a strategy to adapt this code efficiently such that the desired logical error rate is reached. In fact, the feedback which is given only if a certain logical error rate has been achieved, is specifically designed to encourage exploration of the search space.

The specific choice of error models for this scenario is partially motivated by the results in Fig. 11. There we observe that the main difference between the simulated erasure and more realistic Pauli error channels lies in the magnitude of the estimated logical error rate. We therefore now consider a quantum memory which is first subjected to the simple noise channel in Eq. (4) with p=0.14 and the agent is tasked again to reduce the logical error rate below PLrew=0.001. Then, after having learned a beneficial strategy for finding a good QEC code, the same agent is confronted with an increased error rate of p=0.16. Fig. 12 shows that, in this second stage of the task, the agent benefits from having learned to protect the quantum memory against the first error channel. In particular, we see from Fig. 12 that agents not initially trained on the first noise channel behave randomly and do not find reasonably good strategies to deal with the high noise level. This is because at a physical error rate of p=0.16 the initial code requires many modifications before the desired logical error rate can be reached. In fact, the required number of basic modifications is so large that a random search is just not sufficient to find a reasonably good code in the allotted time of 6,000 trials. Although the observed behaviour of the learning agents showcases the benefit of a memory, the error models are nevertheless too similar to reveal the potential for transfer learning.

Let us therefore consider another scenario with more drastic changes in the noise model. In particular, we attempt to model a setting more closely resembling the transfer of knowledge from simulation to experiment: That is, we start by training 60 agents on an error channel that captures partial knowledge of the real error channel disturbing a quantum system. Since dephasing is the prevalent noise in many quantum computation architectures [63, 3, 64], we choose the error model from Eq. (4) with p=0.1 for training. Here, we deliberately neglect generally more realistic Pauli errors to showcase the transfer learning ability, but we allow sufficiently many iterations so the agents can optimize their behavior w.r.t. to the unrealistic noise model. Then, in the second stage of this scenario, we select (at random) one of the best agents and confront it with a more realistic error model ~1 which features both X-, and Z-errors as well as spatial correlations. The error model, similar to that in Eq. (6), is

~1,k(ρ)=pX,kXρX+pZ,kZρZ+(1-pX,k-pZ,k)ρ, (7)

where k labels the qubit the channel acts on. All qubits have base error rates of pX,k=0.02 and pZ,k=0.14. In addition, the base probability pX,k of an X-error on qubit k is increased by 0.15 if k𝒩(i) where i labels one specific plaquette of the lattice to model, e.g., a manufacturing defect as in Sec. 5.2.

To further challenge the learning algorithm in this scenario, we only allow a limited number of trials in the second stage (less than 10% of the first stage) since any actual experiment would be much more expensive than simulations. Despite the significant difference between the error models and ~1, we observe in Fig. 12 that the agents are able to significantly capitalize on the knowledge obtained from the initial training simulations (blue curve). In contrast, the same agent without pre-training (plotted in green in Fig. 12) barely learns at all during the allotted number of trials.

This advantage is indeed remarkable. However, it would be of no practical use if the benefits from transfer learning were not robust to variations in the experimental noise channel. That is, we expect the advantage of transfer learning to extend to other noise channels, too. Therefore, let us exchange the error channel ~1 by another, new model ~2 where we replace spatial correlations by a doubled base X-error rate, i.e.

~2(ρ)=pXXρX+pZZρZ+(1-pX-pZ)ρ, (8)

with pZ=0.14 and pX=0.04. Now, choosing the same pre-trained agent as before and transferring it to the new setting, we find that, despite the drastic difference between ~1 and ~2, the advantage gained from transfer learning is still substantial (which can be concluded from the comparison between the purple and pale rose curve in Fig. 12). This remarkable and robust advantage showcases the benefits of transfer learning for QEC in resource-limited, near-term quantum devices.

7 Discussion

Reinforcement learning (RL) has recently seen a great deal of success, from human-level performance in Atari games [51] to beating the world champion in the game of Go [65]. In 2017, RL was included in the list of 10 breakthrough technologies of the year in the MIT technology review [42]. As machine learning is claiming its place in the physicist’s toolbox [76], RL is starting to appear in quantum physics research [50, 39, 67, 31, 6, 7].

In this work, we have presented an RL framework for adapting and optimizing quantum error correction (QEC) codes. This framework is based on an RL agent that interacts with a quantum memory (and its classical control), providing the latter with instructions for modifications of the code to lower the logical error rate below a desired threshold. For the simulations discussed here, the quantum memory is realized as a surface code to which the agent may add qubits by way of fault-tolerant code deformations [17, 15]. The agent receives a reward once the specified logical error rate is reached. The internal structure of the agent has been modeled within the Projective Simulation [19] approach to RL. The classical control system estimates the logical error rate in the simulated QEC procedure using an optimal linear-time decoder [23, 24].

Our results demonstrate that the agents learn to protect surface code quantum memories from various types of noise, including simple i.i.d. errors, but also more complicated correlated and non-isotropically distributed errors. In particular, this RL approach provides interesting solutions for nontrivial QEC code optimization problems. A particularly noteworthy feature is the ability of the agents to transfer their experience from one noise model or task to another even if they are seemingly very different. This suggests that such a QEC strategy based on RL can be used to switch from off-line optimization to on-line adaptive error correction. That is, provided a reasonable guess for the type of expected errors, one may start by training an RL agent on simulations. Then, the trained agent can be used to bootstrap optimization in the actual hardware.

The scope of our simulations has been designed specifically with such applications to current state-of-the-art quantum computing platforms [34, 77, 14] in mind. Starting with 18 initial qubits, the agents we consider are able to extend this number by up to 50 additional qubits, thus also accounting for expected near-term technology developments. A potential bottleneck for extensions to much larger qubit numbers lies in the scaling behavior of the learning complexity. There, one expects that the increase in learning time scales unfavorably with the increase in the size of the percept space (and action space, both of which depend on the number of qubits). We envisage that this problem can be circumvented through parallel processing by assigning individual agents to different surface code patches of a fixed size. All agents can then operate in parallel, allowing one to exploit a number of already available RL techniques for parallel learning that explore the use of shared experience between agents [43, 12].

At the same time, the practical success of machine learning techniques also depends on suitably narrowing down the optimization problem. For example, in Ref. [31], neural networks were used to determine sequences of quantum gates and measurements to protect a logical qubit. This allows the algorithm to search the whole space of quantum circuits. However, this comes at a cost. The space of all possible QEC strategies that the algorithm explores is so vast that scaling inevitably becomes an issue. While this work demonstrates a successful strategy on up to 4 data qubits subject to uncorrelated bit-flip errors, significant advances would be needed to generalize this approach to larger, potentially varying numbers of data qubits and more realistic noise. In contrast to the approach of [31], the method developed here can adapt and optimize QEC codes in terms of their resource efficiency, i.e., the number of data qubits needed to achieve a desired maximal logical error rate, and operates without detailed information about or precise simulation of the underlying quantum states.

Moreover, let us note that the presented general framework for RL-based optimization of error correction codes goes beyond the specific simulations we have performed here. That is, the approach is directly applicable also if one wishes to consider RL paradigms other than PS, QEC codes other than surface codes, or noise models other than those considered. A particular contributor to this flexibility is that both the errors and the error decoding appear as part of a black-box environment to the agent, who only ever perceives whether the logical error rate is above or below target threshold. For instance, one is left with the option of choosing different decoders, and even of incorporating machine learning into the decoder itself [72, 10, 21, 67, 9, 54, 46, 44, 6]. Having this freedom in choosing the decoder is particularly relevant if different types of QEC codes are considered, e.g., if one allows fault-tolerant code switching through code deformation [15] or lattice surgery [59] as part of the optimization. In summary, while the simulations have been carried out within a specific setting, this framework lays the groundwork for applying sophisticated RL algorithms to more general (topological) QEC codes and more realistic noise models [2, 53, 33, 55].


HPN and HJB achnowledge support from the Austrian Science Fund (FWF) through the project DK-ALM: W1259-N27. HJB was also supported by the Ministerium für Wissenschaft, Forschung, und Kunst Baden-Württemberg (AZ:33-7533.-30-10/41/1). NF acknowledges support from the Austrian Science Fund (FWF) through the project P 31339-N27, the START project Y879-N27, and the joint Czech-Austrian project MultiQUEST (I 3053-N27 and GF17-33780L). HPN would like to acknowledge helpful comments from Davide Orsucci, Robert Raussendorf and Lea Trenkwalder. ND would like to thank Michael Beverland, Krysta Svore, Dave Wecker and Nathan Wiebe for useful discussions.

Appendix A Learning in Projective Simulation

In Sec. 4 we have given a brief introduction the projective simulation (PS) model for reinforcement learning (RL). Here we complete the description with a detailed account of how the h-values that govern the transition probabilities in Eq. (3) are updated. In other words, we explain how learning manifests itself in the PS model. The update rule from time step t to t+1 is

h(t+1)=h(t)+λ(t)g(t)+γ(1-h(t)) (9)

where the g-matrix, or so-called glow matrix, redistributes rewards λ to past experiences such that experiences that lie further in the past are rewarded only by a decreasing fraction of λ(t). In other words, past experiences are remembered (they ‘glow’) less strongly and therefore less likely to influence future behaviour, which is represented by a decreasing factor, i.e. the glow value. The glow matrix contains the long-term memory of the agent. A long-term memory is crucial when rewards are delayed, i.e., not every action results into a reward. The glow matrix is updated in parallel with the h-matrix. At the beginning, g is initialized as an all-zero matrix. Every time an edge (i,j) is traversed during the decision-making process, the associated glow value gij is set to MiM0 where Mi is the number of actions available for percept i and M0 is the number of initial actions. In order to internalize how much of the reward λ is issued to past experiences in Eq. (9) the g-matrix is also updated after each interaction with the environment. Therefore, we introduce the so-called glow parameter η[0,1] of the PS model and define an update rule as follows,

g(t+1)=(1-η)g(t). (10)

Besides glow, the agent is also subject to a forgetting mechanism, which is presented by the parameter γ in Eq. (9). Effectively, the h-values are decreased by γ[0,1] (but never below 1) whenever an update is initialized. Here, the forgetting mechanism is used specifically to reinforce exploratory behavior. In order to save memory, we also introduce a deletion mechanism where unused percept clips are deleted, i.e., if the average of outgoing h-values is below a value 1+δ. However, deletion can only be applied to a specific clip once a certain number of rewarded interactions τ with the environment have passed since its creation. This parameter τ is hence called immunity time. Moreover, after any given trial, all clips that have been created during said trial are deleted if no reward was given at all. Note that finding the optimal values for β,η,γ,τ and δ is generally a hard problem. However, it can be learned by the PS model itself [45] at the expense of longer learning times. However, hyperparametrization is often easier and faster in PS than in comparable RL models [48].

Appendix B Details of the environment

In Sec. 5, we describe the quantum memory environment the PS agent is interacting with. The core component of the classical control (see Fig. 1) is the SQUAB algorithm [25] which simulates error correction on an arbitrary surface code. Note that error channels Eq. (4)- (6) are simulated through erasure channels in SQUAB and hence yield error rates that differ slightly from their actual Pauli counterparts. Fortunately, the use of erasure channels is well motivated (see Sec. 6.1).

The remainder of the environment is dedicated to the processing of percepts and actions. In the following, we give the details of what constitutes percepts and actions. Roughly, a percept represents the code structure of the current surface code. A surface code is fully described by the graph it is defined on. A graph can be represented by its adjacency matrix A where rows represent vertices. Each vertex v is represented by a set of edges adjacent to v, Av=(e1,e2,). Edges are just labeled according to their first appearance in A. That is, A0=(0,1,). Since we require a spatial resolution of the code, each vertex v and plaquette p is assigned a label v,p{0,1,2,}. Edges, however, have no meaningful label. The specific code representation considered in this paper is hence a tuple of ordered adjacency matrices for the primal and dual graph. Actions have a straightforward representation that encodes Fig. 3. Each action is a vector a=(d,v,p1,p2). d=0,1 decides whether or not this action acts on the dual lattice. v labels a vertex on the corresponding lattice and p1,p2 are two non-neighbouring plaquettes sharing edges with v on the lattice’s dual. In the spirit of our basic moves from Fig. 3, a labels the vertex v that is being split such that plaquettes p1,p2 become connected by an edge. For practical reasons, the actions are limited such that the connectivity of each vertex and plaquette can be at most 8 and at least 3. In that way, double edges are also excluded.

Appendix C Searching the space of surface codes

With this basic set of moves at our disposal (see Fig. 3), one could be tempted to run an exhaustive search over the space of all surface codes. However, this problem is intractable, as we will see below. Suppose we start from a topological code and search for a better code by increasing the code size. In order to preserve the topological structure of the code, additional qubits are injected by adding extra edges to the lattice or its dual as in Fig. 3. Such actions increase the number of data qubits, while preserving the number of logical qubits. For instance, the 3×3 square lattice admits 18 primal actions and 18 dual actions (2 per square face). A single extra qubit hence allows to reach 36 more topological codes. With two extra qubits, we already obtain 1440 new codes. Some of these codes may be identical but we treat them as distinct codes here in order to avoid the extra cost of comparing all new codes. That means that we navigate in a tree whose root is the initial code. Each node represents a topological code and the successors of a node are obtained by adding an edge as in Fig. 3.

Imagine that we want to explore a region of the space of surface codes centered at the 3×3 square lattice. We use the software SQUAB [23, 24] to perform 10,000 decoding trials for each code. This costs roughly 0.018 seconds for a single code with 20 data qubits (using a processor 2.4 GHz Intel Core i5). SQUAB returns an estimation of the maximum likelihood decoder performance over the quantum erasure channel for a given topological code (see Sec. 6.1). To simplify, we focus on the residual Z-error after correction and we ignore the X-part of the error. This is similar to the scenario considered in the main text with an error channel of the form Eq. (4). Now, we want to explore all codes up to distance r around the initial, 3×3 surface code on a square lattice (see Fig. 2). In other words, we intend to explore the full “ball” of radius r around the central node. The number of codes C(r) at distance r from the root grows as follows, C(0)=1, C(1)=36, C(2)=1,440, C(3)=62,893, C(4)=2,961,504. Now, adding one extra qubit increases the number of codes by about a factor of 50. Running SQUAB in a ball of radius 5 with 10,000 trials for each code would then require more than 30 days of computation. Similarly, going through all the codes obtained by adding up to 10 qubits would at least require 20 million years of CPU time. This is without even counting the cost of generating the search graph and realizing all the moves. The cost is even more discouraging when a more realistic noise model is considered increasing the simulation cost per node. Hence, it is not reasonable to consider performing a numerical simulation of all the codes within the neighbourhood of an initial code in order to select the best candidate.

Figure 13: Exploration of random branches of the ball of radius 8 centered at the 3×3 surface code on a square lattice. Each node corresponds to a surface code on a torus. The color shading indicates the logical error rate. This rate is estimated using SQUAB with 10,000 trials per node at an error rate of pZ=0.15 where X errors are ignored.

Exploring the whole neighbourhood of a code for a sufficiently large radius is hopelessly difficult. However, we can still use random search techniques to explore and visualize the environment, i.e. the space of surface codes. To this end, we explore a subset of branches and calculate the respective logical error rates at each node. We start by evaluating all the codes at distance one from the root to ensure a minimum number of nodes. Then we continue building each successor of a code in the search tree with probability pexpl. Starting from a 3×3 surface code, we are able to partially explore a ball of radius 8 with exploration probability pexpl=0.03 within a few minutes. This leads us to a set of 1,230 randomly selected codes shown in Fig. 13. We observe that moves which effectively increase the logical error rate are common. Moreover, after unnecessarily increasing the logical error rate, it is still possible to decrease the logical error rate again. In particular, it is very likely that the best codes are hidden among unexplored branches of the search tree. In order to explore in priority the most promising regions of the space of all topological codes, we chose RL.

Figure η γ δ SQUAB iterations
6 0.05 0.01 0.01 1,000,000
6 0.01 0.01 0.01 1,000,000
8 0.05 0.0006 0.001 1,000,000
10 trials 06,000 0.05 0.01 0.01 1,000,000
10 trials 6,00010,000 0.05 0.0005 0.001 4,000,000
12 0.05 0.0006 0.001 1,000,000
12 trials 06,000 0.05 0.01 0.01 1,000,000
12 trials 6,0006,500 0.05 0.0006 0.001 1,000,000
Table 1: Parameters of the PS agent (see Appendix A) and SQUAB algorithm as used for the various tasks considered in Sec. 5 and 6. Two parameters remained the same for every setting: the softmax parameter β=2, and the immunity time τ=30.

Appendix D Parameters

For the results that are described in Sec. 5 and displayed in Figs. 68, we deployed a PS agent as described in Sec. 4 and Appendix A. There are many parameters that can be tuned. For the specific results in this paper, the chosen parameters are displayed in Table. 1. In particular, parameters can be tuned on-line (e.g. see Ref. [45]). Considering a task as in Fig. 10 or 12, it makes perfect sense to also change parameters between settings.


  • [1] S. Aaronson and D. Gottesman (2004) Improved simulation of stabilizer circuits. Phys. Rev. A 70, pp. 052328. External Links: Document, Link, arXiv:quant-ph/0406196 Cited by: §6.1.
  • [2] D. Aharonov, A. Kitaev, and J. Preskill (2006-02) Fault-tolerant quantum computation with long-range correlated noise. Phys. Rev. Lett. 96, pp. 050504. External Links: Document, Link Cited by: §7.
  • [3] P. Aliferis, F. Brito, D. P. DiVincenzo, J. Preskill, M. Steffen, and B. M. Terhal (2009) Fault-tolerant computing with biased-noise superconducting qubits: a case study. New J. Phys. 11 (1), pp. 013061. External Links: Document, Link, arXiv:0806.0383 Cited by: §6.2.
  • [4] J. M. Amini, H. Uys, J. H. Wesenberg, S. Seidelin, J. Britton, J. J. Bollinger, D. Leibfried, C. Ospelkaus, A. P. VanDevender, and D. J. Wineland (2010) Toward scalable ion traps for quantum information processing. New J. Phys. 12 (3), pp. 033031. External Links: Document, Link, arXiv:0909.2464 Cited by: §3.3, §5.2.
  • [5] S. Anders and H. J. Briegel (2006) Fast simulation of stabilizer circuits using a graph-state representation. Phys. Rev. A 73, pp. 022334. External Links: Document, Link, arXiv:quant-ph/0504117 Cited by: §6.1.
  • [6] P. Andreasson, J. Johansson, S. Liljestrand, and M. Granath (2018) Quantum error correction for the toric code using deep reinforcement learning. External Links: arXiv:1811.12338 Cited by: §1, §7, §7.
  • [7] M. August and J. M. Hernández-Lobato (2018) Taking gradients through experiments: lstms and memory proximal policy optimization for black-box quantum control. External Links: arXiv:1802.04063 Cited by: §7.
  • [8] Y. Aytar and A. Zisserman (2011-11) Tabula rasa: model transfer for object category detection. In 2011 International Conference on Computer Vision, Vol. , pp. 2252–2259. External Links: Document, ISSN 2380-7504 Cited by: §6.2.
  • [9] P. Baireuther, M. D. Caio, B. Criger, C. W. J. Beenakker, and T. E. O’Brien (2019) Neural network decoder for topological color codes with circuit level noise. New J. Phys. 21, pp. 013003. External Links: Document, Link, arXiv:1804.02926 Cited by: §1, §7.
  • [10] P. Baireuther, T. E. O’Brien, B. Tarasinski, and C. W. J. Beenakker (2018) Machine-learning-assisted correction of correlated qubit errors in a topological code. Quantum 2, pp. 48. External Links: Document, Link, arXiv:1705.07855 Cited by: §1, §7.
  • [11] R. Barends, J. Kelly, A. Megrant, A. Veitia, D. Sank, E. Jeffrey, T. C. White, J. Mutus, A. G. Fowler, B. Campbell, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, C. Neill, P. O’Malley, P. Roushan, A. Vainsencher, J. Wenner, A. N. Korotkov, A. N. Cleland, and J. M. Martinis (2014-04-23) Superconducting quantum circuits at the surface code threshold for fault tolerance. Nature 508, pp. 500–503. External Links: Link, arXiv:1402.4848 Cited by: §2, §3.3, §5.2.
  • [12] E. Barrett, J. Duggan, and E. Howley (2014) A parallel framework for bayesian reinforcement learning. Connect. Sci. 26 (1), pp. 7–23. External Links: Document Cited by: §7.
  • [13] C. H. Bennett, D. P. DiVincenzo, and J. A. Smolin (1997-04) Capacities of quantum erasure channels. Phys. Rev. Lett. 78, pp. 3217–3220. External Links: Document, Link, arXiv:quant-ph/9701015 Cited by: §5, §6.1.
  • [14] H. Bernien, S. Schwartz, A. Keesling, H. Levine, A. Omran, H. Pichler, S. Choi, A. S. Zibrov, M. Endres, M. Greiner, V. Vuletić, and M. D. Lukin (2017) Probing many-body dynamics on a 51-atom quantum simulator. Nature 551, pp. 579–584. External Links: Document, Link, arXiv:1707.04344 Cited by: §1, §5, §7.
  • [15] H. Bombín and M. A. Martin-Delgado (2009) Quantum measurements and gates by code deformation. J. Phys. A: Math. Theor. 42 (9), pp. 095302. External Links: Document, Link, arXiv:0704.2540 Cited by: §2, §3.2, §6, §7, §7.
  • [16] R. Bowler, J. Gaebler, Y. Lin, T. R. Tan, D. Hanneke, J. D. Jost, J. P. Home, D. Leibfried, and D. J. Wineland (2012) Coherent Diabatic Ion Transport and Separation in a Multizone Trap Array. Phys. Rev. Lett. 109, pp. 080502. External Links: Document, Link, arXiv:1206.0780 Cited by: §3.3, §5.2.
  • [17] S. Bravyi and A. Kitaev (1998) Quantum codes on a lattice with boundary. External Links: arXiv:quant-ph/9811052 Cited by: §2, §2, §3.1, §3.2, §6, §7.
  • [18] S. Bravyi and R. König (2013) Classification of Topologically Protected Gates for Local Stabilizer Codes. Phys. Rev. Lett. 110, pp. 170503. External Links: Document, Link, arXiv:1206.1609 Cited by: §3.3.
  • [19] H. J. Briegel and G. De las Cuevas (2012) Projective simulation for artificial intelligence. Sci. Rep. 7, pp. 400. External Links: Document, Link, arXiv:1104.3787 Cited by: §2, §4, §7.
  • [20] E. Campbell, A. Khurana, and A. Montanaro (2018) Applying quantum algorithms to constraint satisfaction problems. External Links: arXiv:1810.05582 Cited by: §1.
  • [21] C. Chamberland and P. Ronagh (2018) Deep neural decoders for near term fault-tolerant experiments. Quant. Sci. Techn. 3 (4), pp. 044002. External Links: Document, Link, arXiv:1802.06441 Cited by: §1, §7.
  • [22] R. Cleve and D. Gottesman (1997) Efficient computations of encodings for quantum error correction. Phys. Rev. A 56, pp. 76–82. External Links: Document, Link, arXiv:quant-ph/9607030 Cited by: §6.1.
  • [23] N. Delfosse, P. Iyer, and D. Poulin (2016) A linear-time benchmarking tool for generalized surface codes. External Links: arXiv:1611.04256 Cited by: Appendix C, §1, §2, §6.1, §7.
  • [24] N. Delfosse and P. Iyer (2016) SQUAB – a fast benchmarking software for surface quantum computing architectures. Note: [Online; accessed 20-December-2018] External Links: Link Cited by: Appendix C, §1, §2, §6.1, §7.
  • [25] N. Delfosse and N. H. Nickerson (2017) Almost-linear time decoding algorithm for topological codes. External Links: arXiv:1709.06218 Cited by: Appendix B, §1, Figure 11, §6.1.
  • [26] N. Delfosse and G. Zémor (2017) Linear-Time Maximum Likelihood Decoding of Surface Codes over the Quantum Erasure Channel. External Links: arXiv:1703.01517 Cited by: §2, §5, Figure 11, §6.1.
  • [27] E. Dennis, A. Kitaev, A. Landahl, and J. Preskill (2002) Topological quantum memory. J. Math. Phys. 43 (9), pp. 4452–4505. External Links: Document, Link, arXiv:quant-ph/0110143 Cited by: §2, §3.1.
  • [28] D. P. DiVincenzo and P. W. Shor (1996) Fault-Tolerant Error Correction with Efficient Quantum Codes. Phys. Rev. Lett. 77, pp. 3260–3263. External Links: Document, Link, arXiv:quant-ph/9605031 Cited by: §6.1.
  • [29] V. Dunjko and H. J. Briegel (2018) Machine learning & artificial intelligence in the quantum domain: a review of recent progress. Rep. Prog. Phys. 81, pp. 074001. External Links: Document, Link, arXiv:1709.02779 Cited by: §1.
  • [30] V. Dunjko, Y. Ge, and J. I. Cirac (2018) Computational Speedups Using Small Quantum Devices. Phys. Rev. Lett. 121, pp. 250501. External Links: Document, Link, arXiv:1807.08970 Cited by: §1.
  • [31] T. Fösel, P. Tighineanu, T. Weiss, and F. Marquardt (2018) Reinforcement Learning with Neural Networks for Quantum Feedback. Phys. Rev. X 8, pp. 031084. External Links: Document, Link, arXiv:1802.05267 Cited by: §7, §7.
  • [32] A. G. Fowler, M. Mariantoni, J. M. Martinis, and A. N. Cleland (2012) Surface codes: towards practical large-scale quantum computation. Phys. Rev. A 86, pp. 032324. External Links: Document, Link, arXiv:1208.0928 Cited by: §2, §3.1, §3.1.
  • [33] A. G. Fowler and J. M. Martinis (2014-03) Quantifying the effects of local many-qubit errors and nonlocal two-qubit errors on the surface code. Phys. Rev. A 89, pp. 032316. External Links: Document, Link Cited by: §7.
  • [34] N. Friis, O. Marty, C. Maier, C. Hempel, M. Holzäpfel, P. Jurcevic, M. B. Plenio, M. Huber, C. Roos, R. Blatt, and B. Lanyon (2018) Observation of Entangled States of a Fully Controlled 20-Qubit System. Phys. Rev. X 8, pp. 021012. External Links: Document, Link, arXiv:1711.11092 Cited by: §1, §7.
  • [35] K. Fujii and Y. Tokunaga (2012) Error and loss tolerances of surface codes with general lattice structures. Phys. Rev. A 86, pp. 020303(R). External Links: Document, Link, arXiv:1202.2743 Cited by: §1, §2, §3.2, §3.3, §5.1, §5.1.
  • [36] D. Gottesmann (1997) Stabilizer Codes and Quantum Error Correction. Ph.D. Thesis, Caltech. External Links: arXiv:quant-ph/9705052, Link Cited by: §1, §2, §3.1, §3.1.
  • [37] M. Grassl, Th. Beth, and T. Pellizzari (1997-07) Codes for the quantum erasure channel. Phys. Rev. A 56, pp. 33–38. External Links: Document, arXiv:quant-ph/9610042 Cited by: §5.
  • [38] S. Hangl, E. Ugur, S. Szedmak, and J. Piater (2016) Robotic playing for hierarchical complex skill learning. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2799–2804. External Links: Document, Link, ISSN 2153-0866, arXiv:1603.00794 Cited by: §2.
  • [39] R. Iten, T. Metger, H. Wilming, L. del Rio, and R. Renner (2018) Discovering physical concepts with neural networks. External Links: arXiv:1807.10300 Cited by: §7.
  • [40] T. Karzig, C. Knapp, R. M. Lutchyn, P. Bonderson, M. B. Hastings, C. Nayak, J. Alicea, K. Flensberg, S. Plugge, Y. Oreg, C. M. Marcus, and M. H. Freedman (2017-06) Scalable designs for quasiparticle-poisoning-protected topological quantum computation with majorana zero modes. Phys. Rev. B 95, pp. 235305. External Links: Document, Link, arXiv:1610.05289 Cited by: §2.
  • [41] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi (1983) Optimization by simulated annealing. Science 220 (4598), pp. 671–680. External Links: Document, ISSN 0036-8075, Link Cited by: §5.3.
  • [42] Knight, Will (2017) Reinforcement learning – by experimenting, computers are figuring out how to do things that no programmer could teach them.. Note: [Online; accessed 20-December-2018] External Links: Link Cited by: §7.
  • [43] M. R. Kretchmar (2002) Parallel reinforcement learning. In The 6th World Conference on Systematics, Cybernetics, and Informatics, pp. 165–170. Cited by: §7.
  • [44] Y. Liu and D. Poulin (2019) Neural Belief-Propagation Decoders for Quantum Error-Correcting Codes. Phys. Rev. Lett. 122, pp. 200501. External Links: Document, Link, arXiv:1811.07835 Cited by: §1, §7.
  • [45] A. Makmal, A. A. Melnikov, V. Dunjko, and H. J. Briegel (2016) Meta-learning within Projective Simulation. IEEE Access 4, pp. 2110–2122. External Links: Document, Link, arXiv:1602.08017 Cited by: Appendix A, Appendix D.
  • [46] N. Maskara, A. Kubica, and T. Jochym-O’Connor (2019) Advantages of versatile neural-network decoding for topological codes. Phys. Rev. A 99, pp. 052351. External Links: Document, Link, arXiv:1802.08680 Cited by: §1, §7.
  • [47] J. Mautner, A. Makmal, D. Manzano, M. Tiersch, and H. J. Briegel (2015) Projective Simulation for Classical Learning Agents: A Comprehensive Investigation. New Gener. Comput. 33 (1), pp. 69–114. External Links: Document, Link, arXiv:1305.1578 Cited by: §2.
  • [48] A. A. Melnikov, A. Makmal, and H. J. Briegel (2018) Benchmarking projective simulation in navigation problems. IEEE Access 6, pp. 64639–64648. External Links: Document, Link, arXiv:1804.08607 Cited by: Appendix A, §2.
  • [49] A. A. Melnikov, A. Makmal, V. Dunjko, and H. J. Briegel (2017) Projective simulation with generalization. Sci. Rep. 7, pp. 14430. External Links: Document, Link, arXiv:1504.02247 Cited by: §2, §6.2.
  • [50] A. A. Melnikov, H. Poulsen Nautrup, M. Krenn, V. Dunjko, M. Tiersch, A. Zeilinger, and H. J. Briegel (2018) Active learning machine learns to create new quantum experiments. Proc. Natl. Acad. Sci. U.S.A. 115, pp. 1221–1226. External Links: Document, Link, arXiv:1706.00868 Cited by: §2, §6.2, §7.
  • [51] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis (2015) Human-level control through deep reinforcement learning. Nature 518, pp. 529–533. External Links: Document, Link Cited by: §7.
  • [52] T. Monz, P. Schindler, J. T. Barreiro, M. Chwalla, D. Nigg, W. A. Coish, M. Harlander, W. Hänsel, M. Hennrich, and R. Blatt (2011-03) 14-Qubit Entanglement: Creation and Coherence. Phys. Rev. Lett. 106, pp. 130506. External Links: Document, Link, arXiv:1009.6126 Cited by: §1.
  • [53] H. K. Ng and J. Preskill (2009-03) Fault-tolerant quantum computation versus gaussian noise. Phys. Rev. A 79, pp. 032318. External Links: Document, Link Cited by: §7.
  • [54] X. Ni (2018) Neural network decoders for large-distance 2d toric codes. External Links: arXiv:1809.06640 Cited by: §1, §7.
  • [55] N. H. Nickerson and B. J. Brown (2019-04) Analysing correlated noise on the surface code using adaptive decoding algorithms. Quantum 3, pp. 131. External Links: Document, Link, ISSN 2521-327X Cited by: §7.
  • [56] M. A. Nielsen and I. L. Chuang (2000) Quantum Computation and Quantum Information. Cambridge University Press, Cambridge, U.K.. Cited by: §1, §1.
  • [57] F. Pastawski and B. Yoshida (2015) Fault-tolerant logical gates in quantum error-correcting codes. Phys. Rev. A 91, pp. 012305. External Links: Document, Link, arXiv:1408.1720 Cited by: §3.3.
  • [58] N. Patricia and B. Caputo (2014-06) Learning to learn, from transfer learning to domain adaptation: a unifying perspective. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, Vol. , pp. 1442–1449. External Links: Document, ISSN 1063-6919 Cited by: §6.2.
  • [59] H. Poulsen Nautrup, N. Friis, and H. J. Briegel (20172017) Fault-tolerant interface between quantum memories and quantum processors. Nat. Commun. 8 (1), pp. 1321. External Links: Document, Link, arXiv:1609.08062 Cited by: §7.
  • [60] J. Preskill (1997) Fault-tolerant quantum computation. In Introduction to Quantum Computation, H.-K. Lo, S. Popescu, and T. P. Spiller (Eds.), pp. 213–269. External Links: Document, Link, arXiv:quant-ph/9712048 Cited by: §1.
  • [61] D. Rosenberg, D. Kim, R. Das, D. Yost, S. Gustavsson, D. Hover, P. Krantz, A. Melville, L. Racz, G. O. Samach, S. J. Weber, F. Yan, J. L. Yoder, A. J. Kerman, and W. D. Oliver (2017) 3D integrated superconducting qubits. npj Quantum Information 3 (1), pp. 42. External Links: Document, Link, arXiv:1706.04116 Cited by: §3.3.
  • [62] L. Saitta and J. Zucker (2013) Abstraction in Artificial Intelligence and Complex Systems. Springer, New York, USA. Cited by: §6.2.
  • [63] P. Schindler, D. Nigg, T. Monz, J. T. Barreiro, E. Martinez, S. X. Wang, S. Quint, M. F. Brandl, V. Nebendahl, C. F. Roos, M. Chwalla, M. Hennrich, and R. Blatt (2013) A quantum information processor with trapped ions. New J. Phys. 15 (12), pp. 123012. External Links: Document, arXiv:1308.3096 Cited by: §1, §2, §3.3, §5.2, §6.2.
  • [64] M. D. Shulman, O. E. Dial, S. P. Harvey, H. Bluhm, V. Umansky, and A. Yacoby (2012) Demonstration of Entanglement of Electrostatically Coupled Singlet-Triplet Qubits. Science 336 (6078), pp. 202–205. External Links: Document, Link, arXiv:1202.1828 Cited by: §6.2.
  • [65] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis (2017) Mastering the game of Go without human knowledge. Nature 550, pp. 354–359. External Links: Document, Link Cited by: §7.
  • [66] R. S. Sutton and A. G. Barto (1998) Reinforcement Learning: An Introduction. MIT press, Cambridge. Cited by: §1, §4, §4.
  • [67] R. Sweke, M. S. Kesselring, E. P. L. van Nieuwenburg, and J. Eisert (2018) Reinforcement learning decoders for fault-tolerant quantum computation. External Links: arXiv:1810.07207 Cited by: §1, §7, §7.
  • [68] B. M. Terhal (2015) Quantum error correction for quantum memories. Rev. Mod. Phys. 87, pp. 307–346. External Links: Document, Link, arXiv:1302.3428 Cited by: §1, §2.
  • [69] S. Thrun (1996) Is learning the n-th thing any easier than learning the first?. In Advances in Neural Information Processing Systems 8, D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo (Eds.), pp. 640–646. External Links: Link Cited by: §2, §5.3, §6.2.
  • [70] T. Tommasi, F. Orabona, and B. Caputo (2010-06) Safety in numbers: learning categories from few examples with multi model knowledge transfer. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. , pp. 3081–3088. External Links: Document, ISSN 1063-6919 Cited by: §6.2.
  • [71] T. Tommasi and B. Caputo (2009) The more you know, the less you learn: from knowledge transfer to one-shot learning of object categories. In British Machine Vision Conference, External Links: Link Cited by: §6.2.
  • [72] G. Torlai and R. G. Melko (2017) Neural Decoder for Topological Codes. Phys. Rev. Lett. 119, pp. 030501. External Links: Document, Link, arXiv:1610.04238 Cited by: §1, §7.
  • [73] D. K. Tuckett, S. D. Bartlett, and S. T. Flammia (2018) Ultrahigh Error Threshold for Surface Codes with Biased Noise. Phys. Rev. Lett. 120, pp. 050505. External Links: Document, Link, arXiv:1708.08474 Cited by: §1.
  • [74] A. Vardy (1997) The intractability of computing the minimum distance of a code. IEEE T. Inform. Theory 43 (6), pp. 1757–1766. External Links: Document, Link Cited by: §1.
  • [75] K. Weiss, T. M. Khoshgoftaar, and D. Wang (2016-05-28) A survey of transfer learning. Journal of Big Data 3 (1), pp. 9. External Links: ISSN 2196-1115, Document, Link Cited by: §2, §5.3, §6.2.
  • [76] L. Zdeborová (2017) New tool in the box. Nat. Phys. 13, pp. 420–421. External Links: Document, Link Cited by: §7.
  • [77] J. Zhang, G. Pagano, P. W. Hess, A. Kyprianidis, P. Becker, H. Kaplan, A. V. Gorshkov, Z.-X. Gong, and C. Monroe (2017) Observation of a many-body dynamical phase transition with a 53-qubit quantum simulator. Nature 551, pp. 601–604. External Links: Document, Link, arXiv:1708.01044 Cited by: §1, §5, §7.