Learning to Train with Synthetic Humans

  • 2019-08-02 17:49:30
  • David T. Hoffmann, Dimitrios Tzionas, Micheal J. Black, Siyu Tang
  • 17


Neural networks need big annotated datasets for training. However, manualannotation can be too expensive or even unfeasible for certain tasks, likemulti-person 2D pose estimation with severe occlusions. A remedy for this issynthetic data with perfect ground truth. Here we explore two variations ofsynthetic data for this challenging problem; a dataset with purely synthetichumans and a real dataset augmented with synthetic humans. We then study whichapproach better generalizes to real data, as well as the influence of virtualhumans in the training loss. Using the augmented dataset, without consideringsynthetic humans in the loss, leads to the best results. We observe that notall synthetic samples are equally informative for training, while theinformative samples are different for each training stage. To exploit thisobservation, we employ an adversarial student-teacher framework; the teacherimproves the student by providing the hardest samples for its current state asa challenge. Experiments show that the student-teacher framework outperformsnormal training on the purely synthetic dataset.


Quick Read (beta)

Learning to Train with Synthetic Humans

David T. Hoffmann 1 1 Max Planck Institute for Intelligent Systems, Germany
2 University of Tübingen, Germany 3 ETH Zürich
1dhoffmann, dtzionas, black, [email protected]
   Dimitrios Tzionas 1 1 Max Planck Institute for Intelligent Systems, Germany
2 University of Tübingen, Germany 3 ETH Zürich
1dhoffmann, dtzionas, black, [email protected]
   Michael J. Black 1 1 Max Planck Institute for Intelligent Systems, Germany
2 University of Tübingen, Germany 3 ETH Zürich
1dhoffmann, dtzionas, black, [email protected]
   Siyu Tang 1 1 Max Planck Institute for Intelligent Systems, Germany
2 University of Tübingen, Germany 3 ETH Zürich
1dhoffmann, dtzionas, black, [email protected] 2233

Neural networks need big annotated datasets for training. However, manual annotation can be too expensive or even unfeasible for certain tasks, like multi-person 2D pose estimation with severe occlusions. A remedy for this is synthetic data with perfect ground truth. Here we explore two variations of synthetic data for this challenging problem; a dataset with purely synthetic humans and a real dataset augmented with synthetic humans. We then study which approach better generalizes to real data, as well as the influence of virtual humans in the training loss. Using the augmented dataset, without considering synthetic humans in the loss, leads to the best results. We observe that not all synthetic samples are equally informative for training, while the informative samples are different for each training stage. To exploit this observation, we employ an adversarial student-teacher framework; the teacher improves the student by providing the hardest samples for its current state as a challenge. Experiments show that the student-teacher framework outperforms normal training on the purely synthetic dataset.

1 Introduction

Figure 1: Qualitative comparisons between our models and Cao et al. [8].

The broad success of deep neural networks comes at a price: the ever growing need for huge amounts of labeled training data. For many tasks, the lack of data seems to be one of the major limiting factors of progress. It is particularly problematic for the tasks where manual labeling requires significant human effort, or is even unfeasible. For example, in multi-person 2D pose estimation, a major challenge is that people are often partially visible. Manual annotation of body joints that are severely occluded is error prone and the resulting labels are noisy. Computer graphics can help to resolve these issues. 3D rendering engines offer the opportunity to generate a large amount of data with perfect labels: e.g., the location of occluded body parts and the precise pose of the camera.

Nowadays, large scale synthetic datasets with reasonable realism can be generated relatively easy and the idea of synthesizing training data has been widely explored. In general, there are two common strategies for generating a synthetic dataset: rendering a purely synthetic dataset, and augmenting real training images with synthetic instances. The advantage of the former is the full control over the virtual 3D world and ability to generate high variance datasets [4, 5, 12, 29, 35, 36, 38, 44]. The advantage of the second approach is that some of the instances in the dataset are real, resulting in overall higher realism [2, 11, 40].

We generate both types of synthetic datasets. One purely synthetic dataset and a mixed dataset, which is generated by augmenting the MPII pose estimation dataset [3] with synthetic humans. In particular, we design these datasets to improve on frequent failure cases that we observe with state-of-the-art models (see Fig 1), namely uncommon camera angles and strong occlusion. By comparing generalization performance using these two datasets we obtain insights into which way of generating data is preferable. We further investigate how strongly the lack of photorealism of the synthetic humans limits generalization. To make the synthetic data more realistic, we propose a simple synthetic-to-real human style transfer algorithm, based on the work of Dundar et al. [10].

These experiments show that naive training with synthetic data leads only to limited improvements. One explanation is that training on large synthetic datasets leads to overfitting of the model to the features of synthetic data. We observe that some synthetic images convey more information than others. Overfitting to features of synthetic data could be limited by generating only useful, i.e. difficult synthetic data, and thus limiting the training on synthetic data.

As a step in this direction, we propose a method to use synthetic datasets more effectively. Specifically, we introduce an adversarial student-teacher framework. The teacher learns online which training data is still difficult. This information is then used to increase the sampling probability of similar examples. By taking into account feedback from the student, the teacher keeps on updating the sampling probabilities throughout training and adapts them to the specific needs of the student. Training with the teacher on the purely synthetic data outperforms normal training.

Our contributions can be summarized as follows: 1) We propose a large-scale synthetic multi-person dataset, a mixed dataset, and a domain-adapted version of the latter. 2) We explore which way of generating synthetic data is superior for our task. 3) We propose a student-teacher framework to train on the most difficult images and show that this method outperforms random sampling of training data on the synthetic dataset. We provide datasets and code11 1 https://ltsh.is.tue.mpg.de.

2 Related Work

Synthetic datasets with humans. The need for labelled training data has fueled development of datasets with synthetic humans. Many methods use 3D models of the human body to generate data [4, 5, 16, 35]. Other approaches augment 3D training data by utilizing 2D pose datasets [9, 37], while [40, 43] augment datasets with cut-outs of objects or animals. Closely related to our approach, [36, 44] render the SMPL model [27] on top of random indoor images. These methods generate datasets with a single synthetic human. Multi-person datasets were created by employing video games for pedestrian detection [29] and pose tracking [12]. Similarly, [30] develop a simulation environment in a game engine, including virtual humans. Related to our approach, [38] augment real training images similar to the ones in [44] but with multiple synthetic humans occluding each other.

Domain Adaptation. The quality of synthetic data is often insufficient to generalize well to real data. Several domain adaptation methods have been developed to overcome this problem. Shrivastava et al. [42] train a Generative Adversarial Network (GAN) to refine synthetic images, while keeping the label information intact. Recently, Cycle-GAN has been used to map images from one domain to another [4, 31]. However, GAN-based methods are prone to unstable training and require tedious hyper-parameter tuning. As a practical alternative, recently [10] proposed domain stylization to stylize synthetic images to real ones, using the fast photorealistic image stylization method of [25].

Human pose estimation. Multi-person pose estimation has attracted substantial attention over the last years [8, 14, 15, 19, 23, 28, 32, 33]. One of the most popular datasets is the MPII multi-person pose estimation dataset [3]. Among the best performing methods on MPII are: [33], which uses a pose partition network, [28], which uses context information, [15], which refines pose predictions, and [32], which predicts “tag maps” to solve the grouping problem. The most widely used method is OpenPose [8], a bottom-up approach that first predicts keypoints and then estimates Part Affinity Fields (PAFs) to group them.

Learning to train. Bengio et al. [6] introduce curriculum learning for learning systems, exploiting the idea that different data samples are informative at different training stages. As a proof of concept they manually define the samples for each stage with gradually increasing difficulty. Multiple methods have focused on automating curriculum learning [7, 13, 20, 24]. These approaches try to maximize information gain during training by carefully monitoring the learning success of the model. Alternatively, adversarial methods [21, 22, 41] try to pick the hardest samples at each training stage; [21] favors samples resulting in higher loss, [22] learns weights for the loss of each training sample as a soft curriculum, while [41] uses online hard example mining for object detection. Peng et al. [34] propose an adversarial training scheme to optimize data augmentation online. They train two teacher networks to learn a probability distribution over the hyper-parameters for data augmentation; one predicts the most difficult image rotations, while the other predicts parameters for deep feature occlusion.

3 Multi-person Synthetic Data

In the following we describe the generation of the datasets (Sections 3.1, 3.2, 3.3) and the domain adaptation used to increase visual appearance of the mixed dataset (Section 3.4).

3.1 Data Generation Pipeline

Figure 2: Schematic of the data generation pipeline.

To generate realistic synthetic training data we build on top of [36, 44]. We use multiple different sources of data to build a realistic synthetic scene. Images and ground truth annotations are rendered using Cycles, the rendering engine of Blender22 2 https://www.blender.org. An overview can be seen in Fig. 2.

Body Model and MoCap Data. We use the parametric body model SMPL+H [39] to generate realistic synthetic humans. SMPL+H is parameterized by pose θ78 and shape β10. We collect realistic pose and shape parameters by fitting SMPL+H to standard MoCap data by using MoSh [26].

Details. We draw inspiration from [36, 44] who generate small video sequences, each having different but fixed parameters for the position, pose, shape and texture of a synthetic human, the background image, camera position, lighting, etc. In contrast, we generate single images and render multiple synthetic humans, while randomizing the number of them. As a result, we generate a dataset with much higher variance. Images with inter-penetrating meshes of virtual humans are rejected to avoid artifacts in the generated ground truth.

Further details regarding the description of the data generation pipeline, posing of hands and a quantitative comparison of our datasets to other datasets can be found in Supp. Mat.

Figure 3: Example images from the purely synthetic dataset. It contains high occlusion, extreme poses, various camera angles and various challenging backgrounds.

3.2 Synthetic Dataset

Background Images. To generalize well to in-the-wild pose estimation datasets the background images should come from many different scenes. To this end we use images from SUN397 [45] and reject all images with a resolution smaller than 512×512 pixels to ensure high quality backgrounds. Additionally, we reject all images containing humans, as we do not have ground truth annotations for them. We use mask-RCNN [1, 17] as our human detector.

Generative Factors. We sample the number of synthetic humans per image from a Poisson distribution with λ=9, to encourage many humans per image, while avoiding too extreme values. The datasets of [36, 44] have only very small variance in camera position. However, preliminary experiments show that the camera position significantly influences the difficulty of multi-person pose estimation. Therefore we increase the range of possible camera positions, by sampling the camera pitch uniformly from [0,45]. The resolution of the final rendered images is set to 640×640 pixels. We refer to this dataset as 𝒟S.

3.3 Mixed Dataset

We build upon the finding of [40] that realistic occluding objects lead to larger improvements than abstract objects. We choose our occluders to be from the same class as our target objects; i.e. humans, to simulate crowded scenes with multiple humans. To generate the dataset we use the pipeline described above with a few differences. Instead of SUN397 [45] we use the training images of the MPII human pose dataset [3] as background images. To keep the MPII ground truth intact, we render the images with the same resolution as the background MPII image, and keep the camera pose fixed. We then augment the MPII human pose dataset by superimposing synthetic humans. Their number is drawn from a Poisson distribution with λ=4 to introduce interesting and intense occlusions as shown in Fig. 4 (A), without extreme occlusions by too many synthetic humans. We render each of the 15,956 images in our training set 5 times with different parameters for increased variance. We refer to this dataset as 𝒟M.

Figure 4: (A) Example images from 𝒟M. (B) Corresponding images of 𝒟Style. For the last image, the segmentation network included non-human parts in the segmentation masks. Resulting artifacts can be seen for rightmost synthetic human.

3.4 Domain Stylization

The appearance of real and synthetic humans differs strongly. Factors contributing to these differences are the low quality of textures and differences in lighting conditions for synthetic humans and background images. Additionally, the small number of human textures limits the variability. These differences in appearance might limit the generalization. We draw inspiration from Dundar et al. [10] to reduce these differences by using the fast photorealistic style transfer method of [25]. Style transfer methods require a pair of images as input, a content image IC and a style image IS. While the style of these images can be largely different, their content should have similarities. Finding such pairs is a non-trivial problem. However, for the 𝒟M, we have a canonical choice of image pairs: the image from 𝒟M and its background as IC and IS, respectively.

Naive application of style transfer methods on the whole image, leads to severe artifacts. Therefore, only the style of semantically similar classes should be transferred. To obtain a good semantic segmentation network Dundar et al. [10] iterate between stylizing a dataset and training a network for semantic segmentation on the stylized dataset. Here, we are only interested in transferring the style of real to synthetic humans. Fine-grained human detection is important to avoid parts of the background bleeding into the foreground after style transfer (see Fig. 4 (B) right panel). We therefore employ Mask-RCNN [1, 17] to predict pixel-wise masks for humans. Ground truth masks for the synthetic humans are generated during data generation. Since the style-transfer algorithm can not handle images of arbitrary size, we rescale the larger images to 600 pixels before applying the style transfer. We refer to the resulting dataset as 𝒟Style. Examples can be seen in Fig. 4 (B).

4 Learning to Train with Multi-person Synthetic Data

We now turn to the task of training a pose estimation network with synthetic data. We first provide general information about our model. This is followed by a brief description of the training procedure. Finally, we detail the student-teacher framework and explain how the teacher is trained.

Pose Estimation Network. For all experiments we use our Tensorflow implementation of the network proposed in [8], which we will refer to as OpenPose network. Our training differs only slightly from [8]; details are provided in Supp. Mat. Because of these differences, we use a self-trained model on real data as the baseline throughout the paper.

Training with Synthetic Data. Following the advice of [18], we freeze the weights of our feature extractor whenever training on synthetic data. In particular, we freeze the first 4 layers of the OpenPose network. Additionally, we make sure that each batch is composed of 50% real and 50% synthetic images. More details on our training procedure with synthetic data are provided in Supp. Mat.

Grouping. Sampling the most difficult samples with higher probability comes with the risk of oversampling a small amount of data. To avoid such behavior, and ease the task for the teacher, we group the synthetic data into meaningful groups. We found empirically that the position of the camera and the distance of people in an image contribute to the difficulty of multi-person pose estimation. Thus, we use these two image characteristics to group our data. We assume 10 groups to yield a good trade-off between precision and difficulty for the teacher. For minimal distance grouping33 3 We sample persons not images. The image is cropped around this person. The minimal distance is defined as smallest distance of this person to any other person., denoted as mD, we decide for linearly spaced values between [0,640) px. For the camera pitch grouping, denoted as C we space the group boundaries linearly in the interval [min(X)+Var(X),max(X)-Var(X)), where X contains all values for the camera pitch in the dataset.

Figure 5: Diagram of the forward pass. The total loss is denoted as t, the loss on real images as t𝒟R and on synthetic images as t𝒟S. The reward signal and backward pass are explained in the main text. For clarity only 6 instead of 10 groups are displayed.

4.1 Adversarial Teacher

The teacher network is trained simultaneously with the student network to adapt the sampling strategy dynamically to the current training state of the student. A schematic of the forward pass can be seen in Fig. 5. The input to the teacher should represent the training state of the student. We choose the real image resulting in highest mean loss per joint within the previous N training steps. This provides some information about the type of images that are still difficult for the student. The output of the teacher network is a probability distribution P~ over a set of groups {g1,,gi}. This probability distribution is used to sample one of them. For the next N training steps, synthetic training samples are drawn from this group only.

Training the Teacher. The objective of the adversarial teacher should be to maximize the loss, t𝒟S. Unfortunately, sampling and data augmentation are non-differentiable operations, prohibiting end-to-end training of the teacher network. An alternative method to provide a supervision signal is needed. We draw inspiration from [34] and employ a reward/penalty training scheme. To determine whether the teacher is rewarded or penalized we monitor t𝒟S. If t𝒟St-1𝒟S the teacher succeeded in finding more difficult examples than before and is rewarded. Unfortunately, t𝒟S has a high variance. To reduce the variance of the teaching signal we reward if

t𝒟S1Hh=0Ht-1-h𝒟S, (1)

where H denotes the number of past loss values to be considered. To avoid favoring images with many people, we use the mean loss per joint on the image. Eq. 1 provides the direction of the gradient descent step but no ground truth is given to compute the gradients. To efficiently get a reward signal we follow [34] and increase the probability of a group being chosen, if the teacher gets rewarded. Probabilities for other groups are decreased accordingly. Formally, we update Pi and Pj, where i denotes the selected group and ji denotes all other groups, by

Pi=P~i+δαPi~, and Pj=Pj~-δαPi~|g|-1. (2)

Here Pi~ denotes the prediction of the teacher, Pi is the updated pseudo ground truth probability, 0α1 controls the size of the update, |g| denotes the number of groups and δ is a sign indicator δ={+1, if Eq. 1 holds; -1, otherwise}. Finally, we obtain gradients to update the teacher network by computing the KL-divergence loss between P and P~. Information on optimization related hyperparameters and the architecture are provided in Supp. Mat.

During training we face a exploration/exploitation trade-off. The group with highest probability, might not be optimal. To overcome this problem, we sample a group from a uniform distribution instead of the predicted probabilities with probability ϵ=0.1.

5 Experiments

We test our models on the MPII multi-person pose dataset [3]. For that purpose we use the toolkit provided by [3]. Following the standard validation procedure, the test metric is only computed for people within close proximity. We split the real training data, denoted as 𝒟R, into a training and a validation set. Our validation set consist of 343 randomly selected groups of people in close proximity. The respective images are not used for training. We report mAP (mean Average Precision), the main metric of the benchmark [3] for each model.

5.1 Which Dataset Generalizes Best?

We hypothesize that multi-person pose estimation methods are limited by a lack of training data. To test this hypothesis we train our model on 𝒟S and 𝒟R+𝒟S. Interestingly, for 𝒟S, the model resulting from training only on 𝒟S, mAP is very low, suggesting that the model overfits to the synthetic data. However, when training on 𝒟R+𝒟S the mAP improves over the mAP of 𝒟R (Tab. 1).

Table 1: Results on the held out validation set. 𝒟R is trained only with real data. 𝒟S was trained solely on synthetic data. 𝒟R+𝒟S is trained on real and synthetic data. 𝒟R+𝒟M was trained on real and the mixed dataset.
Model Head Shoulder Elbow Wrist Hip Knee Ankle mAP
𝒟R 91.3 89.1 79.2 70.4 75.9 71.5 66.7 77.7
𝒟S 37.9 23.5 12.7 7.3 5.6 3.4 3.2 13.4
𝒟R+𝒟S 91.1 89.2 80.5 71.0 75.2 73.6 68.1 78.4
𝒟R+𝒟M 91.3 89.5 80.7 71.7 75.4 72.5 67.7 78.4

While synthetic data can improve the accuracy of multi-person pose estimation, the improvements are relatively small given the extensive amount of additional training data. Multiple factors might limit the generalization. It could well be that the dataset bias between the synthetic and real dataset is just too strong. Generating the 𝒟M is a straightforward way of generating a dataset with similar dataset bias as the original dataset. By training on 𝒟M we can quantify the influence of it on the generalization. When training with 𝒟M we consider only real humans as samples. The rationale behind that decision is that we primarily want to increase the frequency of occlusion of real humans. However, the network is also trained on all synthetic humans that are within the cropped training image. As can be seen in Tab. 1, training on 𝒟R+𝒟M results in similar accuracy as 𝒟R+𝒟S. Therefore, the two methods of generating data are equivalently good. The dataset bias seems not to be the main limiting factor.

5.2 Does Stylization Improve Generalization?

The generalization might be limited by the appearance of synthetic humans. To overcome the limited generalization and measure how the difference in appearance influences performance we train on 𝒟R+𝒟Style. This improves accuracy over 𝒟R, but leads to a decrease of mAP compared to 𝒟R+𝒟M (see Tab. 2). This result is surprising, as many factors believed to limit generalization are improved and the data is visually more realistic. A possible explanation is that the network learns to detect artifacts of the style transfer. Alternatively, the failure cases of the style transfer method might lead to “confusion” of the network.

Table 2: Results for the models 𝒟R+𝒟M, 𝒟R+𝒟Style, 𝒟R+𝒟M+masks and 𝒟R+𝒟Style+masks datasets, where “+ masks” denotes masking out loss generated by synthetic humans. All reported results are on the held out validation set.
Model Head Shoulder Elbow Wrist Hip Knee Ankle mAP
𝒟R+𝒟M 91.3 89.5 80.7 71.7 75.4 72.5 67.7 78.4
𝒟R+𝒟M+masks 92.3 90.9 80.5 72.2 76.0 71.7 68.3 78.9
𝒟R+𝒟Style 91.8 89.8 80.4 70.9 75.5 71.6 67.9 78.3
𝒟R+𝒟Style+masks 91.6 90.6 80.8 71.8 77.7 72.2 68.8 79.1

In an ablation study, we test whether training on synthetic humans actually improves the mAP or if the improvement when training with 𝒟M or 𝒟Style is mostly caused by additional occlusion. For that purpose, we mask out all the loss that is generated by synthetic humans. Comparing 𝒟R+𝒟M+masks with 𝒟R+𝒟M, it can be seen that masking out the loss generated by synthetic humans increases the accuracy. Therefore, the domain gap between synthetic and real humans limits the generalization, and the improvement of 𝒟R+𝒟M over 𝒟R is due to more occlusion. Stylization in combination with masks leads to the best model (see Fig. 1 and Supp. Mat. for qualitative results), suggesting that a smaller gap between synthetic occluder and real parts of the image improves generalization.

5.3 Does Informed Sampling Improve Results?

Finally, we test whether the teacher network can help to use the synthetic data more effectively. For that purpose we use the adversarial teacher with 𝒟R+𝒟S. The results can be seen in Tab. 3. Grouping according to the camera pitch leads to an additional improvement of 0.5 mAP in comparison to 𝒟R+𝒟S. See Fig. 1 and Supp. Mat. for qualitative results. For usage of the teacher in combination with the mixed and stylized datasets we do not observe further improvements. Since we only consider real humans as samples some groups are very small. We assume that oversampling of these groups inhibits improvements.

Table 3: Comparison of the 𝒟R and 𝒟R+𝒟S baselines (copied from Tab. 1) to the student-teacher model using different groupings. Results on the validation set.

Grouping Head Shoulder Elbow Wrist Hip Knee Ankle mAP
𝒟R 91.3 89.1 79.2 70.4 75.9 71.5 66.7 77.7
𝒟R+𝒟S 91.1 89.2 80.5 71.0 75.2 73.6 68.1 78.4
“adversarial Teacher” C 91.7 90.0 80.9 71.2 77.1 73.6 67.7 78.9
“adversarial Teacher” mD 91.5 90.4 80.5 72.2 75.8 73.1 67.6 78.7

As can be seen in Fig. 6, improvements for highly occluded people are strongest for models trained with the teacher. Clear differences to 𝒟R+𝒟M+masks and 𝒟R+𝒟Style+masks can only be seen for the highest occlusion level. For all but the lowest occlusion levels 𝒟R is outperformed by all other models. Thus, our methods of training improve accuracy for difficult high occlusion cases.

Figure 6: Detection performance for varying ratio of visible joints (mAP). The validation data is grouped into 5 linearly spaced groups in range [0,1]. The groups contain 14, 111, 289, 286, 180 persons respectively. The teacher methods are hard to distinguish because of similar values.

Sampling Probabilities. The teacher often assigns high probability to few groups early in training. In most cases the teacher converges to a uniform sampling strategy over the groups, as training progresses. This is not equivalent to random sampling, as samples are not uniformly distributed across groups. As a result the training data follows a uniform distribution for the respective image characteristic. A uniform distribution over an image characteristic like camera pitch is more extreme than the distribution in the real training data. Training on more extreme samples seems to improve the generalization. We find that improvements are largest for the camera pitch grouping. This might be due to the bias to small camera pitch in the real training data.

6 Conclusion

In summary, we created multiple synthetic datasets and analyze their effectiveness. We show that training with synthetic data improves multi-person pose estimation methods. We find that both our methods of generating synthetic datasets perform on par. Surprisingly, our approach for improving visual appearance of synthetic humans decreased the accuracy. More elaborate domain adaptation methods might provide better results. For example, GAN-based approaches ensure that no obvious artifacts can be used to distinguish real from synthetic. We find that improvements of the 𝒟R+𝒟M and 𝒟R+𝒟Style can be explained by more occlusion, as mAP increases when masking out the loss of synthetic humans. Here the stylization leads to further improvements, suggesting that visual quality of occluding objects is important. Finally, we find that training on the most difficult synthetic samples at each point of training improves the results. This suggest that, for large synthetic datasets, random sampling is not optimal and better strategies exist. More research in this direction is necessary to draw final conclusions. 𝒟R+𝒟Style+masks outperforms the “adversarial Teacher” C model. We assume that the potential of the teacher is limited by the small amount and the quality of human textures. Given better textures we expect this approach to outperform 𝒟R+𝒟Style+masks.

Limitations. The teacher network is limited in multiple ways. First, the current implementation requires grouping of data. The size and spacing of groups might have a large influence on the training success and applicability of teacher networks. Furthermore, the grouping is based on one feature only. This is suboptimal, since difficulty of an image is determined by multiple factors. We plan to extend the teachers to handle multiple characteristics at once. A more elaborate formulation that does not require grouping might be superior. Last, the teacher can be applied to other tasks and networks, here we evaluate it only for multi-person pose estimation with the OpenPose network.

The style transfer occasionally produces artifacts in the stylized image. Especially the skin color of the synthetic humans might be unnatural. In rare cases, large parts of the background are included in the human mask. These failures in segmentation can lead to ghost-like synthetic humans (see Supp. Mat.).

Acknowledgement. S. Tang acknowledges funding by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) Projektnummer 276693517 SFB 1233.
Disclosure. MJB has received research gift funds from Intel, Nvidia, Adobe, Facebook, and Amazon. While MJB is a part-time employee of Amazon, his research was performed solely at, and funded solely by, MPI. MJB has financial interests in Amazon and Meshcapade GmbH.


  • [1] W. Abdulla (2017) Mask r-cnn for object detection and instance segmentation on keras and tensorflow. Github. Note: \urlhttps://github.com/matterport/Mask_RCNN Cited by: §3.2, §3.4.
  • [2] H. A. Alhaija, S. K. Mustikovela, L. Mescheder, A. Geiger, and C. Rother (2018) Augmented reality meets computer vision: efficient data generation for urban driving scenes. IJCV 126 (9), pp. 961–972. Cited by: §1.
  • [3] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele (2014-06) 2D human pose estimation: new benchmark and state of the art analysis. In CVPR, Cited by: §1, §2, §3.3, §5.
  • [4] S. Bak, P. Carr, and J. Lalonde (2018) Domain adaptation through synthesis for unsupervised person re-identification. In ECCV, Cited by: §1, §2, §2.
  • [5] I. B. Barbosa, M. Cristani, B. Caputo, A. Rognhaugen, and T. Theoharis (2018) Looking beyond appearances: synthetic training data for deep cnns in re-identification. CVIU 167, pp. 50–62. Cited by: §1, §2.
  • [6] Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009) Curriculum learning. In ICML, Cited by: §2.
  • [7] U. Büchler, B. Brattoli, and B. Ommer (2018) Improving spatiotemporal self-supervision by deep reinforcement learning. In ECCV, Cited by: §2.
  • [8] Z. Cao, T. Simon, S. Wei, and Y. Sheikh (2017) Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, Cited by: Figure 1, §2, §4.
  • [9] W. Chen, H. Wang, Y. Li, H. Su, Z. Wang, C. Tu, D. Lischinski, D. Cohen-Or, and B. Chen (2016) Synthesizing training images for boosting human 3d pose estimation. In 3DV, Cited by: §2.
  • [10] A. Dundar, M. Liu, T. Wang, J. Zedlewski, and J. Kautz (2018) Domain stylization: a strong, simple baseline for synthetic to real image domain adaptation. arXiv preprint arXiv:1807.09384. Cited by: §1, §2, §3.4, §3.4.
  • [11] N. Dvornik, J. Mairal, and C. Schmid (2018) On the importance of visual context for data augmentation in scene understanding. arXiv preprint arXiv:1809.02492. Cited by: §1.
  • [12] M. Fabbri, F. Lanzi, S. Calderara, A. Palazzi, R. Vezzani, and R. Cucchiara (2018) Learning to detect and track visible and occluded body joints in a virtual world. In ECCV, Cited by: §1, §2.
  • [13] Y. Fan, F. Tian, T. Qin, J. Bian, and T. Liu (2017) Learning what data to learn. arXiv preprint arXiv:1702.08635. Cited by: §2.
  • [14] H. Fang, S. Xie, Y. Tai, and C. Lu (2017) Rmpe: regional multi-person pose estimation. In ICCV, Cited by: §2.
  • [15] M. Fieraru, A. Khoreva, L. Pishchulin, and B. Schiele (2018) Learning to refine human pose estimation. In CVPR Workshops, Cited by: §2.
  • [16] M. F. Ghezelghieh, R. Kasturi, and S. Sarkar (2016) Learning camera viewpoint using cnn to improve 3d body pose estimation. In 3DV, Cited by: §2.
  • [17] K. He, G. Gkioxari, P. Dollár, and R. Girshick (2017) Mask r-cnn. In ICCV, Cited by: §3.2, §3.4.
  • [18] S. Hinterstoisser, V. Lepetit, P. Wohlhart, and K. Konolige (2018) On pre-trained image features and synthetic images for deep learning. In ECCV, Cited by: §4.
  • [19] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, and B. Schiele (2016) Deepercut: a deeper, stronger, and faster multi-person pose estimation model. In European Conference on Computer Vision, pp. 34–50. Cited by: §2.
  • [20] L. Jiang, Z. Zhou, T. Leung, L. Li, and L. Fei-Fei (2017) Mentornet: learning data-driven curriculum for very deep neural networks on corrupted labels. arXiv preprint arXiv:1712.05055. Cited by: §2.
  • [21] A. Katharopoulos and F. Fleuret (2017) Biased importance sampling for deep neural network training. arXiv preprint arXiv:1706.00043. Cited by: §2.
  • [22] T. Kim and J. Choi (2018) ScreenerNet: learning self-paced curriculum for deep neural networks. arXiv preprint arXiv:1801.00904. Cited by: §2.
  • [23] M. Kocabas, S. Karagoz, and E. Akbas (2018) Multiposenet: fast multi-person pose estimation using pose residual network. In ECCV, Cited by: §2.
  • [24] M. P. Kumar, B. Packer, and D. Koller (2010) Self-paced learning for latent variable models. In NIPS, Cited by: §2.
  • [25] Y. Li, M. Liu, X. Li, M. Yang, and J. Kautz (2018) A closed-form solution to photorealistic image stylization. In ECCV, Cited by: §2, §3.4.
  • [26] M. Loper, N. Mahmood, and M. J. Black (2014) MoSh: motion and shape capture from sparse markers. TOG 33 (6), pp. 220. Cited by: §3.1.
  • [27] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black (2015) SMPL: a skinned multi-person linear model. TOG 34 (6), pp. 248. Cited by: §2.
  • [28] Y. Luo, Z. Xu, P. Liu, Y. Du, and J. Guo (2019) Multi-person pose estimation via multi-layer fractal network and joints kinship pattern. TIP 28 (1), pp. 142–155. Cited by: §2.
  • [29] J. Marin, D. Vázquez, D. Gerónimo, and A. M. López (2010) Learning appearance in virtual scenarios for pedestrian detection. In CVPR, Cited by: §1, §2.
  • [30] M. Müller, V. Casser, J. Lahoud, N. Smith, and B. Ghanem (2018) Sim4CV: a photo-realistic simulator for computer vision applications. IJCV, pp. 1–18. Cited by: §2.
  • [31] Z. Murez, S. Kolouri, D. Kriegman, R. Ramamoorthi, and K. Kim (2018) Image to image translation for domain adaptation. In CVPR, Cited by: §2.
  • [32] A. Newell, Z. Huang, and J. Deng (2017) Associative embedding: end-to-end learning for joint detection and grouping. In NIPS, Cited by: §2.
  • [33] X. Nie, J. Feng, J. Xing, and S. Yan (2018) Pose partition networks for multi-person pose estimation. In ECCV, Cited by: §2.
  • [34] X. Peng, Z. Tang, F. Yang, R. S. Feris, and D. Metaxas (2018) Jointly optimize data augmentation and network training: adversarial data augmentation in human pose estimation. In CVPR, Cited by: §2, §4.1.
  • [35] L. Pishchulin, A. Jain, C. Wojek, M. Andriluka, T. Thormählen, and B. Schiele (2011) Learning people detection models from few training samples. In CVPR, Cited by: §1, §2.
  • [36] A. Ranjan, J. Romero, and M. J. Black (2018) Learning human optical flow. In BMVC, Cited by: §1, §2, §3.1, §3.1, §3.2.
  • [37] G. Rogez and C. Schmid (2018) Image-based synthesis for deep 3d human pose estimation. IJCV 126 (9), pp. 993–1008. Cited by: §2.
  • [38] G. Rogez, P. Weinzaepfel, and C. Schmid (2019) Lcr-net++: multi-person 2d and 3d pose detection in natural images. TPAMI. Cited by: §1, §2.
  • [39] J. Romero, D. Tzionas, and M. J. Black (2017-11) Embodied hands: modeling and capturing hands and bodies together. ACM TOG, (Proc. SIGGRAPH Asia) 36 (6). Cited by: §3.1.
  • [40] I. Sárándi, T. Linder, K. O. Arras, and B. Leibe (2018) How robust is 3d human pose estimation to occlusion?. arXiv preprint arXiv:1808.09316. Cited by: §1, §2, §3.3.
  • [41] A. Shrivastava, A. Gupta, and R. Girshick (2016) Training region-based object detectors with online hard example mining. In CVPR, Cited by: §2.
  • [42] A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb (2017) Learning from simulated and unsupervised images through adversarial training. In CVPR, Cited by: §2.
  • [43] S. Tripathi, S. Chandra, A. Agrawal, A. Tyagi, J. M. Rehg, and V. Chari (2019) Learning to generate synthetic data via compositing. arXiv preprint arXiv:1904.05475. Cited by: §2.
  • [44] G. Varol, J. Romero, X. Martin, N. Mahmood, M. J. Black, I. Laptev, and C. Schmid (2017) Learning from synthetic humans. In CVPR, Cited by: §1, §2, §3.1, §3.1, §3.2.
  • [45] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba (2010) Sun database: large-scale scene recognition from abbey to zoo. In CVPR, Cited by: §3.2, §3.3.