In the last decade, deep learning (DL) has outperformed model-based andstatistical approaches in predicting the remaining useful life (RUL) ofmachinery in the context of condition-based maintenance. One of the majordrawbacks of DL is that it heavily depends on a large amount of labeled data,which are typically expensive and time-consuming to obtain, especially inindustrial applications. Scarce training data lead to uncertain estimates ofthe model's parameters, which in turn result in poor prognostic performance.Quantifying this parameter uncertainty is important in order to determine howreliable the prediction is. Traditional DL techniques such as neural networksare incapable of capturing the uncertainty in the training data, thus they areoverconfident about their estimates. On the contrary, Bayesian deep learninghas recently emerged as a promising solution to account for uncertainty in thetraining process, achieving state-of-the-art performance in many classificationand regression tasks. In this work Bayesian DL techniques such as Bayesiandense neural networks and Bayesian convolutional neural networks are applied toRUL estimation and compared to their frequentist counterparts from theliterature. The effectiveness of the proposed models is verified on the popularC-MAPSS dataset. Furthermore, parameter uncertainty is quantified and used togain additional insight into the data.
Quick Read (beta)
\pdfstringdef\[email protected]A Comparative Study between Bayesian and Frequentist Neural Networks for Remaining Useful Life Estimation in Condition-Based Maintenance\[email protected]title.0\[email protected]A Comparative Study between Bayesian and Frequentist Neural Networks
for Remaining Useful Life Estimation in Condition-Based Maintenance✩
In the last decade, deep learning (DL) has outperformed model-based and statistical approaches in predicting the remaining useful life (RUL) of machinery in the context of condition-based maintenance. One of the major drawbacks of DL is that it heavily depends on a large amount of labeled data, which are typically expensive and time-consuming to obtain, especially in industrial applications. Scarce training data lead to uncertain estimates of the model’s parameters, which in turn result in poor prognostic performance. Quantifying this parameter uncertainty is important in order to determine how reliable the prediction is. Traditional DL techniques such as neural networks are incapable of capturing the uncertainty in the training data, thus they are overconfident about their estimates. On the contrary, Bayesian deep learning has recently emerged as a promising solution to account for uncertainty in the training process, achieving state-of-the-art performance in many classification and regression tasks. In this work Bayesian DL techniques such as Bayesian dense neural networks and Bayesian convolutional neural networks are applied to RUL estimation and compared to their frequentist counterparts from the literature. The effectiveness of the proposed models is verified on the popular C-MAPSS dataset. Furthermore, parameter uncertainty is quantified and used to gain additional insight into the data.
keywords:Bayesian deep learning, C-MAPSS, Condition-based maintenance, Neural network, Remaining useful life, Uncertainty
, \pretocmd\[email protected] \apptocmd\[email protected] \newdateformatmydate\monthname \THEDAY, 2019 \crefformatsection#2Section #1#3 \crefformatsubsection #2Section #1#3 \crefformatsubsubsection#2Section #1#3 \crefformatequation#2Eq. #1#3 \crefformatfigure#2Fig. #1#3 \crefformatsubfigure#2Fig. \StrMid#111.\StrMid#145#3 \crefformattable#2Table #1#3
Condition-based maintenance (CBM) is a maintenance strategy that optimally plans maintenance operations based on real-time monitored health condition of machinery. In the recent years, it has become more and more popular thanks to its effectiveness in reducing unnecessary interventions and improving the reliability of machinery Lei et al. (2018). One of the most critical tasks of CBM is the estimation of the remaining useful life (RUL) of physical systems, i.e. \textquotethe available time before a failure occurs within a component or sub-component Ellefsen et al. (2019).
Several methods exist to predict the RUL value. Traditionally, RUL prediction was accomplished through model-based and statistical approaches Jardine et al. (2006). While the former describe the degradation process through mathematical models of the underlying failure mechanism, the latter fit a statistical model to available failure data, without relying on any physical principles. Recently, AI-based techniques have gained in popularity thanks to their ability to learn degradation patterns directly from observations Lei et al. (2018). In particular, deep learning (DL) has established itself as a powerful method to extract information from the data without any prior knowledge about the underlying phenomenon, at the cost of an increased computational power demand. DL techniques, such as neural networks (NNs) and their multiple variations, e.g. dense neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory (LSTM) networks and convolutional neural networks (CNNs), have been successfully applied to RUL prediction, outperforming traditional prognostic algorithms in simulated turbofan engine degradation data benchmarks Sateesh Babu et al. (2016); Zheng et al. (2017); Gugulothu et al. (2017); Li et al. (2018).
One of the major drawbacks of DL is that it heavily depends on the availability of a large quantity of labeled data Roh et al. (2018), which are typically expensive and time-consuming to obtain, especially in industrial applications. The scarcity of data leads to uncertain estimates of the model’s parameters, which in turn result in poor prognostic performance. This consideration suggests the need for a mechanism to quantify the uncertainty inherent in the training data, in order to determine how confident the output prediction is. Uncertainty modeling requires switching from a frequentist to a Bayesian perspective, i.e. assuming that the model parameters are random variables, whose prior distribution is updated according to Bayes’ rule after encountering the data, to yield a posterior distribution Wagenmakers et al. (2008). While this is computationally affordable in model-based and statistical approaches, it is prohibitive in DL due to the large amount of parameters to update. Only recently, thanks to breakthroughs in the field of Bayesian deep learning, variational inference techniques have been proposed to approximate the intractable posterior distribution, making it possible to train Bayesian neural networks (BNNs) Blundell et al. (2015); Gal and Ghahramani (2016); Fortunato et al. (2017); Shridhar et al. (2019). BNNs have been widely employed in computer vision, achieving state-of-the-art performance in many tasks Kendall and Gal (2017), but, to the best of the author’s knowledge, they have not been applied to RUL prediction yet.
The goal of this work is to test the effectiveness and investigate the advantages of BNNs in RUL estimation. In particular, the focus is on Bayesian dense neural networks (BDNNs) and Bayesian convolutional neural networks (BCNNs), and their performance is assessed on simulated run-to-failure turbofan engine degradation data.
The main contributions of this study are as follows:
Bayesian DL techniques are successfully applied to the field of CBM, achieving competitive performance compared to the state-of-the-art.
It is shown how the uncertainty provided by Bayesian deep models is useful to gain additional insight into the dataset for what concerns data quantity.
The source code11 1 See https://github.com/luca310795/bayesian-deep-rul is released to promote further research in the field.
The remainder of this paper is structured as follows. In creftype 2 recent and related work on DL for RUL prediction is summarized . In creftype 3 an overview of the theoretical foundations of BNNs is provided and the notion of uncertainty is formally defined. In creftype 4 the regression models used in the experiments are introduced. In creftype 5 the experimental setup is presented and the results are discussed. Finally, in creftype 6 the conclusions are drawn and future work directions are outlined.
2 Related work
In this section recent DL approaches to estimate RUL from run-to-failure data are reviewed. The selected studies mainly focus on improving the predictive performance by means of more and more complex NN architectures. Additionally, two works that address the problem of data quantity in DL are mentioned.
In the last decade, considerable effort has been invested to advance the state-of-the-art of RUL prediction. In Tian (2009), Tian developed a DNN model to predict the RUL of a group of pumps based on vibration data collected from bearings. First, each condition monitoring measurement series is fitted by a generalization of the Weibull distribution failure rate function Lai et al. (2006), in order to reduce the noise inherent in the data. Then, the age and the fitted condition monitoring measurement values at the present and previous observations are given as inputs to the network, which returns the life percentage as an output. The closer the life percentage is to , the closer the pump is to its end-of-life. Experimental results showed that the proposed method achieved relatively low average prediction error rates, outperforming other techniques such as the modified version of the method by Wu et al. Wu et al. (2007). A major drawback of DNNs is that they have no built-in mechanism to capture temporal and spatial relationships, thus additional preprocessing (such as fitting the raw measurements using the generalized Weibull distribution failure rate function) is required to extract meaningful information from the data.
The issue of modeling temporal dependencies is solved, in principle, by RNNs. In Gugulothu et al. (2017), Gugulothu et al. proposed a novel RUL estimation approach – Embed-RUL – that uses a sequence-to-sequence RNN-based model to generate embeddings for multivariate time series. These embeddings tend to differ for normal and defective machines, thus they successfully capture the degradation pattern in the time series. Experiments performed on NASA’s International Conference on Prognostics and Health Management (PHM08) prognostic data challenge dataset Saxena and Goebel (2008a) and on a proprietary real-world pump dataset showed that Embed-RUL outperformed the previous state-of-the-art. Further improvements over RNN’s capability of modeling temporal relationships are offered by LSTM networks, which capture not only short-term dependencies but also long-term ones. In Zheng et al. (2017), Zheng et al. proposed a DL architecture with LSTM layers followed by dense layers and a final output neuron. Extensive experiments on NASA’s Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) Saxena and Goebel (2008b); Saxena et al. (2008), PHM08 and a milling dataset Agogino and Goebel (2007) demonstrated that LSTM networks were capable of revealing hidden patterns in the data, significantly outperforming former RUL estimation approaches.
The problem of capturing spatial dependencies among different sensor measurements is addressed by CNNs. In Sateesh Babu et al. (2016), Babu et al. proposed a novel CNN architecture for RUL prediction, which consists of convolutional and average pooling layers followed by a final output neuron. This method improved the accuracy of the RUL predictions on the C-MAPSS dataset compared to a multilayer perceptron, a support vector machine, and a relevance vector machine. More recently, in Li et al. (2018), Li et al. achieved even better performance compared to both the LSTM network in Zheng et al. (2017) and the CNN in Sateesh Babu et al. (2016). They employed a deep CNN architecture with convolutional layers followed by dense layer and a final output neuron. Additionally, they used dropout Srivastava et al. (2014) as a regularization technique and they trained the network with the adaptive learning rate method Adam Kingma and Ba (2015). In both studies, a sliding window approach is adopted to take into account the temporal relationships within the time series, which would be otherwise ignored since no recurrent layer was used.
An attempt to combine the ability of LSTM networks to capture temporal dependencies with the ability of CNNs to extract hierarchical spatial features was made by Jayasinghe et al. in Jayasinghe et al. (2018), with promising results for datasets obtained from complex environments.
All these studies rely on completely labeled run-to-failure training series. However, in real-life scenarios, most data are unlabeled from the beginning and the labeling process is expensive, time-consuming and error-prone. In order to reduce the amount of labeled data needed to train the model, semi-supervised DL techniques based on generative models and restricted Boltzmann machines have recently been applied with success by Yoon et al. Yoon et al. (2017) and Ellefsen et al. Ellefsen et al. (2019), respectively. Differently from them, in this work the focus is not on proposing alternative solutions to the issue of data labeling, but on determining whether enough labeled data were provided to effectively train the model.
3 Theoretical background
In this section the theoretical foundations of BNNs are introduced and the notion of uncertainty is formally defined. For a more rigorous mathematical discussion see Blundell et al. (2015); Shridhar et al. (2019); Kendall and Gal (2017).
3.1 Bayesian neural networks
In standard NNs uncertainty is completely neglected. Weights assume deterministic values and output predictions are point estimates Blundell et al. (2015). On the contrary, in BNNs weights are represented by probability distributions over possible values and output predictions are random variables. If we know the posterior distribution of the weights given the training data, , the predictive distribution of a label of an unseen test sample is calculated as , i.e. the expected prediction under each possible configuration of the weights given , which is equivalent to using an ensemble of an uncountably infinite number of NNs Blundell et al. (2015).
Theoretically, can be obtained through exact Bayesian inference, but practically it is infeasible as the number of weights in an NN is very large and its functional form is not suitable for integration Blundell et al. (2015). A possible solution is to approximate the posterior distribution of the weights through variational inference, a technique that determines the optimal parameters of a surrogate distribution (which is often assumed to be a diagonal Gaussian) that minimizes the Kullback-Leibler (KL) divergence with the true posterior of the weights Blundell et al. (2015):
where is the prior distribution of the weights. The resulting cost function is known as the evidence lower bound (ELBO) or the variational free energy (VFE) and it explicitly represents the trade-off between satisfying the simplicity of the prior (complexity cost) and the complexity of the data (likelihood cost) Blundell et al. (2015). It is denoted as
Exactly minimizing the ELBO is computationally intractable in many cases. Instead, various approximations are adopted. In Blundell et al. (2015), a backpropagation-like algorithm is proposed for Bayesian variational inference in NNs – Bayes by Backprop – which uses Monte Carlo sampling to draw weights from the surrogate distribution and obtain unbiased estimates of gradients of the cost function in creftype 2. The exact ELBO is approximated as Blundell et al. (2015):
where denotes the -th Monte Carlo sample drawn from the surrogate distribution. Gradient-based optimization can be used to minimize the approximated ELBO and learn the optimal parameters of the surrogate distribution of the weights in an NN.
Bayes by Backprop can be applied to CNNs too, in order to approximate the intractable posterior distribution of the weights in each convolutional filter. In Shridhar et al. (2019), Shridhar et al. assume a diagonal Gaussian surrogate distribution, i.e. , where and are the indices of the input and output layers, respectively, and the height and width of any given filter, respectively, and the mean and variance of the distribution, respectively, which need to be learned by the network. In particular, to facilitate the learning process, they define , where is a learnable parameter that intuitively represents the deviation from the mean.
Differently from Blundell et al. (2015), they utilize the local reparameterization trick Kingma et al. (2015) to sample the layer activation instead of the weights for a higher computational efficiency. The mean and variance of each weight are learned independently through two convolutional operations. In the first, the output (which is a function of both the mean and the variance) is interpreted as the mean and optimized towards a single point estimate. In the second, the variance is learned. Since this formulation of the variance includes the mean , only must be learned in the second convolution, hence only one parameter is updated per convolutional operation. This procedure can be applied to dense layers too Shridhar et al. (2019).
Adopting a Bayesian perspective in DL leads to the following advantages:
Regularization is naturally provided, since a prior is placed upon the weights.
Predictive uncertainty can be quantified, as the output value is not a point estimate but a probability distribution.
An infinite ensemble of networks is trained implicitly.
However, these advantages come at a cost: if we assume a diagonal Gaussian surrogate distribution, the number of weights in a BNN is doubled compared to a frequentist NN of the same size Blundell et al. (2015), and training takes longer since at each step two operations are performed instead of one and the expensive Monte Carlo sampling is applied to approximate the ELBO.
3.2 Uncertainty in Bayesian deep learning
In Bayesian modeling there are two main types of uncertainty: epistemic and aleatoric Kendall and Gal (2017). Epistemic uncertainty represents the uncertainty in the model’s parameters. It is important when the data are scarce, since it can be explained away only if enough training samples are provided. Aleatoric uncertainty captures noise inherent in the observations such as sensor or motion noise and cannot be reduced even if more data are available. It is important in large data situations, where epistemic uncertainty vanishes.
In regression tasks, the uncertainty of a Bayesian deep model (e.g. a BNN) with respect to the random output (where is the input sample and is the mapping between input space and output space defined by the model’s weights ) is quantified by the predictive variance, which can be approximated as Kendall and Gal (2017):
with predictions obtained by approximating the predictive mean , where is the -th Monte Carlo sample drawn from the posterior distribution of the weights. The first term in the predictive variance, , corresponds to the aleatoric uncertainty, i.e. the amount of noise inherent in the data; the second is the epistemic uncertainty, which measures how confident the model is about its predictions. This term vanishes when the parameter uncertainty is zero (i.e. all draws assume the same constant value). As mentioned in creftype 1, estimating the RUL in industrial applications is often problematic due to the scarcity of labeled data, thus it is more useful to quantify epistemic rather than aleatoric uncertainty. For this reason, in this work the focus is on the epistemic uncertainty, and the aleatoric one is neglected (i.e. in creftype 4).
4 Regression models
In this section the proposed regression models for RUL prediction are introduced. As mentioned in creftype 1, the goal of this work is to test the effectiveness of BDNNs and BCNNs in RUL estimation and investigate their advantages over standard NNs. For the sake of a fair comparison, three frequentist DL models are considered as a baseline and compared to their Bayesian counterparts, for a total of six models. Frequentist models are trained through backpropagation with mean square error (MSE) as a loss function, Bayesian models through Bayes by Backprop with ELBO (see creftype 3). Prior and surrogate distributions of the weights are initialized as diagonal Gaussians and , respectively, where the learnable parameters and are initialized through Xavier uniform initializer Glorot and Bengio (2010). The input is assumed to be an matrix, where represents the time sequence dimension and the number of features. The procedure to prepare the data in 2D format is discussed in detail in creftype 5.2.3.
The models are as follows:
Bayesian-Dense3 (B-D3): Bayesian counterpart of F-D3, in which standard dense layers are replaced by Bayesian ones (see creftype 55(b)). The softplus activation (whose output is strictly positive) is applied to the final output neuron, in order to ensure a positive non-zero variance Shridhar et al. (2019).
Frequentist-Conv2Pool2 (F-C2P2): CNN architecture proposed in Sateesh Babu et al. (2016) by Babu et al. (see creftype 55(c)). The input matrix is fed to a convolutional layer with channels followed by a average pooling layer. A second convolution with channels is applied, followed by another average pooling. The output is flattened into a 1D vector and a final output neuron returns the predicted RUL value. After each convolutional layer, the sigmoid activation is applied.
Frequentist-Conv5Dense1 (F-C5D1): CNN architecture proposed in Li et al. (2018) by Li et al. (see creftype 55(e)). The input matrix is fed to consecutive convolutional layers with channels followed by a convolution with channel. The output is flattened into a 1D vector and a neurons dense layer is applied, followed by a final output neuron that returns the RUL value. After each convolutional layer, the tanh activation is used. The input is zero-padded before each convolution to keep the dimensions unaltered. Dropout with is applied to the flattened 1D vector to reduce overfitting. Xavier normal initializer Glorot and Bengio (2010) is used to initialize the weights. In Li et al. (2018), an additional tanh is applied to the output of the neurons dense layer; it had to be removed because in the proposed implementation it led to gradient saturation, preventing the network from training properly.
Bayesian-Conv5Dense1 (B-C5D1): Bayesian counterpart of F-C5D1, in which standard convolutional and dense layers are replaced by Bayesian ones (see creftype 55(f)). To ensure a positive non-zero variance, the tanh activations are replaced by sigmoids (whose output is strictly positive) and softplus, which is also applied to the neurons dense layer and to the final output neuron. Sigmoid instead of softplus is used in the first convolutions since a performance improvement was empirically observed. No more than sigmoids are applied since experiments showed that exceeding that threshold leads to gradient saturation. Dropout is removed due to the natural regularization provided by the Bayesian approach.
The proposed models were implemented in PyTorch Paszke et al. (2017), using the open source implementation of Bayesian layers22 2 See https://github.com/kumar-shridhar/PyTorch-BayesianCNN, last accessed on March 14, 2019. provided by Shridhar et al. along with Shridhar et al. (2019), and the data loader33 3 See https://github.com/charlesq34/pointnet2, last accessed on March 14, 2019. provided by Qi et al. along with Qi et al. (2017). Note that, since the current implementation of Bayesian layers does not support bias, in order to enable a fair comparison it is not used in frequentist layers either.
5 Experimental study
In this section the dataset is introduced, the data preprocessing phase is presented in detail, the performance metrics considered in the experiments are defined, the prognostic procedure is described and the experimental results are discussed.
The proposed models are tested on the simulated turbofan engine degradation data of the publicly available C-MAPSS dataset, which includes four subsets of multivariate temporal data obtained from sensors. Each subset contains a training and a test set. The training set includes run-to-failure sensor records of multiple engines collected under different operating conditions and fault modes. At the beginning, the engines are assumed to be healthy but their initial wear and manufacturing variation is unknown. As time progresses, they start to degrade until a system failure occurs. The last data entry corresponds to the time cycle at which the unit is declared defective. On the contrary, the sensor records in the test set end at some point before the failure. The goal is to predict the RUL of each engine in the test set. For verification, the true RUL values of the test engines are provided. Each subset includes columns: engine number, time cycle, operational settings (that determine the operating condition) and sensor measurements. The detailed information of the four subsets, denoted as FD001, FD002, FD003 and FD004, is presented in creftype 1.
5.2 Data preprocessing
A preprocessing phase is required before feeding the data to the model. It consists of four steps: feature selection, normalization, sliding window segmentation and label rectification. The parameters of the preprocessing phase are reported in creftype 1. The resulting specifics of the proposed regression models are reported in creftype 2.
5.2.1 Feature selection
In FD001 and FD003, sensors , , , , , , and exhibit constant measurements throughout the whole engine’s lifetime, thus they do not provide any useful information to predict the RUL Ellefsen et al. (2019). Moreover, FD001 and FD003 are subjected to a single operating condition. For these reasons, only out of sensors are used as the input features, whose indices are , , , , , , , , , , , , and , and the operational settings are discarded. On the contrary, in FD002 and FD004 there are no constant measurements, and the presence of operating conditions makes it harder to detect degradation patterns Ellefsen et al. (2019). Therefore, all sensors and operational settings are used as the input features.
The collected sensor data are normalized to be within the range by means of min-max normalization Patro and Sahu (2015):
where denotes the original -th data point of the -th sensor, the normalized value of , the maximum value of the original measurement data from the -th sensor and the minimum value.
5.2.3 Sliding window segmentation
In multivariate time series-based problems, we can generally extract more information by analyzing the temporal sequence data, rather than just considering the single data points sampled at each time step independently. In order to preserve the degradation patterns hidden in the temporal dimension, the sliding window approach of Li et al. Li et al. (2018) is adopted to divide the time series into overlapping segments of fixed size. Let denote the size of the time window and the number of features (i.e. sensors). At each time step, all the future sensor records within the time window are collected into an matrix (see creftype 1). In particular, given a time series of length , we can extract exactly segments. If , the time series is discarded. The obtained segments are then labeled with the RUL of the last data point in the time window and used as input samples to feed the model.
|Training series 100||260||100||249|
|Test series 100||259||100||248|
|Operating conditions 1||6||1||6|
|Fault modes 1||1||2||2|
|Training samples \[email protected]@numprint17731||\[email protected]@numprint48819||\[email protected]@numprint21820||\[email protected]@numprint57763|
|Test samples 100||259||100||248|
|F-D3 4||\[email protected]@numprint62100||\[email protected]@numprint68100||\[email protected]@numprint62100||\[email protected]@numprint56100|
|B-D3 4||\[email protected]@numprint124200||\[email protected]@numprint136200||\[email protected]@numprint124200||\[email protected]@numprint112200|
|F-C2P2 3||868||\[email protected]@numprint1246||868||\[email protected]@numprint1092|
|B-C2P2 3||\[email protected]@numprint1736||\[email protected]@numprint2492||\[email protected]@numprint1736||\[email protected]@numprint2184|
|F-C5D1 7||\[email protected]@numprint45230||\[email protected]@numprint51230||\[email protected]@numprint45230||\[email protected]@numprint39230|
|B-C5D1 7||\[email protected]@numprint90460||\[email protected]@numprint102460||\[email protected]@numprint90460||\[email protected]@numprint78460|
\[email protected]@numprint124200 \[email protected]@numprint136200\[email protected]@numprint124200\[email protected]@numprint112200868 \[email protected]@numprint1246868\[email protected]@numprint1092\[email protected]@numprint1736 \[email protected]@numprint2492\[email protected]@numprint1736\[email protected]@numprint2184\[email protected]@numprint45230 \[email protected]@numprint51230\[email protected]@numprint45230\[email protected]@numprint39230\[email protected]@numprint90460 \[email protected]@numprint102460\[email protected]@numprint90460\[email protected]@numprint78460
5.2.4 Label rectification
A common approach in the literature Sateesh Babu et al. (2016); Zheng et al. (2017); Li et al. (2018); Heimes (2008) is to adopt a piece-wise linear degradation model, i.e. to assume that the RUL target function is constant until a threshold value, , after which it linearly decreases to . From an implementation perspective, this translates into rectifying the labels, i.e. to artificially set the RUL to for all those samples whose RUL is strictly greater than . The rationale behind it is that a system typically works normally in the early age and starts to degrade only after a certain degree of wear. However, the value of varies across the literature and label rectification is often applied to both the training and the test set (e.g. in Li et al. (2018)), making it difficult to compare the prognostic performance of different methods. Modifying the test labels according to is indeed equivalent to creating an ad hoc version of the test set on which the performance metrics are biased towards over-optimistic estimates. In order to facilitate the comparison with current and future studies, the results both with and without rectified test labels are reported.
5.3 Performance metrics
In this study three metrics were considered for evaluating the performance of the proposed regression models, i.e. root mean square error (RMSE), mean absolute error (MAE) and the scoring function (S) proposed in Saxena et al. (2008) by Saxena et al. and also used in the PHM08 prognostic data challenge dataset Saxena and Goebel (2008a), which are formulated as follows:
where is the total number of test samples and is the error (i.e. the difference) between the estimated RUL value and the true RUL value for the -th test sample. Good prognostic methods should obtain relatively low values on all the three metrics. As shown in creftype 2, the RMSE equally penalizes early and late predictions, while the asymmetric scoring function penalizes late predictions more than early ones. Minimizing S is important because late predictions typically lead to more serious consequences than early predictions, as the maintenance will be scheduled too late. However, in order to correctly evaluate the performance, it is useful to monitor RMSE and MAE too, since S is extremely sensitive to outliers due to the exponentiation.
In addition to the three evaluation metrics, in Bayesian models the epistemic uncertainty (see creftype 3.2), averaged over all the test samples, is considered too.
5.4 Prognostic procedure
The dataset is preprocessed as described in creftype 5.2. Note that the data prepared in 2D format are fed directly to the model, without the need of any prior feature engineering step.
Based on the specific dataset information, the regression model is initialized. For each epoch, the training samples and their corresponding RUL labels are randomly divided into multiple mini-batches of size and fed to the model. The model’s weights are updated according to the backpropagated gradients of the loss function computed batchwise. The Adam Kingma and Ba (2015) optimization algorithm is used to train the model for epochs with a base learning rate of . After epochs, the learning rate is reduced by a factor of for fine-tuning. In Bayesian models, only Monte Carlo sample is drawn from the posterior distribution to compute the loss44 4 Drawing more than sample at training time is too computationally expensive on the hardware used in this work..
Table 3: Hyperparameters of the proposed prognostic procedure. Hyperparameter Value Mini-batch size 512 Max epoch 250 Base learning rate 0.001 Learning rate decay factor 10 Learning rate decay epoch 200 Monte Carlo samples for training 1 Monte Carlo samples for testing 150
The test samples are fed to the trained model to predict the RUL values and calculate the performance metrics. In Bayesian models, Monte Carlo samples are drawn from the posterior distribution to calculate the performance metrics55 5 Since testing is a one-time operation, we can afford to draw many samples to calculate the performance metrics with higher precision..
The hyperparameters of the proposed prognostic procedure are reported in creftype 3.
5.5 Experimental results and discussion
In this section the prognostic performance of the proposed regression models is evaluated and the importance of epistemic uncertainty in assessing the quantity of training data is discussed. The experimental results are averaged over trials to reduce the effect of statistical fluctuations, and mean and standard deviation values are reported. All the experiments were run on a publicly available Google Compute Engine Deep Learning VM instance with vCPUs, GB RAM, NVIDIA Tesla K80 GPU and PyTorch 1.2 + fast.ai 1.0 (CUDA 10.0) framework.
|Subset||Metric||F-D3||B-D3||F-C2P2||B-C2P2||F-C5D1||B-C5D1||Ellefsen et al. (2019)|
32.04 0.5932.330.6131.430.2331.340.2335.290.5729.870.12–14.34 0.3214.100.3328.860.0224.231.0915.200.3614.700.48–33.02 0.6233.640.4932.970.3633.130.5236.211.7131.350.15–
5.5.1 Performance evaluation
As shown in creftype 4, in most of the cases Bayesian models perform better (with respect to RMSE, MAE and score) than their frequentist counterparts. This is probably due to the natural regularization effect provided by the Bayesian approach, which helps to reduce the generalization error. In particular, B-C5D1 achieves the best RMSE in FD001, FD002 and FD004 (as expected, since its frequentist counterpart accomplishes state-of-the-art performance Li et al. (2018)), and in FD002 and FD004 it even improves the rectified state-of-the-art metrics reported in Ellefsen et al. (2019). B-D3, despite its simplicity, achieves the best RMSE in FD003. Since B-D3 has more parameters than B-C5D1 but no convolutional layers, we conclude that, comparing to the other subsets, the temporal dimension of the data in FD003 is less informative, i.e. no clear degradation pattern can be detected. This leads to the conclusion that on FD003 larger dense models are preferable to smaller models that use convolutional layers in the attempt to extract useful information from the run-to-failure series. More generally, the performance improvement deriving from extracting the temporal information (through the convolutional layers in combination with the sliding window segmentation approach, see creftype 5.2.3) is moderate in the other subsets too. This is a factor to take into consideration, especially when the computational power is limited, since the time required for training B-C5D1 is enormously larger than B-D3 (see creftype 4), and large training times make it difficult to properly tune the hyperparameters.
For what concerns the comparison between the four subsets (see creftype 4), a substantial performance drop is observed in FD002 and FD004, which is in line with expectations, since, as mentioned in creftype 5.2.1, it is harder to detect degradation patterns due to the presence of operating conditions. Moreover, the values of epistemic uncertainty decrease inversely with the number of training samples (see creftype 1): the more the training data, the lower the epistemic uncertainty. In particular, the epistemic uncertainty in FD001 and FD003 is considerably higher than FD002 and FD004. Interestingly, if we sort the models by increasing values of epistemic uncertainty, the order is the same in all the subsets (). Investigating the dependency between epistemic uncertainty and network architecture, we notice that the former is more tightly correlated with the number of layers than the number of weights (see creftype 2). The explanation is that weight uncertainty propagates through the layers, thus deep networks require more training data than shallow ones to be equally confident about their predictions. This confirms once again the fact that deep networks are harder to train than shallow ones.
Aside from performance considerations, the real advantage of Bayesian models is their capability of returning a probability distribution (see creftype 5) instead of a point estimate as an output. The additional information provided by the predictive distribution enables us to boost the model in different ways. For example, if we use the predictive mean as the RUL estimate, we can define a threshold for the predictive standard deviation above which the estimate must be rejected because the model is not confident enough about it. Another possibility is to adopt a more conservative prognostic policy by using the -th percentile (with low values of ) as the RUL estimate, in order to reduce the risk of late prediction.
5.5.2 Data quantity
As pointed out in creftype 5.5.1, FD001 and FD003 are characterized by a higher epistemic uncertainty, reasonably due to the scarcity of training data. In order to validate this explanation, the most successful Bayesian models (B-C5D1 on FD001, FD002 and FD004, B-D3 on FD003) were retrained by progressively decreasing the data quantity. The results are reported in creftype 5. A quantity percentage of indicates that only of the training data, chosen at random, were used for training. In line with the theory, a strong correlation exists between epistemic uncertainty, data quantity and performance: the fewer the data, the higher the uncertainty66 6 Except for B-D3, where the epistemic uncertainty saturates at , remaining almost unchanged at due to the shallowness of the model., the worse the performance. Surprisingly, if we halve the amount of training data, the performance degradation is negligible. Similarly, if we use a quarter of the data, the performance degradation is still moderate. A relevant performance drop is observed only when we use of the data. If we analyze these results in relation to the values of epistemic uncertainty, we gain additional insight into the data. In particular, we conclude the following:
The information contained in the data is redundant, since performance degrades only marginally when the number of training samples is decreased. This means that the preprocessing procedure is not as effective as expected, as it fails in extracting important information from the raw data.
Performance on FD001 and FD003 would benefit from enlarging the training set with new run-to-failure series, since epistemic uncertainty is to times higher than FD002 and FD004, which means that the training data are scarce. Data augmentation techniques could help in increasing the size of the training set.
Performance on FD002 and FD004 would not benefit from enlarging the training set with new run-to-failure series, since epistemic uncertainty is already close to zero, which means that the training data are enough.
6 Conclusion and future work
In this work Bayesian deep learning was successfully applied to RUL estimation in the context of condition-based maintenance. The notion of epistemic uncertainty was introduced and it was shown how to quantify it by means of the Bayesian framework. In the experimental study, the proposed regression models were tested on the four subsets of the publicly available C-MAPSS dataset. Results demonstrated that Bayesian models achieve competitive performance compared to their frequentist counterparts, and that epistemic uncertainty is useful to determine whether the data are too scarce. In particular, if we halve the number of training samples, the increase in the epistemic uncertainty is only moderate and the performance degradation negligible. This means that the preprocessing procedure fails in extracting important information from the raw data.
Future work that aims at improving the state-of-the-art of RUL prediction should focus on refining the preprocessing phase. In addition to this, more effort should be spent on tuning the hyperparameters, especially for Bayesian models. In Bayesian models the number of hyperparameters is indeed larger than in frequentist ones (e.g. prior distribution, surrogate distribution, etc.), hence we have more combinations to explore. Another possible research direction that builds directly upon this work consists in applying the Bayesian RNN introduced in Fortunato et al. (2017) by Fortunato et al. to RUL estimation, in order to combine the effectiveness of recurrent architectures in capturing temporal dependencies with the capability of BNNs to measure weight uncertainty. Computational power limitations due to the expensive Monte Carlo sampling could be overcome through sampling-free uncertainty estimation approaches such as the one proposed in Postels et al. (2019) by Postels et al. Finally, since BNNs proved successful especially in classification tasks Blundell et al. (2015); Shridhar et al. (2019), it would be interesting to cast the RUL prediction problem as a classification one (since labels are integer and can be capped at ), so as to harness all the benefits of BNNs. One of the advantages of BNNs for classification is that, differently from regression, aleatoric uncertainty can be easily quantified Kwon et al. (2018), which is useful in large data situations.
- Milling data set. In NASA Ames Prognostics Data Repository, NASA Ames Research Center, Moffett Field, CA. External Links: Cited by: §2.
- Weight uncertainty in neural networks. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, pp. 1613–1622. External Links: Cited by: §1, §3.1, §3.1, §3.1, §3.1, §3, §6.
- Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture. Reliability Engineering & System Safety 183, pp. 240–251. External Links: Cited by: §1, §2, §5.2.1, §5.5.1, Table 4.
- Bayesian recurrent neural networks. CoRR abs/1704.02798. External Links: Cited by: §1, §6.
- Bayesian convolutional neural networks with Bernoulli approximate variational inference. In 4th International Conference on Learning Representations (ICLR) workshop track, External Links: Cited by: §1.
- Understanding the difficulty of training deep feedforward neural networks. Journal of Machine Learning Research - Proceedings Track 9, pp. 249–256. External Links: Cited by: item 5, §4.
- Predicting remaining useful life using time series embeddings based on recurrent neural networks. CoRR abs/1709.01073. External Links: Cited by: §1, §2.
- Recurrent neural networks for remaining useful life estimation. In 2008 International Conference on Prognostics and Health Management, Vol. , pp. 1–6. External Links: Cited by: §5.2.4.
- A review on machinery diagnostics and prognostics implementing condition-based maintenance. Mechanical Systems and Signal Processing 20, pp. 1483–1510. External Links: Cited by: §1.
- Temporal convolutional memory networks for remaining useful life estimation of industrial machinery. ArXiv abs/1810.05644. External Links: Cited by: §2.
- What uncertainties do we need in Bayesian deep learning for computer vision?. CoRR abs/1703.04977. External Links: Cited by: §1, §3.2, §3.2, §3.
- Adam: a method for stochastic optimization. In 3rd International Conference on Learning Representations, External Links: Cited by: §2, item 2.
- Variational dropout and the local reparameterization trick. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS’15, Cambridge, MA, USA, pp. 2575–2583. External Links: Cited by: §3.1.
- Uncertainty quantification using Bayesian neural networks in classification: application to ischemic stroke lesion segmentation. External Links: Cited by: §6.
- Weibull distributions and their applications. In Springer Handbook of Engineering Statistics, pp. 63–78. External Links: Cited by: §2.
- Machinery health prognostics: a systematic review from data acquisition to RUL prediction. Mechanical Systems and Signal Processing 104, pp. 799–836. External Links: Cited by: §1, §1.
- Remaining useful life estimation in prognostics using deep convolution neural networks. Reliability Engineering & System Safety 172, pp. 1–11. External Links: Cited by: §1, §2, item 5, Figure 3, §5.2.3, §5.2.4, §5.4, §5.5.1.
- Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, External Links: Cited by: §4.
- Normalization: a preprocessing stage. CoRR abs/1503.06462. External Links: Cited by: §5.2.2.
- Sampling-free epistemic uncertainty estimation using approximated variance propagation. International Conference on Computer Vision (ICCV). External Links: Cited by: §6.
- PointNet++: deep hierarchical feature learning on point sets in a metric space. CoRR abs/1706.02413. External Links: Cited by: §4.
- A survey on data collection for machine learning: a big data - AI integration perspective. CoRR abs/1811.03402. External Links: Cited by: §1.
- Deep convolutional neural network based regression approach for estimation of remaining useful life. In Database Systems for Advanced Applications, S. B. Navathe, W. Wu, S. Shekhar, X. Du, X. S. Wang, and H. Xiong (Eds.), Cham, pp. 214–228. External Links: Cited by: §1, §2, item 3, §5.2.4.
- PHM08 challenge data set. In NASA Ames Prognostics Data Repository, NASA Ames Research Center, Moffett Field, CA. External Links: Cited by: §2, §5.3.
- Turbofan engine degradation simulation data set. In NASA Ames Prognostics Data Repository, NASA Ames Research Center, Moffett Field, CA. External Links: Cited by: §2.
- Damage propagation modeling for aircraft engine run-to-failure simulation. In 2008 International Conference on Prognostics and Health Management, Vol. , pp. 1–9. External Links: Cited by: §2, §5.3.
- A comprehensive guide to Bayesian convolutional neural network with variational inference. CoRR abs/1901.02731. External Links: Cited by: §1, §3.1, §3.1, §3, item 2, §4, §6.
- Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15, pp. 1929–1958. External Links: Cited by: §2.
- An artificial neural network approach for remaining useful life prediction of equipments subject to condition monitoring. In 2009 8th International Conference on Reliability, Maintainability and Safety, Vol. , pp. 143–148. External Links: Cited by: §2.
- Bayesian versus frequentist inference. In Bayesian Evaluation of Informative Hypotheses, pp. 181–207. External Links: Cited by: §1.
- A neural network integrated decision support system for condition-based optimal predictive maintenance policy. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 37, pp. 226–236. External Links: Cited by: §2.
- Semi-supervised learning with deep generative models for asset failure prediction. CoRR abs/1709.00845. External Links: Cited by: §2.
- Long short-term memory network for remaining useful life estimation. In 2017 IEEE International Conference on Prognostics and Health Management (ICPHM), Vol. , pp. 88–95. External Links: Cited by: §1, §2, §2, §5.2.4.