In many risk-aware and multi-objective reinforcement learning settings, theutility of the user is derived from a single execution of a policy. In thesesettings, making decisions based on the average future returns is not suitable.For example, in a medical setting a patient may only have one opportunity totreat their illness. Making decisions using just the expected future returns --known in reinforcement learning as the value -- cannot account for thepotential range of adverse or positive outcomes a decision may have. Therefore,we should use the distribution over expected future returns differently torepresent the critical information that the agent requires at decision time bytaking both the future and accrued returns into consideration. In this paper,we propose two novel Monte Carlo tree search algorithms. Firstly, we present aMonte Carlo tree search algorithm that can compute policies for nonlinearutility functions (NLU-MCTS) by optimising the utility of the differentpossible returns attainable from individual policy executions, resulting ingood policies for both risk-aware and multi-objective settings. Secondly, wepropose a distributional Monte Carlo tree search algorithm (DMCTS) whichextends NLU-MCTS. DMCTS computes an approximate posterior distribution over theutility of the returns, and utilises Thompson sampling during planning tocompute policies in risk-aware and multi-objective settings. Both algorithmsoutperform the state-of-the-art in multi-objective reinforcement learning forthe expected utility of the returns.