In order to collaborate safely and efficiently, robots need to anticipate howtheir human partners will behave. Some of today's robots model humans as ifthey were also robots, and assume users are always optimal. Other robotsaccount for human limitations, and relax this assumption so that the human isnoisily rational. Both of these models make sense when the human receivesdeterministic rewards: i.e., gaining either $100 or $130 with certainty. But inreal world scenarios, rewards are rarely deterministic. Instead, we must makechoices subject to risk and uncertainty--and in these settings, humans exhibita cognitive bias towards suboptimal behavior. For example, when decidingbetween gaining $100 with certainty or $130 only 80% of the time, people tendto make the risk-averse choice--even though it leads to a lower expected gain!In this paper, we adopt a well-known Risk-Aware human model from behavioraleconomics called Cumulative Prospect Theory and enable robots to leverage thismodel during human-robot interaction (HRI). In our user studies, we offersupporting evidence that the Risk-Aware model more accurately predictssuboptimal human behavior. We find that this increased modeling accuracyresults in safer and more efficient human-robot collaboration. Overall, weextend existing rational human models so that collaborative robots cananticipate and plan around suboptimal human behavior during HRI.