Abstract
When training a predictive model over medical data, the goal is sometimes togain insights about a certain disease. In such cases, it is common to usefeature importance as a tool to highlight significant factors contributing tothat disease. As there are many existing methods for computing featureimportance scores, understanding their relative merits is not trivial. Further,the diversity of scenarios in which they are used lead to differentexpectations from the feature importance scores. While it is common to make thedistinction between local scores that focus on individual predictions andglobal scores that look at the contribution of a feature to the model, anotherimportant division distinguishes model scenarios, in which the goal is tounderstand predictions of a given model from natural scenarios, in which thegoal is to understand a phenomenon such as a disease. We develop a set ofaxioms that represent the properties expected from a feature importancefunction in the natural scenario and prove that there exists only one functionthat satisfies all of them, the Marginal Contribution Feature Importance (MCI).We analyze this function for its theoretical and empirical properties andcompare it to other feature importance scores. While our focus is the naturalscenario, we suggest that our axiomatic approach could be carried out in otherscenarios too.