Abstract
The widely used 'Counterfactual' definition of Causal Effects was derived forunbiasedness and accuracy - and not generalizability. We propose aCombinatorial definition for the External Validity (EV) of interventioneffects. We first define the concept of an effect observation 'background'. Wethen formulate conditions for effect generalization based on their sets of(observed and unobserved) backgrounds. This reveals two limits for effectgeneralization: (1) when effects are observed under all their enumerablebackgrounds, or, (2) when backgrounds have become sufficiently randomized. Weuse the resulting combinatorial framework to re-examine several issues in theoriginal counterfactual formulation: out-of-sample validity, concurrentestimation of multiple effects, bias-variance tradeoffs, statistical power, andconnections to current predictive and explaining techniques. Methodologically, the definitions also allow us to replace the parametricestimation problems that followed the counterfactual definition bycombinatorial enumeration and randomization problems in non-experimentalsamples. We use this non-parametric framework to demonstrate (ExternalValidity, Unconfoundness and Precision) tradeoffs in the performance of popularsupervised, explaining, and causal-effect estimators. We also illustrate howthe approach allows for the use of supervised and explaining methods innon-i.i.d. samples. The COVID19 pandemic highlighted the need for learningsolutions to provide predictions in severally incomplete samples. Wedemonstrate applications in this pressing problem.