We investigate properties of goodness-of-fit tests based on the Kernel SteinDiscrepancy (KSD). We introduce a strategy to construct a test, called KSDAgg,which aggregates multiple tests with different kernels. KSDAgg avoids splittingthe data to perform kernel selection (which leads to a loss in test power), andrather maximises the test power over a collection of kernels. We providetheoretical guarantees on the power of KSDAgg: we show it achieves the smallestuniform separation rate of the collection, up to a logarithmic term. KSDAgg canbe computed exactly in practice as it relies either on a parametric bootstrapor on a wild bootstrap to estimate the quantiles and the level corrections. Inparticular, for the crucial choice of bandwidth of a fixed kernel, it avoidsresorting to arbitrary heuristics (such as median or standard deviation) or todata splitting. We find on both synthetic and real-world data that KSDAggoutperforms other state-of-the-art adaptive KSD-based goodness-of-fit testingprocedures.