With the growing prevalence of psychological interventions, it is vital tohave measures which rate the effectiveness of psychological care, in order toassist in training, supervision, and quality assurance of services.Traditionally, quality assessment is addressed by human raters who evaluaterecorded sessions along specific dimensions, often codified through constructsrelevant to the approach and domain. This is however a cost-prohibitive andtime-consuming method which leads to poor feasibility and limited use inreal-world settings. To facilitate this process, we have developed an automatedcompetency rating tool able to process the raw recorded audio of a session,analyzing who spoke when, what they said, and how the health professional usedlanguage to provide therapy. Focusing on a use case of a specific type ofpsychotherapy called Motivational Interviewing, our system gives comprehensivefeedback to the therapist, including information about the dynamics of thesession (e.g., therapist's vs. client's talking time), low-level psychologicallanguage descriptors (e.g., type of questions asked), as well as otherhigh-level behavioral constructs (e.g., the extent to which the therapistunderstands the clients' perspective). We describe our platform and itsperformance, using a dataset of more than 5,000 recordings drawn from itsdeployment in a real-world clinical setting used to assist training of newtherapists. We are confident that a widespread use of automated psychotherapyrating tools in the near future will augment experts' capabilities by providingan avenue for more effective training and skill improvement and will eventuallylead to more positive clinical outcomes.