Existing observation instruments to measure differentiated instruction often lack insight into the degree to which teachers’ decisions match the actual needs of their students, and neglect the importance of preparation and evaluation. This article describes the psychometric evaluation of a comprehensive instrument (Assessing Differentiation in All Phases of Teaching; ADAPT) that does not suffer from these shortcomings. To assess its quality, 41 raters used it to score videos of lessons and interviews of 86 primary school teachers. A 5-dimensional item-response model showed good fit and high internal consistency, and a decision study was conducted to determine the reliability and agreement coefficients for different numbers of raters. For the intended low-stakes use, a single rater would be enough to provide a reliable estimate of a teacher’s overall score. Finally, rater experiences showed that ADAPT has high practical value due to the comprehensive manual and detailed score descriptions and examples. The instrument can therefore not only be used for research purposes, but can also serve as a valuable resource for teachers and teacher educators in practice.