The aim of this study is to contribute to the body of knowledge on the use of contextual mathematical problems. Word problems are a predominant genre in mathematics classrooms in assessing students’ ability to solve problems from everyday life. Research on word problems, however, reveals a range of difficulties in their use in mathematics education. In our research we took an alternative approach: we designed image-rich numeracy problems as alternatives for word problems. A set of word problems was modified by systematically replacing the descriptive representation of the problem situation by a more depictive representation and an instrument was designed to measure the effect of this modification on students’ performance. The instrument can measure the effect of this alternative approach in a randomized controlled trial. In order to use the instrument at scale, we made this instrument also usable as a diagnostic test for an upcoming nationwide examination on numeracy. In this article we explain and discuss the design of the instrument and the validation of its intended uses.