Adversarial thinking is essential when dealing with cyber incidents and for finding security vulnerabilities. Capture the Flag (CTF) competitions are used all around the world to stimulate adversarial thinking. Jeopardy-style CTFs, given their challenge-and-answer based nature, are used more and more in cybersecurity education as a fun and engaging way to inspire students.
Just like traditional written exams, Jeopardy-style CTFs can be used as summative assessment. Did a student provide the correct answer, yes or no. Did the participant in the CTF competition solve the challenge, yes or no. This research project provides a framework for measuring the learning outcomes of a Jeopardy-style CTF and applies this framework to two CTF events as case studies. During these case studies, participants were tested on their knowledge and skills in the field of cybersecurity and queried on their attitude towards CTF education. Results show that the main difference between traditional written exam and a Jeopardy-style CTF is the way in which questions a re formulated. CTF education is stated to be challenging and fun because questions are formulated as puzzles that need to be solved in a gamified and competitive environment. Just like traditional written exams, no additional insight into why the participant thinks the correct answer is the correct answer has been observed or if the participant really did learn anything new by participating. Given that the main difference between a traditional written exam and a Jeopardy-style CTF is the way in which questions are formulated, learning outcomes can be measured in the same way. We can ask ourselves how many participants solved which challenge and to which measurable statements about “knowledge, skill and attitude” in the field of cybersecurity each challenge is related. However, when mapping the descriptions of the quiz-questions and challenges from the two CTF events as case studies to the NICE framework on Knowledge, Skills and Abilities in cybersecurity, the NICE framework did not provide us with detailed measurable statements that could be used in education. Where the descriptions of the quiz-questions and challenges were specific, the learning outcomes of the
NICE framework are only formulated in a quite general matter. Finally, some evidence for Csíkszentmihályi’s theory of Flow has been observed. Following the theory of Flow, a person can become fully immersed in performing a task, also known as “being in the zone” if the “challenge level” of the task is in line with the person’s “skill level”. The persons mental state towards a task will be different depending on the challenge level of the task and required skill level for completing it. Results show that participants state that some challenges were difficult and fun, where other challenges were easy and boring. As a result of this9 project, a guide / checklist is provided for those intending to use CTF in education.