Many students don't know how learning happens, nor what they have to do to make it happen. This awareness of how to learn falls under the purview of metacognition and in a process called self-regulation. Domain-general work investigating student self-regulated learning processes has suggested that improving students' knowledge of how they learn can compensate for low initial ability in a discipline. As a result, we sought to help students reflect on their learning and deconstruct mistakes to better identify opportunities to enhance learning in an introductory geoscience course. Towards this aim we developed the Confidence-based Learning Accuracy Support System (CLASS), a web-based tool that allows instructors to measure students' metacognitive awareness via confidence judgements on content-based assessment questions. CLASS is grounded in the findings of educational psychology research and the self-regulated learning theoretical framework and has potential benefits for both students and instructors. Essentially, students take an online quiz and estimate their confidence in each of their responses. CLASS then determines their score, but also communicates results in relation to student confidence and calculates calibration and bias scores - two empirically-derived measures of the gap between student confidence and performance. CLASS quizzing was implemented in a large-enrollment introductory physical geology course as a formative assessment tool providing both students and instructors with a wealth of information regarding the course's teaching and learning processes (e.g., which topics students are over-confident or under-confident about) that could not have been ascertained from traditional assessment techniques.