Post-secondary education ideally teaches students both content knowledge (e.g., what feldspar is) and procedural knowledge (e.g., how to identify feldspar). While the former is traditionally assessed through exams, projects, etc., the latter is harder to measure. One such generally under-assessed but valuable type of procedural knowledge is the ability to solve authentic real-world problems. Interdisciplinary STEM education research has identified a series of common decisions made by skilled practitioners (or "experts"). We have applied this decision-making framework to earth science by creating an assessment themed around natural hazards, in which participants decide which section of road to prioritize for reinforcement in Seattle, WA. This assessment was pilot tested through iterations of think-aloud interviews and revisions. We collected data about how well the assessment performed by having 14 experts and a total of 143 students (or "novices") in three introductory- and intermediate-level university earth science courses take the assessment through an online survey. Quantitative and qualitatively coded results indicated differences between expert and novice decision-making practices. Future iterations of this assessment will convert free-response sections into more easily graded multiple-choice versions, allowing instructors to more efficiently assess student decision-making skill development.