QLRA Results 2012
The 2013 test has been reduced to 20 questions to make it easier to administer in the classroom. Three of questions were eliminated that were deemed to be either redundant or too easy. The order of the questions was slightly shifted to assess whether the last questions were being "skipped over".
We were interested in exploring whether multiple choice questions were easier or more difficult than having open-ended questions. Wellesley College tests their entering class each year and agreed to using 3 QLRA questions (#1,8,10) on their 2012 entrance exam which is all open ended. The results suggest no clear pattern as 2 of the questions (#1,10) were easier open ended and 1 much harder (#8 which required reading a tax table and thus having multiple choice answers seems to be an advantage).
Computer Multiple Choice (2nd and 3rd columns)
Paper/Pencil Open-Ended (4th and 5th columns)
Item
N
Mean
N
Mean
1
91
0.63
608
0.77
8
91
0.66
608
0.37
10
91
0.76
608
0.89
Interpretation of Scores
Given that this was a pilot project we must take care in interpreting scores. Bowdoin College has been administering a Q-test to all entering students for over a decade and using it for placement advice. Thirteen of the questions (QLRA 13: #2, 7,8,9,13,15,16,17,18,19,21,22,23) on the QLRA exam are identical to questions on Bowdoin's 30 question entrance exam:
Correl
N
Mean QLRA 13 (4th and 5th columns)
STDEV QLRA 13 (6th and 7th columns)
Total
0.959
1,659
6.82
52.5%
3.315
25.5%
Bowdoin
0.913
497
9.02
69.4%
2.72
20.9%
These 13 questions correlate very strongly with overall score on both the QLRA and on the Bowdoin Q-exam as indicated in the first column. At Bowdoin I identify students who score below 50% on our entrance exam as being good candidates for my Math 050: QR course. Typically about 50 students fall into this category so it makes for the mantra: "50 under 50 for Math 50." Thus the argument can be made that students scoring below 50% on the QLRA should take care in course selection during their transition to college life. The Q-score on the Bowdoin 30 question entrance exam is one of our best predictors of academic success. Performance on the exam is positively and significantly correlated with cumulative GPA (r =0.39 N ~ 3,000 students from last 6 years), and MCSR GPA (r = 0.48) where MCSR represents our math/science quantitative courses. Also note that the Q-score is more strongly correlated (r = 0.48) with first year cumulative GPA, again making the case for paying attention to this score in first year advising and course selection.
A multivariate analysis was conducted this summer at Bowdoin. Multivariate regression allows one to control for multiple explanatory influences simultaneously. The first key result is that the models indicate that the Q-score is significantly predictive of Cumulative GPA and MCSR GPA even when controlling for a variety of other potential influences. This finding provides evidence that the associations indicated by the simpler bivariate correlations are likely to hold legitimate predictive power regarding future academic performance. Importantly, the models include Math and Verbal SAT/ACT scores among the explanatory variables. Thus, two entering students with identical Math and Verbal SAT scores but different Q-scores are predicted to have different GPA, suggesting that there is additional information in the Q-score beyond that of the aptitude test scores that is valuable for assessing future academic performance at Bowdoin. The fact that Q-score is at the high end of the range for both sets of coefficients (for cumulative and MCSR GPA) indicates the potential power of the Q-score to predict academic success across the curriculum, not just in MCSR courses.