Initial Publication Date: June 10, 2016

Project Evaluation

Goals of the External Evaluation

The goals of the external evaluation were to

  1. provide an independent, objective view as to how well the project management is functioning as a team,
  2. evaluate the extent to which the needs assessment meetings are reaching the project goals, and
  3. to review stakeholder reports and the project team's final report for accuracy and completeness.

Evaluation Procedures

Evaluation activities included focus group observation as well as daily road checks (formative assessment) and surveys (summative assessment) for participants related to the focus group goals. Thematic content analysis of the qualitative responses from participants highlighted common themes among the focus groups and are reported as paragraph summaries specific to the individual meetings.

Key Findings and Recommendations

Key Findings of the Overall Evaluation

  • The majority of participants in the focus groups enjoyed the meeting activities and appreciated the small group discussions and networking opportunities.
  • The administrators and instructors felt that the goals related to identifying instructional resources and understanding their importance were addressed in the focus group. The goals related to student preparation and instructional support for those teaching out of field still needed to be addressed.
  • The education researchers and resource providers felt that the goals related to identifying areas for future research and methods for disseminating resources were met at the focus group sessions. The resource providers felt that the goal related to designing resources that incorporate knowledge and habits of mind important to employers still needed to be addressed.
  • In the open-ended feedback, participants indicated that the focus group content sometimes lacked specificity in addressing the issue of minority participation and did not fully identify action items to tackle after the meeting.
  • Recruitment was difficult in terms of getting minority participation and a diverse representation of MSIs.
  • The ideal model activity encouraged participants to think about the issue of broadening participation from a systems perspective. The models are useful tools for the geoscience community to address issues related to increasing minority participation.
  • The participants noted that the focus groups were well organized and facilitated and encouraged all voices to be heard.

Key Recommendations

  • The three-day Geo-Needs focus group meetings were ambitious, and there was not enough time to tackle all of the goals that they set out. The evaluation recommends that the project management team consider a follow-up meeting with participants. At the follow-up, the Geo-Needs project team can target the remaining goals using information gleaned from the ideal model and from participant reporting.
  • All participants noted that having examples of what works and what does not would be useful for developing their own action plans that address the issue of underrepresented student participation. Consider reconnecting with meeting participants to generate lists of exemplary programs that can be shared among participants.
  • Employ the brainstorming/superhero technique with participants in a follow-up meeting to generate a list of short-term and long-term goals to address the issue. This list could also be used to help identify subareas of the issues surrounding broadening participation.
  • Participants indicated that the focus groups were great places for networking and starting new research collaborations. Consider formalizing working groups from the meetings so that collaborators could identify short-term and long-term goals to address issues related to broadening participation.
  • Identify a few specific issues related to the larger issue of broadening participation for working groups to investigate. Develop goals and action items for these specific subareas.

Focus Group Data and Observations

Participants provided feedback to the evaluator with respect to their satisfaction with the interaction among focus group participants and their agreement on whether the goals of the focus groups were met. They provided information on how the focus group influenced their thinking about diversity issues , and how they plan to apply what they learned to their own work at their institutions. The participants also provided feedback on logistics, facilitation, and individual focus group activities and content.

Participant Interaction

Focus group participants were asked to rank their level of satisfaction with their interactions with other meeting participants (See figure). Administrators (n=6) and Instructors (n=13) ranked their interactions with other participants in the same focus group and in the concurrent focus group. Administrators were more satisfied with their interactions with Instructors than Administrators from their own focus group. Instructors were more satisfied with their interactions with other Instructors than their interactions with Administrators. Education Researchers (n=10) and Resource Providers (n=8) were to rank their interactions with other participants in the same focus group and in the concurrent focus group. Education Researchers were more satisfied with their interactions with other Education Researchers than with Resource Providers. Resource Providers only ranked their interactions with other Resource Providers, and found them satisfactory.

Meeting Goals

Participants were asked to rank the extent to which they believed specific goals were met at each meeting. Administrators (n=6; see figure) agreed that the focus group met the goals related to determining instructional opportunities, discussing how geoscience offerings enhance the institutional mission, determining how to best inform students about workforce possibilities, and identifying resources needed by institutions to establish/sustain geoscience programs. Administrators felt that the focus group did not meet the goals related to determining how to prepare students for careers, how to serve the communities in which 2YCs/MSIs are situated, and what programs would enhance career opportunities.

Instructor (n=13; see figure) responses varied more than what was seen in the Administrator responses. Overall, the Instructors agreed that the focus group met the goals related to identifying instructional resources, barriers to use of innovative resources, and support mechanisms that help instructors introduce students to career opportunities, and determining administrative supports for instructors. Instructors felt that the focus group did not meet the goals related to identifying professional development programs, classroom resources for startup programs, delivery and support mechanisms for innovative curriculum, and sustainability needs.

Resource Providers (n=9; see figure) agreed that the focus group met the goals related to determining how providers can disseminate materials and communicate to potential users. Resource Providers were less in agreement about whether the focus group met the goals related to how to identify what is needed by the community; the roles of 2YC/MSIs in resource development; and strategies for developing dynamic materials to meet a changing workforce and society; and how to develop resources for knowledge gains, skills, and habits of mind.

Education Researchers (n=10; see figure) agreed that the focus group met the goals related to determining the future work needed to better understand the pathways and preparation for the workforce. Education Researchers were less in agreement about whether the focus group met the goals related to determining mechanisms for research, uncovering barriers to research, identifying mechanisms to promote research, and exploring how research and evaluation can inform geoscience education.

Evaluation Findings

How has this meeting influenced your thinking about the issues surrounding diversity in the geosciences?

Administrators came away from the focus group with a feeling that there is still a lot of work to do regarding diversity in the geosciences, and that theissues surrounding diversity are complex . They recognized that while minority students have an interest in the geosciences, some students require supplemental coursework in reading and math in order to meet the requirements of some of the geoscience classes. The Administrator group felt that there needs to be more research into the factors that affect diversity in the geosciences, and that these findings need to be used to increase academic institutions' awareness of the barriers that keep minority groups from the geosciences. The Administrators identified the importance of cultural literacy and sensitivity in recruitment and retention of minority students, and that public opinion may help/hinder participation from these student groups.

Instructors felt that the focus group brought an awareness of the concerns related to diversity in the geosciences, and that the group identified some realistic solutions to some of the problems posed (though the solutions were not one size fits all). The Instructors liked that the focus group made the participants think about the big picture surrounding diversity issues. They appreciated learning new things, finding new resources to use, and networking with other instructors and administrators, and they plan to share what they learned with their colleagues. This group stressed theimportance of the public in recruitment and retention of diverse students.

Resource Providers thought that the focus groups did not add new insights but did reaffirm/reinforce the importance of the diversity issue. The meeting raised awareness and helped these participants to see new possibilities and make new contacts. The ideal model helped the Resource Providers understand where they fit in to the connections among Administrators, Instructors, and Education Researchers and helped to identify the priorities for minority-serving institutions.

Education Researchers found that the focus groups provided insight to the issue of diversity, introduced the participants to the published research on the topic, and highlighted the need to think outside the box to come up with solutions. The Education Researchers found that the ideal model helped pull together many aspects of broadening participation and helped them visualize the connections and relationships on the topic among stakeholders. They noted that the geoscience community is invested in broadening participation, but little progress has been made on the issue to date. Some felt that the focus group did not necessarily influence their thinking on broadening participation, but reinforced what they already knew and broadened the scope of the issue.

How will you use and/or apply what you learned at the meeting in your own work with underrepresented minority students?

Administrators identified useful resources in the focus group and plan to share the information with faculty at their home institutions. Some administrators left the focus group wanting to expand the geoscience programs on their campus and planned to use the website Building Strong Departments to help them do that. They planned to develop procedures to address diversity and inform recruitment strategies at their institution.

Instructors identified items that faculty members could act upon at their home institutions to address recruitment and retention. They planned to use the resources from SERC and also share them with their colleagues. The Instructors noted the importance of community outreach, making relationships with other institutions and collaborating on grants as potential next steps. They enjoyed the focus group activities and planned on using them in their classes.

Resource Providers came away with a greater awareness of the issue and the resources that are available to assist in addressing the issue of broadening participation. One participant indicated that they would look into "market segmentation" to help disseminate their resources. Some mentioned that they would involve minority-serving institutions/diverse communities and the Geo-Needs website in the resource development process. The Resource Providers plan to look into the research for ideas on reaching diverse audiences, and came away with new ideas for collaborations and projects.

Education Researchers planned to get faculty at their home institution to discuss diversity issues, apply what they learned from the meeting to programs that they run, and research what resources are available at their home institutions. They wanted to become informed about the best practices for mentoring and came away with the understanding that trust between faculty and students takes time to develop. They found utility in the ideal model and would keep it in mind when thinking about these issues. The Education Researchers felt that the meeting helped establish (and re-establish) connections with issues surrounding broadening participation and research. Many found the presentation about validation theory particularly useful and hope to see how they can apply it to their own research. They hope to move the research agenda forward by following up with the contacts they made at the meeting and would like to partner with other researchers who attended.

What challenges (if any) do you foresee having to overcome in your efforts to use or apply what you have learned at the meeting?

Administrators indicated that lack of time and resources are always a challenge for implementation. They also noted the need for qualified faculty and the need to address structural problems with the alignment of campus programs to help with implementation. They mentioned that faculty resistance/lack of buy in might also hinder the use of resources to help with recruitment and retention of minority students.

Instructors noted lack of time, resources, administrative support, and buy in from colleagues as obstacles to applying the knowledge gained at the meeting. Instructors have difficulty navigating the structure of their home institutions in order to implement changes to help address diversity issues.

Resource Providers were hoping to have more case studies highlighted in the meeting to know what works and why. They also listed lack of time, money, and other resources as an obstacle to change. Developing partnerships with new stakeholders was also listed as an obstacle to implementation.

Education Researchers identified lack of departmental support, time, energy, and maintaining focus as obstacles to implementing change at their institutions. They also suggested that change would require altering the traditional perspective on what geoscience education looks like. Another noted that having a limited network of collaborators on the topic of broadening participation and difficulty making new research relationships as an obstacle to moving forward.

I had a positive experience and enjoyed interacting with everybody. I would have loved to spend more time interacting with everybody learning more about their interesting work! I commend the facilitators for a well-organized and well-facilitated meeting. It took a tremendous amount of work to develop and facilitate this meeting and to navigate the challenges of doing that organization/facilitation as a team. Also, it is challenging to have thoughtful discussions about diversity with a large group of people. The facilitators did an excellent job of navigating the challenges in a respectful and professional manner. I hope that the facilitators found the meeting helpful! —Education Researcher

Evaluation Conclusions

Overall, the Geo-Needs project was guided by the goals of using of existing instructional resources to engage URMs in the geosciences and developing an ideal model to help stakeholders address this issue. Feedback collected from participants via open-ended and Likert-scale surveys indicate that the focus group goals were important but were perhaps too ambitious for the three-day focus group sessions. Because of the nature of the focus groups, the facilitators wanted the participants to contribute their knowledge and expertise to address the issues at hand, while the participants had hoped for more concrete action plans as a product of the focus groups. The facilitators did a tremendous job navigating the complexity of the issues and allowing voices to be heard. Because of this, the discussion often brought about important topics related to the goals of the overarching project, but perhaps not the individual focus group. This likely contributed to the participants' comments of a "lack of focus" to some of the focus group content. The larger-scale discussion of broadening participation brought up important topics related to recruitment and retention at MSIs. The Geo-Needs project can use the information gleaned from the ideal models and from participant discussions to shape the focus of future work. The project is, after all, a needs assessment. With the products from the focus groups this summer, the project management team has a framework for future work. Those goals that were not fully met in the focus group sessions can be addressed in a follow-up meeting where participants can work within the ideal model framework to identify specific goals related to the issue and develop action plans to address those goals. The ideal model alone is an important product of the focus groups and can be disseminated to programs throughout the geosciences to help utilize existing resources for increasing minority participation.