E-STEM Evaluation

The core components (field course, professional development activities, and badges) of the E-STEM program were evaluated using a mixed-method approach that analyzed student interviews, student work, responses from students to pre and post surveys, and the PIs' reports on their experiences and implementation of the E-STEM program. The evaluation of the E-STEM program relied on the program goals as presented on the About Page.

In the evaluation of the summer field course, the SERC evaluation team administered surveys and conducted interviews with students from the 2017 and 2018 cohorts and analyzed assessment data from 2018 assignments. Group interviews conducted with the PIs aimed to identify the student and faculty outcomes of the professional development activities and the badge program.

Field Course

During the field course, students had the opportunity to practice and master critical field skills; learn about the incredible breadth of career opportunities; in environmental fields; and network with potential employers. The student surveys and interviews provided insight into the effect of the field course experience on student knowledge, perspectives, and skills. Key findings from the surveys and interviews include:

• Students had a statistically significant increase in their knowledge of particular environmental professionals and career pathways, suggesting that students made important strides from pre- to post-field course in their ability to network and to take the steps necessary to prepare for careers in environmental science.
• The field course drew students with a high level of interest in environmental careers, which was maintained at the close of the program.
• Students showed substantial changes over time in their knowledge about specific careers in environmental science.
• Statistically significant changes in students' confidence in environmental field skills and their comments from interviews, indicate that students believe they made gains in a large number of field skills.

Student gains in field skills are reflected in the student work as well. Students demonstrated improvement in their mapping skills and presented some improvement in their field notes skills throughout the field course.

Professional Development Activities

The PD activities were initially designed for students who had attended the field course, and were ultimately open to all students who fulfilled the prerequisites. The PD activities aimed to accomplish four goals:

• Goal 1. Increase students' awareness of and access to a broad group of professionals working in ESTEM fields;
• Goal 2. Facilitate student understanding of potential pathways to future careers using the visiting speakers as role models;
• Goal 3. Provide opportunities for students to read and discuss scientific/technical literature and reports; and
• Goal 4. Assist students in applying to internships, 4YCUs, graduate programs, first jobs, etc. through resume, cover letter, and application preparation.

The COA course and USF seminar attended to all four goals. The Mt SAC workshops focused on Goal 4.

Goals 1 and 2. "Increasing student awareness of and access to professionals" and "Facilitating understanding of career pathways"

The same efforts targeted the first two goals. The COA course and USF seminar had students regularly interact with professionals. They hosted visiting professionals who shared information about their career paths with the students. The PIs reported that key to the success of these interactions were active engagement and reflection. To enhance engagement, the PIs worked to identify a diverse pool of professionals who they vetted based upon "how they would interact with students," "their ability to present to the appropriate level (i.e. first- and second-year students)," and their willingness to share their experiences including the challenges they faced. To actively encourage reflection on the part of the students, the PIs at COA and USF used the career portfolio field activity. Additionally, the career portfolio provided students with informational interview-type questions to ask the professionals, further supporting student-stakeholder engagement. 

Goal 3. "Reading and discussing scientific or technical literature and reports"

The COA course and USF seminar both had components to address reading and discussing scientific or technical literature and reports. They assigned readings recommended by the visiting professionals to supplement in-person interactions and also to provide a window into the literature typical of different environmental fields. During the first year, USF seminar students engaged more with the professionals than the readings. Therefore, in the second year of implementation, the reading component was dropped. The COA course students also found the readings to be less engaging than the professionals, encouraging the PI to prioritize personal interactions with professionals over readings. In addition to the readings, the COA students had the opportunity to apply what they had learned to write their own proposal.  Additionally, the proposal assignment required students to cite, and, therefore, find and read, at least five scientific or technical reports.

Goal 4. "Applying to an internship, 4YCU, graduate program, or first job"

The COA course, USF seminar, and the Mt. SAC Workshop Series guided students through an application process. The nature of the process depended on the students' needs and familiarity with opportunities, which varied by institution.

COA

The COA course provided students with the opportunity to develop a job search plan in addition to developing a CV/resume and cover letter for a specific position. When first asked to identify "a permanent job or an entry-level, first-year-out-of-college job," the COA students tended to report back on temporary positions. Upon realizing the students were unfamiliar with different types of jobs and careers (e.g., differences between hourly and salaried positions; knowledge about benefits), the COA PI added instruction to assist in developing this knowledge and to emphasize the difference between a career and a job through which you gain experience. Ultimately, the PI reported that assigning students to search for entry-level positions felt "forced," since they were predominantly first- and second-year students. To optimize relevance and allow students to learn these skills in a manner that more closely aligned with their needs, the course included assistance in applying to internships and summer positions. 

USF

Early on, the USF seminar focused on helping students understand job postings. In searching for positions for one of the seminar assignments, the students expressed confusion about some of the language in the postings and concern about postings they saw requiring master's degrees or long lists of qualifications. To address student concerns in these areas, the PI worked on helping students to "demonstrate understanding" by encouraging her students to work through a process in which they identified a job ad, extracted keywords and important points, rewrote the posting in their own words, and then wrote a skeleton response of a cover letter to address their rewritten posting. One particular challenge the PI faced in the USF seminar was students misunderstanding the difference between personal essays and cover letters. To encourage engagement in addressing this misunderstanding, the PI had students play an active role in reviewing the applications of their peers and collaboratively identifying common problems. Unlike the COA students, many of the USF students were familiar with more general employment information, and thus the PI focused on motivating students to consider what a professional position looks like. 

Mt. SAC

The Mt. SAC workshop series guided students through each step in the application process. The students were focused on finding transfer schools and internships accessible to 2YC students. Consequently, the PI focused on increasing student awareness of accessible internships and transfer schools while assisting in the application process. This was provided in a variety of settings. The students worked in small groups to discuss a list of 2YC-accessible internships provided to them by the PI. The PI recognized that the students needed internships where they weren't necessarily going to be competing with junior and senior-level undergrads and compiled the list – which included the STEM C's Program, the Mosaic's Program and the Latino Internship Heritage Program – to address their needs. The students then developed their CVs and cover letters in other workshop sessions in small groups and with one-on-one guidance from faculty participants. The Mt. SAC PI also found herself, outside the workshop series, facilitating connections between students and ESTEM professionals or geoscience departments that were of interest to students for transferring.

Badges

The badges address the final "documenting and archiving skills acquisition" goal. In developing the badges, the PIs realized that the following factors prevented the badges from fully serving their role as documents and archives of skills acquisition.

1. The meaning of skills acquisition for 2nd year students; and
2. The holistic nature of the skills and content knowledge the badges (mapping, field notes, hydrology, and ecology) captured.

The majority of the students participating in the field course were second year students, and the PIs did not expect for them to reach professional proficiency in the skills through a 2-week field course. Ultimately, the PIs reported that the badges served as a means with which students could reflect on their progress on badge skills and content knowledge. Key to achieving this use was (1) having activities throughout the course use the badge rubrics, and (2) providing timely and regular feedback on skills via the badge rubrics.

Due to the holistic nature of the badge rubrics, there were a number of technical and "soft" skills that contributed to a single badge, making it particularly difficult to assess a students performance on a binary scale. The PIs suggested two possibilities to address this in future badge programs.

1. Developing badges for specific skills (i.e., stream gauging) and explicitly listing the time dedicated to this skill.
2. Instead of conferring badges, recast the badge program as a training process in which students undergo training and then have a debriefing on the skills beneficial for particular E-STEM careers and/or develop a training plan.

Overall

The PIs reported that approaching this program as a multi-institutional team provided them with much needed support. Additionally, PIs reported that reaching out to stakeholders for inclusion in the field course and professional development activities expanded their own professional networks. They reported now having a more diverse network, including more professionals outside of academia.