Concluding Remarks

Ten years ago, NSF's Division of Undergraduate Education launched a remarkably ambitious program to build an openly accessible digital library, dubbed the National Science, Mathematics, Engineering, and Technology Education Digital Library (NSDL), whose content would support STEM teaching and learning at every level of education, in both formal and informal settings. This report—the outcome of a three-day retrospective workshop on the effort—concludes that NSDL, as it evolved over the years (to the point where it is now titled the National STEM Education Distributed Learning program), was judged to be successful by some attendees and less than successful by others at the April 2012 meeting.

NSDL design and development was both a technical and social enterprise, and participants recognized its success to the degree that its technical infrastructure and services enabled the aggregation, organization, archiving, and distribution of assets held by the library and that NSDL supported and transformed teaching and learning in the STEM disciplines at all levels. NSDL-funded projects made significant contributions in both of these focus areas, and NSDL.org offers global access to digital STEM education resources. Many individual projects succeeded in supporting the expectations and needs of specific communities (defined by discipline, audience, and geography) that were empowered to contribute and use resources and services in new ways to support STEM education. However, as a program, NSDL fell short of the grand vision of an integrated portal that served the diverse interests and needs of STEM education for all audiences.

Reflecting on their individual perspectives on NSDL, workshop participants sought to discover the central messages from their experiences and to understand and articulate the tensions inherent in working on such a diverse, distributed, and collaborative effort. This section summarizes the lessons learned both while building NSDL and reflecting on the building process during the April 2012 meeting. Three central messages emerged through the workshop effort: (1) large, collaborative, interdisciplinary projects require organizational structures to support the efforts of the whole; (2) an organization's vision and goals must actively guide management and planning; and (3) keeping collaborative projects collaborative requires strong communication networks.

Large, collaborative, interdisciplinary projects require organizational structures to support the efforts of the whole.

As noted elsewhere in this report, NSDL was unique within the broad scope of NSF programs. Hundreds of small projects were funded, along with a central organization that was charged with building infrastructure and organizing the community. (For more information, see the NSDL history authored by NSF Program Officer Lee Zia.) The entire body of projects—each bounded by different funding levels and durations and focused on developing specific elements of infrastructure, services, or content to be integrated into NSDL—formed a loose, geographically and intellectually distributed federation that worked to realize a vision of NSDL as articulated in the NSF RFP for the program.

Grant-funded projects need organizational models that balance centralized and distributed management structures.

At the inception of the NSDL program, NSF funded a centralized organization to develop key pieces of infrastructure and coordinate the eventual integration of individual projects with this framework. At the same time, community-based governance was created to represent the needs and views of individual project leaders to the central organization. Each of these groups was deemed important, but no clear organizational structure or accountability mechanism connected the central organization to the individual projects.

The lack of coordination between these two groups set the stage for increased tension and uncertainty among the project participants about NSDL goals, objectives, and decision-making authority. This lack of agreement made it difficult to establish an organizational structure that could truly meet the diverse needs of NSDL participants, develop bridges among the differing opinions, and establish an open or transparent decision-making process. Lack of agreement on such fundamental factors also made it difficult to evaluate NSDL's success, as one is forced to ask, "Which NSDL?"—the NSF program, the central organization, or the individual projects.

The workshop organizers were struck by the fact that workshop participants, 12 years into the program, continued to debate the origins, merits, failures, and accomplishments of the community governance model and the central organization. The fact that this debate continues to define the experiences and perspectives of NSDL project leaders and participants produced this critical recommendation: Early on, large, collaborative, interdisciplinary projects should establish an organizational and governance model around which participants can form at least a rough agreement in order to move the project forward.

A community governance model such as that envisioned for NSDL might have been more successful if two important facets had been in place: (1) clear lines of authority in both the central organization and the community; and (2) project accountability across both groups. The sometimes-competing leadership structures significantly hindered decision making and led to a leadership vacuum (real or perceived). These factors, combined with the lack of authority or power on the part of both structures to hold individual projects accountable to the collective enterprise of NSDL, exacerbated the diffuse nature of the project, making coordination even more difficult.

Finding the "right" organizational structure requires time and testing.

Future efforts should consider balancing centralized and distributed management structures as well as "top down" and "bottom up" approaches to research and innovation. In this regard, timescales for progress are often longer than anticipated. In particular, an iterative process for community building and goal refinement must be allowed to play out. It also is important to recognize emergent outcomes that cannot be determined through a proposal process.

Building NSDL through individually funded projects also created structural issues. Individual awards did not necessarily comprise the whole of the resources and services needed to build NSDL. This funding structure resulted in uneven development that could not be rectified through ad hoc proposal submissions. PIs were primarily responsible for the successful completion of their own project's work plan and only secondarily responsible for the success of NSDL as a whole. Organizations must carefully consider the issue of centralized versus decentralized development to ensure that critical services are established for the good of the whole while allowing enough flexibility for individual projects to creatively develop additional systems that can meet unique needs or produce unexpected outcomes.

One way to mitigate the confusion inherent in a distributed, collaborative project is to create an independent position that serves both the central organization and individual projects. In NSDL's case, such a position could be invested with the authority to make decisions, establish workflow processes, and coordinate among the many projects to resolve collaboration and communication issues. Some workshop participants felt this role should focus on achieving consensus and supporting collaboration rather than directing activities. They also noted the tension between the need to have a directive manager to achieve integration and the desire of projects to work in a more loosely coordinated network.

Numerous challenges result from trying to align distributed, collaborative projects with the goals of NSF and the federal government and the desire to build a financially self-sustaining facility. These goals can come into conflict as needs and expectations change in relation to different timescales, research questions, and levels of project maturity. Workshop participants keenly felt shifts in project direction related to changes in government priorities for the NSDL program and its impact on STEM education. The proposal process resulted in projects with different foci, goals, activities, and products that did not always add up to the larger NSDL vision.

An organization's vision and goals must actively guide its management and planning.

In addition to factors that affect organizational structure, it is also important to consider how an organization's success can depend on arriving at a broad, functional level of buy-in to a clearly articulated vision and goals.

Choose the words used to describe a vision and goals carefully.

From the outset, a multiplicity of meanings and values were attached to the words behind the acronym of NSDL (see Library as Metaphor). The program name and initial RFP became strong indicators of the project's overall direction. However, individual projects established separate proposal goals that did not necessarily represent or align with the broader vision, goals, or values of the larger group. A tension grew between the desire of individual projects to participate in the larger endeavor while also being evaluated on their own goals. It was difficult for projects to acknowledge such divergences from the larger effort and then either realign their own goals or create a path that allowed these differing goals to be accomplished successfully.

Use goals as guideposts to ensure alignment of project activities in a distributed organization.

An organization's goals must guide its planning, project management, and evaluation efforts. Following this rule requires spending at least as much effort in managing the evolution of goals as was invested in their initial definition. All too often, organizations create and record goals and then rarely reference them again, especially in day-to-day practices. In such cases, it becomes difficult to manage the expectations associated with those goals and perhaps even impossible to determine the appropriateness and effectiveness of specific projects and activities.

As NSDL matured, goals of the central organization and individual projects evolved to reflect changes in technology and education. Such evolution can affect any assessment that uses these goals to gauge an organization's successes and failures. Factors that can cause goals to evolve include advances in research and commercial markets and changes in project expectations. Such influences can become powerful forces, requiring careful monitoring to keep an organization true to its vision and goals. Projects should remain flexible to respond to changes in objectives, organizational demands, and external environmental conditions.

For effective collaboration, value the success of individual parts as highly as that of the whole.

From the outset, projects need to establish processes to ensure holistic, continual assessment of their project goals and to disseminate their evaluation results. As in any research effort, expected project outcomes such as functionalities, services, and populations served had to be addressed. At the same time, the creativity of the independent projects also allowed some surprising and serendipitous results that should be counted as positive credits for NSDL. Evaluation of such complex organizations requires an evolving evaluation process with widespread participation.

Sometimes a tension can develop in sharing the results and aggregating the outcomes of an individual project with those of the central organization. Although workshop participants indicated that it is easier to conduct evaluations and report results at the project level, the same can be said for the larger central organization. Projects must take care that the success of one group does not outweigh that of the other, as the success of each individual effort is integral to the success of the entire distributed organization at all levels.

Keeping collaborative projects collaborative requires strong communication networks.

Communication in a large, distributed, collaborative project is a daunting challenge.

In a distributed environment, information communications technology is both necessary and helpful, but it did not fully support the communicative actions needed to establish common ground across NSDL. The communication required to facilitate NSDL's ongoing successful development was dispersed across multiple projects, timescales, communities, developer groups, networks of practice, and geographic and virtual locations.

Addressing many of the issues identified in this report required additional intensive and reflective communication efforts. For example, challenges included facilitating communication between the central organization and individual projects, defining the role of the committees between annual meetings, and supporting the ongoing management of technical development among designers, developers, catalogers, and others. At the same time, NSDL's widely distributed structure and the inability of many PIs to commit to more than a part-time effort meant that this communication was very difficult to achieve. This finding has important, practical implications for future projects.

In distributed contexts, communication needs to be both rich and nurtured.

Even in a highly technical project with access to high-quality information, communication technologies, and face-to-face communication—such as that in the NSDL Annual Meetings—was highly prized. Communication tools and processes need to bridge the timescale and geographic gap among distributed projects.

Projects should not underestimate the amount of communication management or the cost (i.e., fiscal and time) that a distributed program requires.

Communication management brings new members into a project and connects them more strongly to its vision, goals, and participants. The type of face-to-face communication necessary to keep communication lines open and transparent requires significant resources. It can cost as much to build an organization's social network as to develop the actual product.

One of the challenges encountered by the federated NSDL was balancing the promotion of individual projects with that of the broader NSDL organization. Although there were large efforts to build brand recognition and use for NSDL collections and services, we were not successful in building a mechanism that allowed users to hold dual allegiances to their disciplinary or specialized libraries and to NSDL as a whole. Given the breadth of roles that NSDL attempted to play in STEM education, it also proved difficult to build a stable and sustained community of users.

Final Thoughts

As this report has demonstrated, NSDL has made tremendous strides in developing STEM educational digital libraries. It has developed high-quality collections and innovative tools and has created and sustained a broad and lively intellectual community, all in a distributed setting. NSDL's vision served as a catalyst for interdisciplinary research (1) on the transformative role of technology in the classroom; (2) the pedagogies and information literacy skills used by STEM teachers and learners; and (3) on digital libraries in general.

NSDL efforts enhanced learning opportunities for teachers and students by offering pathways for them to explore STEM content, methods, data, visualizations, pedagogies, assessments, and related supporting services. NSDL has opened new possibilities that can transform what and how we teach across the STEM disciplines.

The NSDL program and projects provided a platform for examining the impact of digital STEM resources on teaching and learning. Early on, interdisciplinary perspectives provided models for involving users in the design of digital libraries. Although 10 years of evaluation efforts did not yield measures of impact, they did result in a nuanced understanding of the difficulties of such measurement, concluding that teachers are a reasonable audience to study and that changing teacher practice requires a sustained effort (see essay on Transforming Teaching and Learning).

NSDL served as a catalyst for digital library research and was one of several large research initiatives undertaken about the same time around the world (see the Scaling Technology essay). As a result, the U.S. research community participated and learned from this broader conversation. Again, what made NSDL unique among these other initiatives was its interdisciplinary approach to developing a STEM education digital library. Going forward, digital libraries are increasingly both components and examples of wider information infrastructures, including cyberinfrastructure and cyberlearning, which are "used to facilitate the distributed, collaborative use of content over time and distance" (Borgman, 2007). NSDL can continue to serve as a catalyst and model by functioning as a working platform for cyberlearning implementations and innovative cyberinfrastructure experiments.

The NSDL program and its component projects have demonstrated the power of digital libraries to improve the quality of STEM education and widen the reach of the very best resources it has to offer. At the same time, NSDL has been a grand socio-technical experiment, resulting in important lessons learned in both the social and the technical dimensions. As with any experiment, negative results can be as valuable as positive ones. Viewed through that lens, NSDL has been successful at the program, project, and personal levels, although the divergent perspectives on the definition and scope of NSDL may mask the synergistic outcomes achieved at each one. Almost every workshop participant noted that one of the reasons they valued working on NSDL projects and being part of its community was the opportunity to exchange ideas and collaborate with colleagues from other disciplines. Indeed, many attendees noted that the connections they made while working to realize the vision of NSDL have had a lasting impact on the direction of their work, and indeed, their lives.