Essay: Organizational Evolution
and Effectiveness 
 

Introduction

The work to build NSDL was funded by the NSF, with more than 200 interdisciplinary projects supported under its NSDL program. As a result, NSF peer review processes and grant-funding structures had a huge impact on the ways work was organized, communities were created, and participants communicated. NSF provided strategic guidance through program solicitations and funding tracks but did not direct the daily activities of this large-scale collaboration.

Individual projects managed their own activities, but to accomplish cross-cutting work, a grassroots committee structure was combined with a coordinating Core Integration (CI) team, with the CI taking on increasing levels of responsibility as the NSDL project matured. This approach required a governance structure. Communication relied on face-to-face meetings, including an Annual NSDL Meeting, smaller working group meetings, and online tools.

This essay briefly examines several aspects of NSDL's organizational structure and their impact on organizational development. It then considers the role of evaluation within and across the organization. It concludes with lessons learned around the organizational development and evaluation of a large, federally funded, collaborative project.

Evolution of NSDL as an Organization

Communication, organization, and governance were all essential for a federation of more than 200 projects to move towards the central goal of building NSDL. Initially, the organizational structure combined the following elements:

• A grassroots committee structure, comprised of volunteer representatives from small projects funded by the NSDL program who were guided by policies and bylaws
   
• A Core Integration (CI) team, funded by the NSDL program to develop the library technical infrastructure and to support the NSDL community library-building and governing efforts

The grassroots committee structure provided a mechanism for community input into the design and development of NSDL. These committees also completed a significant amount of cross-disciplinary work on a voluntary basis. Although this all-volunteer effort was perhaps not sustainable as a part of the later organizational structure, the work of these committees included the following accomplishments:

• Defining a collections and privacy policy
• Determining metadata standards and guiding principles
• Identifying mechanisms for evaluating and studying the use of NSDL and its collections

This early work was fundamental to building NSDL. Committee participation also helped develop buy-in among the grant-funded projects and provide a foundation for creating the NSDL community.

As NSDL matured, the CI took on increasing levels of responsibility for all aspects of the library collections, community support, and technical maintenance, while the number of NSDL projects funded decreased but their scale and duration grew. The Pathways funding track, established in 2004, created projects of three to four years duration that were responsible for curating content for entire disciplines or "vertical" audience groups. As funding for small projects diminished, there was less need for committee representation, and the community governance structure seemed redundant with the evolving CI working relationship with Pathways projects.

The committee structure and the CI were each deemed important, but no clear organizational structure or accountability mechanism joined the central organization to the Policy Committee, subcommittees, or individual projects. In 2008, the Policy Committee disbanded.

Effectiveness of NSDL

Early in the NSDL development process, the grassroots committee structure and the CI began looking for ways to demonstrate program and project impact. Because NSDL was highly experimental, as were all digital libraries at that time, no theory or practice sufficiently addressed the specific evaluation issues associated with the emerging NSDL. The Education Impact and Evaluation Committee, formed to address these issues, was challenged by what to evaluate as questions persisted around which entities made up NSDL (i.e., the CI, the individual collections, and other NSF-funded projects). As the socio-technical and cross-disciplinary aspects of the program took shape, they created another evaluation challenge—to identify what exactly constituted impact. It became clear that measures for evaluating physical libraries (e.g., the number of collections and items, reference transactions, or books circulated) did not adequately capture the NSDL-specific context or its possible impact on STEM teaching and learning, although the DigiQUAL project was an early effort to develop new evaluation measures for digital libraries.

In the middle years of NSDL's development, the NSF perspective on what constituted "impact" shifted. Rather than identifying several areas of emphasis, the primary measure for impact became focused on student learning, de-emphasizing other impacts such as changes in teaching practice or developing communities of practice. Led by the Evaluation Committee, many NSDL projects worked together to attempt to find common evaluation approaches, metrics, and tools. However, shifting priorities between the need for formative evaluation and the summative evaluation of impact made it difficult to institute longitudinal evaluation studies. The end result was that evaluation worked well at the project level but was significantly more difficult to institute at the NSDL program-wide level, in part because of lack of funds for the scope of activity such an undertaking would require.

Lessons Learned

The long experiment with the unique combination of community-based governance and a centralized coordinating organization working together within the structure provided by NSF funding led to a number of lessons learned:

Long-term, collaborative NSF projects require a funding model that supports growth and sustainability.

NSF's culture and funding processes had profound effects on the organizational structure of NSDL. The NSDL program solicitations from NSF attracted proposals from a diverse, multidisciplinary group of software developers, librarians, STEM educators, professional societies, and publishers. A positive result was that the variety of funding tracks allowed collaborations among these disciplines, educational sectors, and organizations to build new types of collections and tools and reach new communities of users. There was substantial creative power in these collaborations, which would not have been realized without the program solicitation and multiple funding opportunities.

However, the NSF's peer review process posed a particular challenge by evaluating these interdisciplinary proposals on just two criteria: intellectual merit and broader impact. Within the existing review processes, there was no way to accommodate specific, time-sensitive needs (e.g., technical, content, service) that would support building a working digital library. As one NSDL program officer said, "When you build a house, you want a contract that specifies just one kitchen and a few bedrooms and bathrooms. But if you build a house using an RFP model, you are likely to get many proposals for kitchens and none for bathrooms."

At times, multiple projects addressed the same issues and created different kinds of "kitchens," some of which were more successful than others. Given the absence of a good model for the digital library at the outset of the work, this kind of exploration was important, as were the interdisciplinary connections that were forged to move forward these ideas.

Organizational structures need dedicated funding, time, and flexibility to develop.

Collaborations among such diverse communities as those represented in NSDL are extremely complex and initially require a loose organizational structure to allow participants to find and create areas around which they can coalesce, focus their efforts, and create plans for moving forward. Such organizations also require time for people to build agreement, find focus, and determine ways to communicate to their constituents. Involving participants and creating a community require significant human and financial resources. As a large, collaborative, and grant-funded endeavor, NSF initially encouraged the growth of working groups and standing committees comprised of staff from NSDL projects. Ultimately, this structure was phased out, reflecting the flexibility of the NSF RFP approach to "start fresh" relatively easily.

These kinds of flexibilities can be viewed as a luxury in a production environment, yet they are necessary to create a common understanding and vision to move complex social and technological projects forward. The need to conduct research and rapidly move the results to a production environment contrasted with the needs of community building, which led to a number of tensions within NSDL. For example, (1) researchers needed time to iteratively work through complex issues, often in small, tightly knit teams; (2) the production environment required an agile and quick decision-making process across several distributed institutions; and (3) the governing system needed time to build consensus and agreement across highly distributed sites and the project.

Effective communication across large, collaborative organizations must be open, adaptive, and inclusive.

Successful organizations rely on effective communication among their governing entities, participating members, and users of their products and services. Several communication strategies were implemented for sharing information over the duration of the NSDL program, and many tools were adopted during the rapid evolution of Internet-based communications.

Evaluation of these communication strategies and networking events showed mixed results. Participants often indicated that they did not know where to find the most recent information about other projects' work or CI activities. Managing project-wide communication was more challenging and time consuming for participants and leadership alike than was originally anticipated.

As NSDL matured, the CI expanded its support for communications around the NSDL brand. CI staff and NSDL project members regularly promoted NSDL at regional and national conferences and through blogs, webinars, and podcasts. Since the NSDL community had expanded to include users who were not invested in building a digital library, communication strategies subsequently shifted to align with the needs of these new users that just wanted to find and use STEM resources. One result was that individual projects saw a benefit to promoting their collections as part of the NSDL brand. However, some research-oriented projects had other audiences besides these new users and could not effectively take advantage of these targeted outreach activities. (For more information, see Endnote 1: NSDL Communication Methods and Tools.)¹

Evaluation requires careful planning, systematic application, and integration into an organization's management.

The complexity, size, and vast reach of NSDL required that evaluation be embraced as an organizational value and fully integrated into its organizational and management practices. Integration would have helped mitigate barriers to gathering data (specifically, lack of access to K-12 student users to directly study impact on learning), strict IRB requirements or regulations, and privacy policies that limit use of certain data. Integrating systematic data collection would have ensured that timely results could be used by projects and across NSDL to improve collections and services and to provide strategic information to stakeholders such as funders and users. In addition to valuing evaluation and integrating evaluation activities throughout a project life cycle, additional lessons were learned:

• Methods of evaluating a large, complex organization need to parallel its life cycle.
• Evaluation depends on stakeholder agreement on who, what, and how the organization's products and services are intended to affect its clients.
• Theories of change or logic models are useful organizational planning tools and guides for evaluation. They make it possible to build evaluation instrumentation into technology and ensure that project activities are aligned with the intended outcomes and impacts.
• Evaluation succeeds when using multiple approaches. Both qualitative and quantitative methods are necessary to examine impact from the perspective of multiple stakeholders involved in a large, collaborative project such as NSDL.

Endnotes

Show ¹NSDL Communication Methods and Tools

Hide