The following is a brief discussion of the findings of the Digital POWRR Project's recent white paper, bringing out themes that the white paper's length restriction obliged its authors to downplay.
The
information science and cultural heritage communities have developed a variety
of guidelines and standards that can enable an organization to achieve high
levels of digital preservation.[1]
Many practitioners have struggled to fashion the technical infrastructure
necessary to realize them, however. Information professionals serving
medium-sized and smaller institutions lacking large financial resources have
especially have found it difficult to study and comprehend the complex
challenges presented by the need to curate and preserve digital materials in a
programmatic manner. Unsure that they understand the issue well, many have
hesitated to address it.
Many
larger institutions have also not yet devised the means to curate and preserve
their digital materials in a programmatic manner, but in one instance a state
university system has given rise to a coherent and practicable way forward.
Representatives of the California Digital Library have addressed the University
of California’s extensive and diverse digital curation and preservation needs
by devising and coordinating the operation of a set of free-standing but
interoperable applications, each performing a single or limited number of tasks
in the larger curation and preservation process. They described their method as
a micro-services approach. While they conceived this arrangement with an eye
toward producing an infrastructure able to function in a large institution’s
wide variety of work environments in a context of rapid and pervasive
technological change, a micro-services approach can also help medium-sized and
smaller institutions to identify and meet a very different set of digital
curation and preservation goals.
A Micro-services
approach
In 2009 California Digital Library
staff members began development of a new approach to the curation and
preservation of digital objects produced by the ten-campus, $25 billion,
238,000 student, nearly 20,000 faculty member University of California system.[2]
Seeking to dispense with the assumption that the curation and preservation of
digital objects required the installation and operation of a single, long-lived
application combining the necessary functions behind one user interface, they
proposed to employ a set of twelve independent but compatible applications, or
micro-services, each responsible for performing a single function within the
digital curation and preservation process.
They described utilities devoted to data preservation as identity, storage,
fixity, replication, inventory, and characterization. Utilities devoted to data
curation included ingest, index, search, transformation, notification and
annotation. The developers of the micro-services approach to digital curation
and preservation made a number of compelling arguments for it. Small,
relatively simple utilities would pose fewer challenges in their development,
deployment, maintenance and enhancement than a large, integrated system,
especially in the context of constant technological change. In addition, users
could easily adapt a set of distributed services to local conditions in
different divisions and departments of the university, and easily replace each
of them upon their obsolescence.[3]
The
authors introducing a micro-services approach to the information science
community seemed to worry that their colleagues and users would look askance at
a technical arrangement that they viewed as somehow incomplete or piecemeal. At
the same time that they praised its simplicity and flexibility, the authors
emphasized that librarians and developers could put a set of interoperable
utilities together to provide a very large university community producing
digital materials in increasing numbers, size and type with the ability to
curate and preserve them in a fully comprehensive manner.[4] In
2010 the California Digital Library introduced Meritt, a new repository
developed using the micro-services approach.[5]
Available to the entire University of California Community today, Merritt
provides long-term preservation of digital materials and also enables users to
share data. It makes its functions available via a user interface and an
Application Programming Interface enabling machine to machine communication. As
of March, 2015 Meritt contained approximately 1.5 million digital objects, some
containing over 10,000 individual files or occupying over 100 GB of storage
capacity.[6]
The Challenge:
Digital Curation and Preservation at Medium-Sized and Smaller Institutions
In a
2011 report Merritt’s developers recommended a micro-services approach to
digital curation and preservation to other members of the Association for
Research Libraries, emphasizing how a strategic combination of individual
services could produce “the complex global function needed for effective curation”
at large institutions.[7] At
about the same time, the Digital POWRR (Preserving
Digital Resources with Restricted Resources) Project
(http://digitalpowrr.niu.edu/), a team of librarians, archivists and other
stake-holders from medium-sized and smaller Illinois institutions lacking large
financial resources, used financial support provided by an Institute of Museum
and Library Services National Leadership Grant (LG-05-11-0156-11) to begin a
study of the problems that the curation and preservation of digital objects
presented to their organizations, and how they might address them.
The POWRR institutions were, and
remain, members of CARLI, the Consortium of Academic and
Research Libraries in Illinois, but its leaders informed the study team that
while they recognized the digital curation and preservation challenges their
members faced, they also lacked the financial resources necessary to begin to
address them.
Early in the three-year
investigation period, POWRR team members prepared a case study of each of the
participating institutions, describing it in broad terms while paying
particular attention to the composition of its digital collections and the
unique challenges they presented; the details of its pertinent technical
infrastructure including content management systems and repository software in
use; and a self-assessment summary and review of current digital curation and
preservation activities, if any. At each of the five campuses, team members
identified digital content known to be vulnerable to loss, but realized that
they had not developed programmatic solutions to mitigate that risk. Upon
gathering and reviewing the relevant information, team members identified the
practices and policies that they believed could most help their institutions to
improve their curation and preservation of digital objects. They then performed
a gap analysis by identifying the obstacles that had prevented their
organizations from achieving the desired outcomes. Common factors impeding
digital preservation efforts emerged from the gap analyses, including a lack of
available financial resources; limited or nonexistent staff time dedicated to
digital preservation activities; and insufficient levels of appropriate
technical expertise.[8]
The
case studies allowed the research team to perceive a state of affairs more
complex than the raw data described, however. Lacking the time necessary to
stay abreast of frequent developments in the field of digital preservation, the
expertise or technical infrastructure necessary to install and maintain complex
software solutions, and/or the funds to pay for ready-to-use products that may
exist, librarians and archivists at POWRR institutions often felt overwhelmed.
Faced with what seemed to be an enormous undertaking, and hesitant to commit
scarce resources to the adoption of an integrated digital preservation utility,
many found themselves unable to take even the first steps toward curating and
preserving their digital materials.[9]
Micro-services and
Smaller Institutions
Soon
after beginning their investigation, members of the Digital POWRR team discovered
a basic misunderstanding thwarting their progress towards the development of
more effective digital curation and preservation programs. They had understood
digital curation and preservation as a black and white issue: either an
institution had implemented successful digital curation and preservation
measures or it had not. Behind this assumption lay another: that successful
digital preservation activities required the use of a single application
integrating a number of necessary tasks behind a single user interface. Their
research soon led them to realize that recent scholarship has emphasized that
practitioners should rather understand digital curation and preservation
activities as an ongoing, iterative set of actions, reactions, workflows, and
policies.[10]
Such an approach means that practitioners do not have to begin digital curation
activities by creating or selecting a comprehensive technical solution
appropriate for large sets of digital materials over a long period of time.
Instead, they can start by taking small first steps to triage digital
collections and identify means by which they might cumulatively enhance the
curation and preservation of those found to require immediate attention, while
working to bring the issue to the attention of colleagues and administrators
and, ultimately, advocate for resources necessary for the implementation of
additional capacity. Put another way, information professionals at institutions
lacking effective digital curation and preservation measures can benefit by
focusing their efforts on the activities by which they can curate and preserve digital content in the next six to
twenty-four months rather than waiting a decade to devise an ideal solution.
Professionals in the cultural heritage sector accustomed to thinking in terms
of centuries may find this to be a curious approach, but to wait is to risk the
loss of unique materials.
The National Digital Stewardship
Alliance’s (NDSA) Levels of Digital Preservation can provide practitioners
seeking to employ this approach with a yardstick by which they might measure
their progress toward improved digital curation and preservation capacity. The
NDSA describes the Levels as a work in progress, intended to provide a readily-usable
set of guidelines useful for professionals and/or institutions in various
situations, ranging from those just beginning to think
about how to curate and preserve their digital assets more effectively to those
planning the next steps in enhancing existing systems and workflows. The
guidelines speak to five functional areas that represent the core of digital
preservation systems: storage and geographic location, file fixity and data
integrity, information security, metadata, and file formats. The Levels do not
help to assess the efficacy of digital preservation programs as a whole since
they do not consider important matters such as policies, staffing, or
organizational support.[11]
In addition to providing information
professionals at smaller institutions with an understanding of their present
digital curation and preservation capacities, reference to the NDSA Levels of
Digital Preservation can help them to recognize discrete steps by which they
can improve their curation and preservation of digital objects. These may
include activities such as improving storage practices from reliance on stand-alone
media (e.g. CDs and portable hard drives) to the use of networked or
geographically distributed servers, which the Levels mark as moving a single
square to the right in one of their functional categories. Researchers have
made resources discussing fundamental activities like these freely available.[12]
Institutions
can begin to move closer to a goal of stabilizing and preserving digital
materials by taking a number of the small steps forward as described in the
Levels. They should not hesitate to take Level 1 actions that they could
readily achieve today while they devote their energies to deliberations
considering how to move from Level 2 to Level 3. Thinking of digital curation
and preservation activities as an ongoing activity, and the NDSA Levels of
Preservation as a measure of progress, resonates with two fundamental
understandings at the heart of a micro-services approach: first, that digital
curation and preservation is an uncertain process in which continuous, rapid
technological change often renders monolithic, integrated applications
cumbersome and outdated; and second, that simple tools focused on a specific
aspect or aspects of the process, often available at no charge, can prove more
helpful.
Practitioners can only make use of
individual micro-services tools if they understand which roles they play in the
larger digital curation and preservation process, and how they might contribute
to progress through the Levels. To this end, the Digital POWRR Project depicted
the several stages of a prospective workflow as a pathway to digital
preservation.[13]
See below.
POWRR’s
Overview of the Path to Digital Preservation
Micro-services tools typically perform only a
single or limited number of the tasks or functions that make up each stage of
this process. For example, functions
within the ingest portion of a digital preservation workflow employing
micro-services might include file copying; fixity checking; virus scanning;
file de-duplication; and unique identifier generation.[14] The figure below provides a list of micro-services functions making up the more general
pathway to digital preservation.
Functionality within POWRR's Path to Digital Preservation
An understanding of the discrete
tasks and functions that comprise each stage of a digital curation and
preservation workflow can provide practitioners at medium-sized and smaller
institutions lacking large financial resources with an opportunity to assess
how adding specific functions to their existing practices can benefit them.
Likewise, a reliable registry of available digital curation and preservation
tools, including those providing only a single or limited number of the
functions depicted in Figure 3, can help information professionals to determine
how they can perform those functions most readily in local circumstances. The COPTR (Community Owned Digital
Preservation Tool Registry) web site (coptr.digitpres.org) provides information
about hundreds of tools and services that address some aspect of digital
curation and preservation, ranging from single-function applications to more
robust and complex utilities. The costs of the tools/services noted in COPTR
vary from those that are freely-available via open-source communities to those
that are cost-prohibitive for many smaller institutions. Practitioners should
realize that open-source applications may require programming expertise.
COPTR’s description of an individual tool includes a brief descriptions of its
general usability and the technical expertise it requires, as well as a record
of recent development activity. The Digital POWRR Project study found that in
many cases active user groups supporting a particular tool already exist
online. Many tool developers also make themselves available to individuals and
groups wishing to learn to use their application. [15]
Understanding digital curation and
preservation activities as incremental, and using tools lending themselves to
this approach, information professionals at medium-sized and smaller
institutions lacking large financial resources can make the small first steps
needed to begin to make progress through the NDSA Levels of Digital
Preservation. For example, practitioners may accession and inventory a
collection of digital materials using a free, simple ingest tool called Data
Accessioner and a common spreadsheet application. To learn more about how to do
this, visit the POWRR website.
Meg Miner of Illinois Wesleyan
University, a member of the Digital POWRR Project team, described Data
Accessioner’s usefulness in performing the functions described above in Figure
3 as Auto Metadata Harvest and Package Metadata thus:
I use Data Accessioner (DA) to
capture technical metadata as I move files from transfer media to my as-yet
non-bit-level storage device. I use DA-MT (Data Accessioner Metadata
Transformer) to aggregate the file information from xml to something I can
understand: file types, quantities and sizes by type. I store the aggregate
information in my regular accession files (currently a spreadsheet). My
accession information and an Access copy are in a different hard drive from the
Master copy and XML. Someday I will move the accessions with content I think is
most at-risk (due to format or other unique attribute) into a bit-checking
storage environment…. this workflow costs me no money, no technical expertise
(beyond downloading Java and two processing files via ZIP) and very little extra
time. With DA, I am capturing all the recommended technical information for use
by a back-end preservation system. With DA-MT I can track growth rate of
digital content overall, make a case for purchasing better storage, and keep an
eye on where all the at-risk file types are in the interim.[16]
In this case, use of
a single, open-source tool requiring no programming skills or other technical know-how
has helped a single archivist, working alone, to gather important information
about her institution’s digital collections which, as she notes, will prove
indispensable in the longer-term construction of a larger digital curation and
preservation system. The discrete, specialized tools that characterize a
micro-services approach also can ultimately help this practitioner and others
like her to build such a system, adding new functions and capacity to an
emerging digital curation and preservation workflow customized to address local
needs and idiosyncrasies, including the particular characteristics of
individual collections.
Micro-services
tools can also help practitioners adopting more robust tools. They should not
assume that an application bundling many digital curation and preservation
functions together with a single user interface will necessarily provide an
entirely comprehensive and worry-free experience. In many cases developers of
more extensive applications have assumed that their users have already
performed NDSA Level 1 triage activities (e.g. moving data from disparate
storage media to more secure locations and performing minimal inventory and/or
simple metadata creation). For example, POWRR researchers found that Curator’s
Workbench provided mechanisms for importing MODS records that its developers
assumed already existed, but not an intuitive interface for creating MODS
records from scratch. This presents information professionals at many
institutions with a significant gap between their digital materials’ present
condition and that state necessary even to begin curation and preservation
activities with their new utility. While individual micro-services tools can
enable practitioners to build a local digital curation and preservation system
in an incremental fashion, some of the most basic tools can also prepare
digital objects for ingestion into and storage within more elaborate
applications.
Research
libraries and archives presently face the difficult prospect of devising technical
means by which they may curate and preserve the large volume of digital objects
that their stakeholders and staff members have created, and will create.
Guidelines and standards describing effective practices presently exist, but
many information professionals have encountered great difficulty in providing
the infrastructure necessary to follow them and provide enhanced levels of curation
and preservation. The California Digital Library (CDL) has described and
implemented what it describes as a micro-services approach to the problem.
Dismissing the familiar means by which information professionals have addressed
other large challenges in their field, that of selecting and installing a
large, integrated application, CDL staff members have advocated the development
and strategic deployment of independent but interoperable utilities devoted to
performing specific, finite functions contributing to the larger curation and
preservation process. They have argued that administrators and developers
responsible for digital curation and preservation functions may devise, deploy,
maintain, upgrade, adapt and replace discrete, simple utilities far more easily
than a single application combining many functions behind a single user
interface. In addition, a distributed system allows units of a large university
system, varying widely in their practice, to adapt it to their use. Functioning
together, twelve micro-services utilities have provided the university’s ten
campuses with a comprehensive digital curation and preservation infrastructure
known as Merritt.
A
review of digital curation and preservation challenges facing smaller
institutions lacking large financial resources suggests that the micro-services
approach can benefit them as well. The same benefits that recommended it to the
University of California apply to smaller institutions. Information
professionals serving them may deploy, maintain, upgrade, adapt and replace
simple applications performing specific functions more readily than they might
install, manage, improve and, in a worst case, abandon and replace, a
comprehensive digital curation and preservation system. Due to its modular
nature and the amount of open-source applications available for use in a
micro-services arrangement, units within these institutions stand a better
chance of adapting one to their needs than an integrated, more comprehensive
system. Information professionals employed at medium-sized and smaller
institutions lacking large financial resources can also benefit from a micro-services
approach to digital curation and preservation for another reason. Many have often
reached the same misconception that the designers of the micro-services
approach originally noted: that an all-or-nothing approach, taking years to
select and implement, represented the only approach to digital curation and
preservation services. Believing that they lacked knowledge of the available
applications, information professionals at smaller institutions most often declined
to seek the significant financial resources necessary to make a selection, and
took no other constructive steps toward enhancing their digital curation and
preservation practices. A micro-services approach provides them with an
opportunity to break this pattern of inaction and begin curation and
preservation activities.
In
this context the Digital POWRR Project study encouraged information
professionals at medium-sized and smaller institutions to take basic, simple measures
mitigating their materials’ risk of loss, no matter how limited their scope and
effectiveness might seem. Measuring their progress by reference to the National
Digital Stewardship Alliance’s Levels of Digital Preservation, smaller
institutions might begin to improve their curation and preservation of digital
collections through a series of small steps. The use of individual tools
performing discrete functions can allow practitioners beginning curation and
preservation activities where none have previously existed to perform initial triage
on their digital collections and improve their practice over time, in part by expanding
and customizing their technical infrastructure incrementally, building a set of
applications best suited to local needs. Tools performing an individual or
limited number of specific functions contributing to enhanced levels of digital
curation and preservation can prove especially useful in this context. Aware
that many practitioners that might benefit from these tools have little or no
awareness of their existence or functions, the Digital POWRR Project made
information describing the several stages of a digital curation and preservation
workflow, the discrete activities making up each, and tools performing these functions
freely available online via the COPTR web site.
Notes
[1] See, for example, the Digital
Curation Centre’s Curation Lifecycle
Model, which features checklists for practitioners, at www.dcc.ac.uk/resources/curation-lifecycle-model.
Also see Reference Model for an
Open Archival Information System (OAIS): Recommended Practice CCSDC 650.0-M-2
(2012) at http://public.ccsds.org /publications/archive/650x0m2.pdf and Space Data and Information Transfer Systems – Audit and Certification
of Trustworthy Digital Repositories (ISO 16363:2012), accessed July 9,
2015, www.iso.org/obp/ui /#iso:std:iso:16363:ed-1:v1:en.
[2] “The University of California at
a Glance,” The University of California, 2015, accessed July 9, 2015, www.universityofcalifornia.edu/sites/default/files/uc_at_a_glance_011615.pdf.
[3] Stephen Abrams, Patricia Cruze, and
John Kunze, “Preservation is Not a Place,” International
Journal of Digital Curation 1, no. 4 (2009):8-21; Stephen Abrams, John
Kunze, and David Loy, “An Emergent Micro-Services Approach to Digital Curation
Infrastructure,” International Journal of
Digital Curation 1, no. 5 (2010): 173-174.
[4] Abrams, Kunze, Loy, “An Emergent
Micro-services Approach,” 184.
[5] www.cdlib.org/cdlinfo/2010/09/16/deposit-save-share-find-that-content-and-data-new-uc3-services-launch/, accessed May 6, 2015.
[6] https://merritt.cdlib.org/docs/merritt_handout.pdf; Meritt Service Update: March
2015, accessed May 6, 2015, www.cdlib.org/cdlinfo/2015/04/24/merritt-service-update-march-2015/.
[7] Tyler Walters and Katherine
Skinner. 2011. New Roles for New Times: Digital Curation for Preservation. (Washington,
DC: Association for Research Libraries, 2011), 53.
[8] Jaime Schumacher The Digital POWRR Project: A Final Report to
the Institute of Museum and Library Services, accessed July 9, 2015, http://hdl.handle.net/10843/13678.
[9] A. K. Rinehart,
P-A. Prud’homme, and A. R. Huot, “Overwhelmed to Action: Digital Preservation Challenges at the Under-RIsourced
institution,” OCLC Systems &
Services, 30, no. 1 (2014):28-42. www.emeraldinsight.com /journals.htm?issn=1065-075x&volume=30&issue=1&articleid=17106334&show=html;
M. Proffitt, “Something’s
Got to Give: What Can We Stop Doing in a
Time of Reduced Resources?” RBM: A Journal of Rare Books, Manuscripts, &
Cultural Heritage, Fall 2011, no. 12: 89-91, accessed July 9, 2015, http://rbm.acrl.org/content /12/2/89.full.pdf+html
[10] Gordan J. Daines,
III, “Module 2: Processing Digital Records and Manuscripts,” Archival Arrangement
and Description, ed. Christopher J. Prom
(Chicago: Society of American Archivists, 2013), 87-143.
[11] National Digital Stewardship
Alliance. The NDSA Levels of Digital Preservation, accessed July 9, 2015, www.digitalpreservation.gov/ndsa/activities/levels.html.
[12] See, for example, Julianna
Barrera-Gomez and Ricky Erway Walk This
Way: Detailed Steps for Transferring Born-Digital Content from Media You Can
Read In-house (Dublin, OH: OCLC Research, 2013 ) www.oclc.org/content/dam/research/publications/library/2013/2013-02.pdf.)
Also see the DP 101 page on the Digital POWRR
Project website.
[13] Jaime Schumacher, Lynne Thomas and Drew VandeCreek From Theory to Action: Good Enough Digital
Preservation Solutions for Under-Resourced Cultural Heritage Institutions,
(Washington, DC: Institute for Museum and Library Services, 2014), 6. http://commons.lib.niu.edu/bitstream/10843/13610/1 /FromTheoryToAction_POWRR_WhitePaper.pdf
[14] Schumacher, Thomas and VandeCreek, From Theory to Action, 8-13.
[15] Ibid., 11.
[16] Meg Miner, “Preservation
Processing Update.” Digital POWRR Project (blog), October 25, 2014, http://digitalpowrr.niu.edu/processing-update/
