Why not just use Fedora?
Fedora is the storage layer. It is our gateway to OCFL.
Fedora 6 persists its repository as OCFL and manages versions for us. The three resource types at the top of the Concepts page - Archival Group, Container, Binary - are Fedora’s own, and a Fedora Archival Group is an OCFL object. If Fedora already provides versioned OCFL objects with an HTTP API, why put two more APIs and a workflow in front of it?
Fedora is a repository, and this is digital preservation. Fedora will store whatever you send it, one request at a time. It has no opinion about what a preserved object is, what must be known about every file before it goes in, or what has to happen to a file on the way. Those opinions (and more) are what makes the platform a digital preservation platform.
What the Preservation API provides
Section titled “What the Preservation API provides”An Import Job is one atomic operation
Section titled “An Import Job is one atomic operation”Fedora’s API is one resource per request: PUT a Binary, PUT a Container, PUT another Binary. For a digitised book that is hundreds of requests, and with an Archival Group every one of them is a commit - and every commit is a new OCFL version. Fedora avoids this with transactions, but then you are managing a transaction yourself across hundreds of calls, keeping it alive, and rolling it back cleanly if the four-hundredth file fails.
The platform gives you a unit of change: the Import Job. An Import Job is a JSON document that says everything that should happen to an Archival Group - Containers to add, Binaries to add, patch, delete or rename. The Storage API performs the whole job inside a single Fedora transaction, however many files are involved, and commits once. The result is exactly one new OCFL version per job, or no new version at all if anything failed.
An Import Job can be a diff
Section titled “An Import Job can be a diff”Fedora cannot tell you what has changed. If you have a folder of files that represents the new state of an object, working out which of them are new, which have changed, which have gone and which have merely been renamed is your problem.
The Preservation API can generate the Import Job for you by comparing a working Deposit with the current version of the Archival Group. It does not have to see every file in the workspace to do this: a file that is mentioned in the METS but not present in the current Deposit workspace is taken to be unchanged. This makes updating without a full export work efficiently.
It imposes constraints that Fedora does not
Section titled “It imposes constraints that Fedora does not”Fedora will store a file it has never been given an expected checksum for, record its type as “bytes”, and know it only by the URI you chose to put it at. The platform will not. Before an Import Job reaches Fedora it must supply, for every Binary:
- a SHA-256 digest, which the platform checks against the checksum S3 recorded on upload, hands to Fedora so that Fedora verifies the bytes it received, and compares with what Fedora then recorded;
- a content type;
- a URI-safe slug and the original name, so that “my notes.doc” survives as a name without becoming a path.
Other constraints include:
- An Archival Group managed via the Preservation API must have a METS file at its root.
- A Deposit may only put content under
objects/. - An Archival Group may have only one Import Job running at a time, and a Deposit is good for one.
Deposits: a workspace to assemble an object
Section titled “Deposits: a workspace to assemble an object”The Storage API needs to be told where files are. The Preservation API gives you somewhere to work on them: a Deposit is a workspace in S3 (or other storage), allocated by the API, in which an object is assembled, checked, described and when it is ready, preserved in one Import Job. The same mechanism works in reverse: an existing Archival Group is exported into a Deposit so that it can be changed and preserved again as a new version.
METS: the object model
Section titled “METS: the object model”Fedora describes its resources in RDF, and can hold arbitrary triples about them. The platform
deliberately uses none of that. Everything it knows about a preserved object lives in one file
inside the object itself, mets.xml: the fixity, size and PRONOM format of every file as PREMIS;
what the pipeline tools found; access conditions, rights and catalogue identifiers as MODS; and the
logical structure a viewer needs, as structMaps. The METS section describes this.
This makes the OCFL object self-describing. A copy of the S3 bucket and the OCFL specification is enough to recover everything, including the metadata - no Fedora database, no Preservation API database, no running system. Fedora’s own metadata is not needed to read the object back, and so is not relied on.
The platform also reads METS it did not write. Goobi produces its own; so do EPrints and Archivematica. The parser understands all of them well enough to find checksums and structure, which is how content from those systems is preserved without being reshaped first.
Tools, and a record of having run them
Section titled “Tools, and a record of having run them”Preserving a file is not just storing it. You want to know what format it is in, whether it is infected, and what a camera recorded when it took the picture - and you want the answer written down with the file, not in someone’s notes. The platform runs pipelines - Siegfried for format identification, ClamAV for virus scanning, ExifTool for embedded metadata, via Brunnhilde - over a Deposit, and writes what they found into the METS as PREMIS objects and events. Or you run these and any other tools yourself, in your own environment, and put the output in the same places; the platform reads it the same way. Either way the result is preserved alongside the content, as part of the object.
Fedora has no notion of a tool, or of an event of this type, because that is not its purpose as a repository.
An audience for what happened
Section titled “An audience for what happened”Every Import Job records who asked for it, and every new version is announced on an activity stream that downstream systems - a IIIF builder, a catalogue, a replication process - can follow. This allows for arbitrary systems integration using a modern, easy to consume feed.
What the Storage API adds
Section titled “What the Storage API adds”The Storage API is the only thing that talks to Fedora. It reduces Fedora to the four things the platform uses - Containers, Binaries, Archival Groups and transactions and presents them as a plain hierarchy, with Import Jobs as the only way to change it. It knows nothing about METS, Deposits or tools; a METS file is just another Binary to it.
What about Archivematica?
Section titled “What about Archivematica?”Archivematica is a complete digital preservation system: a pipeline of microservices that takes a transfer through identification, virus scanning, normalisation and packaging, and produces an Archival Information Package as a BagIt bag with its own METS, handed to a storage service. It is opinionated about the whole journey, including format migration.
This platform is narrower and sits lower. It does not normalise: what you deposit is what is preserved, as it is. It does not package: the preserved object is a Fedora Archival Group in OCFL, not a bag, and it is versioned in place rather than replaced. Its pipeline runs the same kinds of tool but only to describe the files, never to change them. METS is its object model, edited and diffed and read back, rather than a manifest written once at the end.
The two are not exclusive. An Archivematica AIP’s METS is one of the shapes the platform’s parser reads, so content that has been through Archivematica can be deposited and preserved here with its existing description intact.