Too early for audits?

I never realized that the procedure of getting to an ISO standard could take several years, but this is true for two standards related to audit and certification of trustworthy digital repositories.  Although we have the ISO 16363 standard on Audit and Certification since 2012, official audits cannot take place against this standard until the related standard Requirements for bodies providing Audit and Certification (ISO 16919) is approved, regulating the appointment of auditors. This standard, similar to the ISO 16363 compiled by the PTAB group in which I participate, was already finished a few years ago, but the ISO review procedure, especially when revisions need to be made, takes long. The latest prediction is that this summer (2014) the ISO 16919 will be approved, after which national standardization bodies can train the future (official) auditors.  How many organizations will then apply for an official certification against the ISO standard is not yet clear, but if you’re planning to do so, it might be worthwhile to have a look at the recent report of the European 4C project  Quality and trustworthiness as economic determinants in digital curation.

The 4C project (Collaboration to Clarify the Cost of Curation) is looking at the costs and benefits of digital curation. Trustworthiness is one of the “economic determinants” of the 15 they distinguish. As quality is seen as a precondition for trustworthiness, the 4C project focusses in this report on the costs and benefits of “standards based quality assurance” and looks at the 5 current standards related to audit and certification: DSA, Drambora, DIN 31644 of the German nestor group, TRAC and TDR. The first part of the report gives an overview of the current status of these standards. Woven in this overview are some interesting thoughts about audit and certification. It all starts with the Open Archival Information System (OAIS) Reference Model. The report suggests that the OAIS model is there to help organisations to create processes and workflows (page 18), but I think this does not right to the OAIS model. If one really reads the OAIS standard from cover to cover (and should not we all do that regularly?) one will recognize that the OAIS model expects a repository to do more than designing workflows and processes. Instead, a repository needs to develop a vision on how to do digital preservation and the OAIS model gives directions. But the OAIS model is not a book of recipes and we all are trying to find the best way to translate OAIS into practice. It is this lack of evidence which approach will offer the best preserved digital objects, that made the authors in the report wonder whether an audit that will take place now might lead to a risky outcome (either too much confidence in the repository or too little). They use the phrase “dispositional trust” . “It is the trustor’s belief that it will have a certain goal B in the future and, whenever it will have such a goal and certain conditions obtain, the trustee will perform A and thereby will ensure B.”(p. 22). We expect that our actions will lead to a good result in the future, but this is uncertain as we don’t have an agreed common approach with evidence that this approach will be successful.  This is a good point to keep in mind I think as well as the fact that there are many more standards applicable for digital preservation then only the above mentioned. Security standards, record management standards and standards related to the creation of the digital object, to name just a few.

Based on publicly available audit reports (mainly TRAC and DSA, and test audits on TDR) the report describes the main benefits of audits for organisations as

  • to improve the work processes,
  • to meet a contractual obligation and
  • to provide a publicly understandable statement of quality and reliability (p. 29).

These benefits are rather vague but one could argue that these vague notions might lead to more tangible benefits in the future like more (paying) depositors, more funding, etc. By the way, one of the benefits recognized in the test audits was the process of peer review in itself and the ability for the repository management to discuss the daily practices with knowledgeable people.

The authors also tried to get more information about costs related to audit and certification, but had to admit in the end that currently there is hardly any information about the actual costs of an audit and/or get certified (why they mention on page 23 financial figures of 2 specific audits without any context is unclear to me) and base themselves mainly on information that was collected during the test audits that the APARSEN project performed and the taxonomy of costs that was created. For costs we need to wait for more audits and for repositories that are willing to publish all their costs in relation to this exercise.

Reading between the lines,  one could easily conclude that it is not recommended to perform audits yet. But especially now the DP community is working hard to discover the best way to protect digital material, it is important for any repository to protect their investments and to avoid that current funding organizations (often tax payers) will back off because of costly mistakes. The APARSEN trial audits were performed by experts in the field and the audited organizations (and these experts) found the discussions and recommendations valuable. As standards are evolving and best practices and tools are developed, a regular audit by experts in the field can certainly safeguard organizations to minimize the risk for the material. These expert auditors need to be aware of the current state of digital preservation, the uncertainties, the risks, the lack of tools and the best practices that are there. The audit results  will help the community to understand the issues encountered by the audited organizations, as audit results will be published.

As I noticed while reading a lot of preservation policies for SCAPE, many organisations want to get certified and put this aim in their policies. Publishers want to have their data and publications in trustworthy, certified repositories. But all stakeholders (funders, auditors, repository management) should realise that the outcomes of an audit should be seen in the light of the current state of digital preservation: that of pioneering.

© 2022 Barbara Sierman