Thursday, 16 November 2017

Retconning this Blog

I've got a load of bits and pieces laying around the internet that I've posted over the years. I'm going to collect them all here with their original dates. I would like to claim it's for preservation but frankly, it just that I can't keep track of the stuff.

Wednesday, 15 November 2017

Thoughts on Fixity Checking in Digital Preservation Systems

I would like to query the rationale for actually doing periodic fixity checking in isolation. This has bugged me for a bit so I am going to unload.

As far as I can see, the main reasons would be undetected corruption on storage and tampering that doesn’t hijack the chain of custody.

All storage media now have built-in error detection and correction using Reed-Solomon, Hamming or something similar which is generally capable of dealing with small multi-bit errors. In modern environments, this gives unrecoverable read error rates of at worst around 1 in 10^14 and generally several orders of magnitude better – which is around one in 12TB total read. Write errors are less frequent – they do occur but can be detected by device firmware and retried elsewhere on the medium. These are absolute worst case figures and result in *detectable* failure long before we even get to computing fixity. The chance of bit flips occurring in such a pattern as to defeat error correction coding is several orders of magnitude less – it is similar to bit flips resulting in an unchanged MD5 hash. Interestingly, in most cases the mere act of reading data allows devices to detect and correct future errors as the storage medium becomes marginal so there is value in doing that.

Consequently, however, undetected corruption is most likely when data moves from the error corrected environment of the medium to less robust environments. At an interconnect level protocols such as SCSI, SATA, Ethernet and FC are all error corrected as is the PCI-E bus itself. The most likely failure points are likely to be a curator’s PC or software. How many curators work on true workstation grade systems with error corrected RAM and error corrected CPU caches? How well tested are your hashing implementations (MD5 had a bug not so long ago)? How about all the scripts that tie everything together? How about every tool in your preservation toolchain? How many of these fail properly when an unrecoverable media error is encountered?

If we consider malicious activity then, again, we have to ask whether it is easier to attack the storage (which may require targeting several geographically dispersed and reasonably secure targets) or the curation workflow, which is localised, generally in a less secure location than a machine room, and can legitimise changes. A robust digital signature environment is the way to deal with this – and fixity hashes *can* be used to make this more efficient (sign the hash rather than the whole object).  

Locally computed hashes can be very useful as a bandwidth efficient way of comparing multiple copies of an object (rsync has done this for ages) to ensure that they are in sync.

So there are reasons to compute hashes, when needed, but fixity is not necessarily a compelling reason given the way modern systems are engineered.

In practice, these checks do detect failures but they are almost exclusively transmission errors as a result of uncontrolled (and unauditable) activities - often by sysadmins or third party suppliers not well versed in digital preservation. In these cases, the wrong data is actually written to storage so there is no fixity to lose. Periodic "fixity" checking can catch these cases, but ideally you want to have visibility of these processes and checking immediately after completion. If the errors are in automated processes, waiting for a periodic check to come round may allow significant damage to occur.

Originally posted to the PASIG mailing list, with updates as a result of discussion with Kyle Rimkus (University of Illinois at Urbana-Champaign).

...also now posted on the DPC blog.

Monday, 6 November 2017

ORCID Token Revocation

At the last Cultivating ORCIDs Meeting in Birmingham in June 2017, I ran a working group looking at different approaches to implementing ORCID IDs. One of the outcomes was the identification of a common issue when it came to ORCID implementations and third party suppliers, namely, that institutional users needed to explicitly grant access to third party suppliers in addition to their own institution. This behaviour has a number of undesirable side effects:
  • Communicating this to users can be difficult since they are not always aware of these third parties
  • Getting consistent takeup across multiple systems can be difficult (user loses interest) which makes downstream integration more awkward than necessary
  • Institution has little visibility of these third party interactions – which can cause problems when suppliers are dropped or other issues arise
  • The only way currently round this is to let a supplier use the institutional key – which however then grants them *ALL* the rights can access that the institution has
On the 10th October a small group of interested parties (with representatives from Oxford, Imperial, Leeds, LSE, ORCID and Jisc) were hosted by Jisc in a small gathering to look at this issue and identify a possible route forward for consideration by the UK ORCID Consortium.

At the meeting, Will Simpson of ORCID presented a very useful non-technical overview of how authentication and ORCID/OAuth tokens worked in terms of managing access permissions. Discussion then moved on to the main topic of how ORCID permissions might be delegated to third party providers and, in particular, how to handle the termination of third party arrangements. During these discussions, Will indicated that support for the optional OAuth functionality for token revocation was being considered by ORCID. OAuth is the technology/standard that ORCID uses for authorisation/access control. At the moment, tokens are granted by default for 20 years, or 1 hour for effectively single, short term, use. Naturally, neither of these match the typical duration of a scholar’s relationship with an institution. Minimising the number of active tokens would be good from both a security and “data hygiene” standpoint, so the ability for an institution to relinquish their token when a scholar leaves would be useful in its own right. Scholars can revoke their tokens manually when they leave but it is unrealistic to rely on them to remember to do so.

At the moment, it is possible to work around this situation by making creative use of the OAuth token refresh facility. This functionality is important since it is what will allow an institution to grant tokens to a third party on behalf of an individual researcher (which will be explored in the next posting), but, in this context, it does provide a slightly unorthodox method for effectively relinquishing a token. Intended for use when an existing token nears expiry, a replacement token may be requested with a new expiry date which then invalidates the previous token. However, this can *actually* be done at any time and a 20-year token *can* be replaced by a 1 hour token which can simply be allowed to expire, resulting in no active tokens.

This is a concatenation of two articles posted on the UK ORCID Consortium blog.

Monday, 25 September 2017

Data2paper Poster for 10th RDA Plenary


https://drive.google.com/file/d/12YbGFt2Yfx1k3rz6v76gRfTOJrqaw8LV/view?usp=sharing

Data2paper is a cloud-based application to automate the process of compiling and submitting a data paper to a journal without the researcher having to leave the research space or wrestle directly with the journal’s submission system.

Serving the wider research community, data2paper works with academic institutions, publishers, data repositories, funding agencies and organizations interested in research and scholarly communication. Data2paper aims at progressing data papers, enabling data re-use and giving researchers credit for their data.

The development of the original idea was funded as part of the Jisc Data Spring initiative.

SWORDV3 Poster for 10th RDA Plenary




SWORD (Simple Web-service Offering Repository Deposit) is a lightweight protocol for depositing and updating content from one location to another. The SWORD vision is ‘lowering the barriers to deposit‘, principally for depositing content into repositories, but potentially for depositing into any system which wants to receive content from remote sources.

The SWORDV3 project has been funded by Jisc with two key aims for the next generation of the SWORD Protocol:
  • To bring SWORD up-to-date with the developments in the repository sphere in the last 5 years, with alignment to new protocols and new use-cases such as data publishing and complex objects.
  • To establish community and governance mechanisms for the standard and supporting code libraries to ensure ongoing maintenance and evolution. This will include a technical validation process to allow third party libraries to be hosted under the SWORD brand.
In September, the project was formally announced at PASIG 2017 in Oxford and the RDA 10th Plenary in Montreal. We are looking for expressions of interest in becoming involved as stakeholders in the project: to make suggestions, review activities and meet as required over the coming months. In particular, we are interested in making contact with people who may wish to develop SWORD V3 libraries for their preferred platforms or languages since we aim to provide some support for such activities during the project. Please contact one of the project team  if you are interested in participating, and indicate if you are interested in the technical or community aspects of the project (or both!).

The key members of the project team are:

Monday, 19 June 2017

Rethinking Digital Preservation (for Research Libraries)

https://drive.google.com/file/d/1Xq8LFTiBKBdShEh2wbBgmGNyZql5k4F-/view?usp=sharingPresentation given at the Digital Preservation for the Arts Social Sciences and Humanties conference. I will expand a lot of the slides in future articles in that I discuss the difference between the preservation of dgital knowledge as opposed to digital objects. Knowledge is  dynamic, evolving and context dependent and so requires a rather different approach that is less focussed on static fixity and more process oriented.