Thursday 16 November 2017
Retconning this Blog
I've got a load of bits and pieces laying around the internet that I've posted over the years. I'm going to collect them all here with their original dates. I would like to claim it's for preservation but frankly, it just that I can't keep track of the stuff.
Wednesday 15 November 2017
Thoughts on Fixity Checking in Digital Preservation Systems
I would like to query the rationale for actually doing periodic fixity checking in isolation. This has bugged me for a bit so I am going to unload.
As far as I can see, the main reasons would be undetected corruption on storage and tampering that doesn’t hijack the chain of custody.
All storage media now have built-in error detection and correction using Reed-Solomon, Hamming or something similar which is generally capable of dealing with small multi-bit errors. In modern environments, this gives unrecoverable read error rates of at worst around 1 in 10^14 and generally several orders of magnitude better – which is around one in 12TB total read. Write errors are less frequent – they do occur but can be detected by device firmware and retried elsewhere on the medium. These are absolute worst case figures and result in *detectable* failure long before we even get to computing fixity. The chance of bit flips occurring in such a pattern as to defeat error correction coding is several orders of magnitude less – it is similar to bit flips resulting in an unchanged MD5 hash. Interestingly, in most cases the mere act of reading data allows devices to detect and correct future errors as the storage medium becomes marginal so there is value in doing that.
Consequently, however, undetected corruption is most likely when data moves from the error corrected environment of the medium to less robust environments. At an interconnect level protocols such as SCSI, SATA, Ethernet and FC are all error corrected as is the PCI-E bus itself. The most likely failure points are likely to be a curator’s PC or software. How many curators work on true workstation grade systems with error corrected RAM and error corrected CPU caches? How well tested are your hashing implementations (MD5 had a bug not so long ago)? How about all the scripts that tie everything together? How about every tool in your preservation toolchain? How many of these fail properly when an unrecoverable media error is encountered?
If we consider malicious activity then, again, we have to ask whether it is easier to attack the storage (which may require targeting several geographically dispersed and reasonably secure targets) or the curation workflow, which is localised, generally in a less secure location than a machine room, and can legitimise changes. A robust digital signature environment is the way to deal with this – and fixity hashes *can* be used to make this more efficient (sign the hash rather than the whole object).
Locally computed hashes can be very useful as a bandwidth efficient way of comparing multiple copies of an object (rsync has done this for ages) to ensure that they are in sync.
So there are reasons to compute hashes, when needed, but fixity is not necessarily a compelling reason given the way modern systems are engineered.
In practice, these checks do detect failures but they are almost exclusively transmission errors as a result of uncontrolled (and unauditable) activities - often by sysadmins or third party suppliers not well versed in digital preservation. In these cases, the wrong data is actually written to storage so there is no fixity to lose. Periodic "fixity" checking can catch these cases, but ideally you want to have visibility of these processes and checking immediately after completion. If the errors are in automated processes, waiting for a periodic check to come round may allow significant damage to occur.
Originally posted to the PASIG mailing list, with updates as a result of discussion with Kyle Rimkus (University of Illinois at Urbana-Champaign).
...also now posted on the DPC blog.
As far as I can see, the main reasons would be undetected corruption on storage and tampering that doesn’t hijack the chain of custody.
All storage media now have built-in error detection and correction using Reed-Solomon, Hamming or something similar which is generally capable of dealing with small multi-bit errors. In modern environments, this gives unrecoverable read error rates of at worst around 1 in 10^14 and generally several orders of magnitude better – which is around one in 12TB total read. Write errors are less frequent – they do occur but can be detected by device firmware and retried elsewhere on the medium. These are absolute worst case figures and result in *detectable* failure long before we even get to computing fixity. The chance of bit flips occurring in such a pattern as to defeat error correction coding is several orders of magnitude less – it is similar to bit flips resulting in an unchanged MD5 hash. Interestingly, in most cases the mere act of reading data allows devices to detect and correct future errors as the storage medium becomes marginal so there is value in doing that.
Consequently, however, undetected corruption is most likely when data moves from the error corrected environment of the medium to less robust environments. At an interconnect level protocols such as SCSI, SATA, Ethernet and FC are all error corrected as is the PCI-E bus itself. The most likely failure points are likely to be a curator’s PC or software. How many curators work on true workstation grade systems with error corrected RAM and error corrected CPU caches? How well tested are your hashing implementations (MD5 had a bug not so long ago)? How about all the scripts that tie everything together? How about every tool in your preservation toolchain? How many of these fail properly when an unrecoverable media error is encountered?
If we consider malicious activity then, again, we have to ask whether it is easier to attack the storage (which may require targeting several geographically dispersed and reasonably secure targets) or the curation workflow, which is localised, generally in a less secure location than a machine room, and can legitimise changes. A robust digital signature environment is the way to deal with this – and fixity hashes *can* be used to make this more efficient (sign the hash rather than the whole object).
Locally computed hashes can be very useful as a bandwidth efficient way of comparing multiple copies of an object (rsync has done this for ages) to ensure that they are in sync.
So there are reasons to compute hashes, when needed, but fixity is not necessarily a compelling reason given the way modern systems are engineered.
In practice, these checks do detect failures but they are almost exclusively transmission errors as a result of uncontrolled (and unauditable) activities - often by sysadmins or third party suppliers not well versed in digital preservation. In these cases, the wrong data is actually written to storage so there is no fixity to lose. Periodic "fixity" checking can catch these cases, but ideally you want to have visibility of these processes and checking immediately after completion. If the errors are in automated processes, waiting for a periodic check to come round may allow significant damage to occur.
Originally posted to the PASIG mailing list, with updates as a result of discussion with Kyle Rimkus (University of Illinois at Urbana-Champaign).
...also now posted on the DPC blog.
Monday 6 November 2017
ORCID Token Revocation
At the last Cultivating ORCIDs Meeting in Birmingham in June 2017, I ran a working group looking at different approaches to
implementing ORCID IDs. One of the outcomes was the identification of a common issue when it
came to ORCID implementations and third party suppliers, namely, that
institutional users needed to explicitly grant access to third party
suppliers in addition to their own institution. This behaviour has a
number of undesirable side effects:
At the meeting, Will Simpson of ORCID presented a very useful non-technical overview of how authentication and ORCID/OAuth tokens worked in terms of managing access permissions. Discussion then moved on to the main topic of how ORCID permissions might be delegated to third party providers and, in particular, how to handle the termination of third party arrangements. During these discussions, Will indicated that support for the optional OAuth functionality for token revocation was being considered by ORCID. OAuth is the technology/standard that ORCID uses for authorisation/access control. At the moment, tokens are granted by default for 20 years, or 1 hour for effectively single, short term, use. Naturally, neither of these match the typical duration of a scholar’s relationship with an institution. Minimising the number of active tokens would be good from both a security and “data hygiene” standpoint, so the ability for an institution to relinquish their token when a scholar leaves would be useful in its own right. Scholars can revoke their tokens manually when they leave but it is unrealistic to rely on them to remember to do so.
At the moment, it is possible to work around this situation by making creative use of the OAuth token refresh facility. This functionality is important since it is what will allow an institution to grant tokens to a third party on behalf of an individual researcher (which will be explored in the next posting), but, in this context, it does provide a slightly unorthodox method for effectively relinquishing a token. Intended for use when an existing token nears expiry, a replacement token may be requested with a new expiry date which then invalidates the previous token. However, this can *actually* be done at any time and a 20-year token *can* be replaced by a 1 hour token which can simply be allowed to expire, resulting in no active tokens.
This is a concatenation of two articles posted on the UK ORCID Consortium blog.
- Communicating this to users can be difficult since they are not always aware of these third parties
- Getting consistent takeup across multiple systems can be difficult (user loses interest) which makes downstream integration more awkward than necessary
- Institution has little visibility of these third party interactions – which can cause problems when suppliers are dropped or other issues arise
- The only way currently round this is to let a supplier use the institutional key – which however then grants them *ALL* the rights can access that the institution has
At the meeting, Will Simpson of ORCID presented a very useful non-technical overview of how authentication and ORCID/OAuth tokens worked in terms of managing access permissions. Discussion then moved on to the main topic of how ORCID permissions might be delegated to third party providers and, in particular, how to handle the termination of third party arrangements. During these discussions, Will indicated that support for the optional OAuth functionality for token revocation was being considered by ORCID. OAuth is the technology/standard that ORCID uses for authorisation/access control. At the moment, tokens are granted by default for 20 years, or 1 hour for effectively single, short term, use. Naturally, neither of these match the typical duration of a scholar’s relationship with an institution. Minimising the number of active tokens would be good from both a security and “data hygiene” standpoint, so the ability for an institution to relinquish their token when a scholar leaves would be useful in its own right. Scholars can revoke their tokens manually when they leave but it is unrealistic to rely on them to remember to do so.
At the moment, it is possible to work around this situation by making creative use of the OAuth token refresh facility. This functionality is important since it is what will allow an institution to grant tokens to a third party on behalf of an individual researcher (which will be explored in the next posting), but, in this context, it does provide a slightly unorthodox method for effectively relinquishing a token. Intended for use when an existing token nears expiry, a replacement token may be requested with a new expiry date which then invalidates the previous token. However, this can *actually* be done at any time and a 20-year token *can* be replaced by a 1 hour token which can simply be allowed to expire, resulting in no active tokens.
This is a concatenation of two articles posted on the UK ORCID Consortium blog.
Subscribe to:
Posts (Atom)