Steven J. Murdoch, University College London
If you want a computer to require that actions meet certain criteria, the actions and criteria must be precisely specified in a programming language. However, many real-world tasks require human interpretation to enforce policies, or rely on information not available to the computer at the time of the decision.
One option is to let anything happen and trust people to act in a trustworthy way, but not everyone should be trusted. We can do better through transparency enhancing technologies that ensure actions are visible so failures can be identified through audit logs. However, there are challenges with this approach. For example, audit logs might contain sensitive information so cannot be disclosed. One technique to address this challenge was demonstrated in VAMS: allowing statistics to be verified without having access to underlying data. However, even if information needed to identify a problem is available, there might be other obstacles to mitigating the damage.
The legal system is the way we usually turn evidence into justice, but it’s imperfect and has proved particularly problematic where computers are used. For example, consider the prosecutions of 900+ subpostmasters based on evidence generated by the Horizon accounting system finally shown to be not “remotely robust”. Part of the problem is that the English legal system presumes that computers are reliable unless shown otherwise, and obtaining evidence that a computer is unreliable is expensive and may be infeasible, particularly for users.
To address the problem we need systems that produce accurate and interpretable evidence to support the cost-effective resolution of a dispute. If the failure of a system to resolve a dispute could result in serious harm, then it is an evidence-critical system. This is related to safety-critical systems – where a failure to operate correctly could result in serious harm – but is not the same. High assurance engineering techniques, expected for safety-critical systems, are expensive even for the most straightforward applications of computers, so it could be argued as unrealistic for all legally relevant computer systems. Actually, the situation is not so bad: safety-critical systems must produce correct and timely responses. Evidence-critical systems need only to never produce an undetectable incorrect response: it’s OK to fail to produce a response, and it’s OK if an incorrect response can be detected.
The Evidence Critical Systems research project aims to identify the right technologies and design principles to build systems that will produce adequate evidence to resolve disputes fairly and address the challenges in presenting and interpreting this evidence. For example:
For (infrequent) updates on this project, subscribe to our mailing list by sending a blank email to firstname.lastname@example.org. If you have comments or suggestions, please email email@example.com.