In order to perform as desired, autonomous systems must collect data that are complete, relevant, accurate, and aligned with the data for which the system was developed and tested. But the harsh, dynamic, complex and adversarial nature of conflict environments poses a wide range of obstacles to the collection of such data. As a result, autonomous systems cannot always be expected to achieve the exact same performance in real-world use that they demonstrated in development or testing. And crucially, they will be liable to failures that are both inevitable and impossible to anticipate: “known unknowns.”
Data and its vagaries therefore have significant implications for the application of international humanitarian law and other rules of war. This report describes common data issues for autonomous systems and explains how they give rise to "known unknown" failures. It then explores the legal and operational implications of such failures, and considers a range of potential policy and technical solutions by which they could be addressed.
Citation: Arthur Holland Michel (2021) "Known Unknowns: Data Issues and Military Autonomous Systems", UNIDIR, Geneva: https://doi.org/10.37559/SecTec/21/AI1
EXECUTIVE SUMMARY
This publication is an executive summary for UNIDIR’s report, Known Unknowns: Data Issues and Military Autonomous Systems.
To read the executive summary click here.
Teaser: Data Issues and Military Autonomous Systems
Sponsor Organizations: The Governments of Germany, the Netherlands, Norway and Switzerland, and Microsoft