“Because of their statistical basis, all ML systems have error rates. Even though in many cases ML systems are far more accurate than human beings, there is danger in assuming that simply because a system’s predictions are more accurate than a human’s, the outcome is necessarily better. Even if the error rate is close to zero, in a tool with millions of users, thousands could be affected by error rates. Consider the example of Google Photos. In 2015 Google Photos’ image recognition software was found to have a terribly prejudicial and offensive error: it was occasionally labeling photos of black people as gorillas. Because the system used a complex ML model, engineers were unable to figure out why this was happening. The only “solution” they could work out to this “racist” ML was merely a band-aid: they removed any monkey-related words from the list of image tags.Now, imagine a similar software system used by U.S Customs and Border Patrol that photographs every person who enters and exits the U.S. and cross-references it with a database of photos of known or suspected criminals and terrorists. In 2016, an estimated
1. 9 million people arrived in the United States.31 Even if the facial recognition system was
1. 9% accurate, the
1. 1% error rate would result in 75,900 people being misidentified. How many of these people would be falsely identified as wanted criminals and detained? And what would the impact be on their lives? Conversely, how many known criminals would get away? Even relatively narrow error rates in cases such as these can have severe consequences.” (Access Now, 2018, p. 13)