“AI can be biased both at the system and the data or input level. Bias at the system level involves developers building their own personal biases into the parameters they consider or the labels they define. Although this rarely occurs intentionally, unintentional bias at the system level is common. This often occurs in two ways:- When developers allow systems to conflate correlation with causation. Take credit scores as an example. People with a low income tend to have lower credit scores, for a variety of reasons. If an ML system used to build credit scores includes the credit scores of your Facebook friends as a parameter, it will result in lower scores among those with low-income backgrounds, even if they have otherwise strong financial indicators, simply because of the credit scores of their friends.- When developers choose to include parameters that are proxies for known bias. For example, although developers of an algorithm may intentionally seek to avoid racial bias by not including race as a parameter, the algorithm will still have racially biased results if it includes common proxies for race, like income, education, or postal code.26Bias at the data or input level occurs in a number of ways:27- The use of historical data that is biased. Because ML systems use an existing body of data to identify patterns, any bias in that data is naturally reproduced. For example, a system used to recommend admissions at a top university that uses the data of previously admitted students to train the model is likely to recommend upper class males over women and traditionally underrepresented groups.- When the input data are not representative of the target population. This is called selection bias, and results in recommendations that favor certain groups over another. For example, if a GPS-mapping app used only input data from smartphone users to estimate travel times and distances, it could be more accurate in wealthier areas of cities that have a higher concentration of smartphone users, and less accurate in poorer areas or informal settlements, where smartphone penetration is lower and there is sometimes no official mapping.- When the input data are poorly selected. In the GPS mapping app example, this could involve including only information related to cars, but not public transportation schedules or bike paths, resulting in a system that favored cars and was useless for buses or biking.- When the data are incomplete, incorrect, or outdated. If there is insufficient data to make certain conclusions, or the data are out of date, results will naturally be inaccurate. And if a machine learning model is not continually updated with new data that reflects current reality, it will naturally become less accurate over time.Unfortunately, biased data and biased parameters are the rule rather than the exception. Because data are produced by humans, the information carries all the natural human bias within it. Researchers have begun trying to figure out how to best deal with and mitigate bias, including whether it is possible to teach ML systems to learn without bias;28 however, this research is still in its nascent stages. For the time being, there is no cure for bias in AI systems” (Access Now, 2018, p. 12)“Designers and users ensure that the AI systems they are developing and deploying:
1. Are trained and tested on properly representative, relevant, accurate, and generalisable datasets (Data Fairness)
1. Have model architectures that do not include target variables, features, processes, or analytical structures (correlations, interactions, and inferences) which are unreasonable, morally objectionable, or unjustifiable (Design Fairness)
1. Do not have discriminatory or inequitable impacts on the lives of the people they affect (Outcome Fairness)
1. Are deployed by users sufficiently trained to implement them responsibly and without bias (Implementation Fairness)” (Leslie, 2019, p. 14)*Data Fairness*"Responsible data acquisition, handling, and management is a necessary component of algorithmic fairness. If the results of your AI project are generated by biased, compromised, or skewed datasets, affected stakeholders will not adequately be protected from discriminatory harm. Your project team should keep in mind the following key elements of data fairness:- Representativeness: Depending on the context, either underrepresentation or overrepresentation of disadvantaged or legally protected groups in the data sample may lead to the systematic disadvantaging of vulnerable stakeholders in the outcomes of the trained model. To avoid such kinds of sampling bias, domain expertise will be crucial to assess the fit between the data collected or procured and the underlying population to be modelled. Technical team members should, if possible, offer means of remediation to correct for representational flaws in the sampling.- Fit-for-Purpose and Sufficiency: An important question to consider in the data collection and procurement process is: Will the amount of data collected be sufficient for the intended purpose of the project? The quantity of data collected or procured has a significant impact on the accuracy and reasonableness of the outputs of a trained model. A data sample not large enough to represent with sufficient richness the significant or qualifying attributes of the members of a population to be classified may lead to unfair outcomes. Insufficient datasets may not equitably reflect the qualities that should rationally be weighed in producing a justified outcome that is consistent with the desired purpose of the AI system. Members of the project team with technical and policy competences should collaborate to determine if the data quantity is, in this respect, sufficient and fit-for-purpose.- Source Integrity and Measurement Accuracy: Effective bias mitigation begins at the very commencement of data extraction and collection processes. Both the sources and instruments of measurement may introduce discriminatory factors into a dataset. When incorporated as inputs in the training data, biased prior human decisions and judgments such as prejudiced scoring, ranking, interview-data or evaluation—will become the ‘ground truth’ of the model and replicate the bias in the outputs of the system. In order to secure discriminatory non-harm, you must do your best to make sure your data sample has optimal source integrity. This involves securing or confirming that the data gathering processes involved suitable, reliable, and impartial sources of measurement and sound methods of collection.- Timeliness and Recency: If your datasets include outdated data then changes in the underlying data distribution may adversely affect the generalisability of your trained model. Provided these distributional drifts reflect changing social relationship or group dynamics, this loss of accuracy with regard to the actual characteristics of the underlying population may introduce bias into your AI system. In preventing discriminatory outcomes, you should scrutinise the timeliness and recency of all elements of the data that constitute your datasets.- Relevance, Appropriateness and Domain Knowledge: The understanding and utilisation of the most appropriate sources and types of data are crucial for building a robust and unbiased AI system. Solid domain knowledge of the underlying population distribution and of the predictive or classificatory goal of the project is instrumental for choosing optimally relevant measurement inputs that contribute to the reasonable determination of the defined solution. You should make sure that domain experts collaborate closely with your technical team to assist in the determination of the optimally appropriate categories and sources of measurement."(Leslie, 2019, p. 15)