project-analysis reflection-questions
"The research analytical process includes selecting the training data, cleaning the data, developing the model through steps of training, evaluating, adjusting, re-training the model. Source of Training Data **The inferences and predictions of an AI system are closely connected to the source of the training data and here especially issues on systemic discrimination or biases are interesting to disclose and reflect upon as many previous studies have shown such effects (Barocas & Selbst, 2016; Bechmann & Bowker, 2019; boyd & Crawford, 2012; Crawford & Calo, 2016; Kroll et al., 2017; Sweeney, 2013). The use of AI systems to uncover or predict social phenomena can thus be tainted by biases in the training data set on certain demographics or proxies thereof, which may lead to unfair and unjust outcomes.- What is the cultural and sociodemographic profile of the datasets used by the researcher to train the models? - To what extent does the cultural and sociodemographic profile of the training data allow for generalizability of the resulting findings or predictors from the research study? - Are there particular groups which may be advantaged or disadvantaged, in the context in which the researcher is deploying an AI-system? What is the potential damaging effect of uncertainty and error-rates to different groups? - How has the demographic profile of the researcher(s) affected the composition of the training data? - How does the training data as ‘ground truth’ affect different demographic profiles and proxies thereof?Data Cleaning Data cleaning is the process of detecting, correcting, replacing and even removing inaccurate and incomplete records from a database and structuring the data in a consistent way that makes it processable in the model. Researchers typically find data cleaning a difficult, timeconsuming, though necessary and important part of creating an AI-system. It is therefore tempting for some to cut corners or otherwise speed up the process, which can lead to concerns about the rigor and validity of the study because it is seldom accounted for in details. The time spent on cleaning a dataset and the assumptions that go into this process should be communicated more clearly in the resulting research paper. A descriptive analysis of the study datasets may help to identify missing information, incorrect coding, outliers, and misaligned data by the reader. - How would you characterize the datasets and their cleaning processes? For which variables was the cleaning process optimized? (Features, labels etc.) - How have (small) adjustments to the training data to make data fit into a model logic potentially influenced the outcome of the model calculations and predictions? - If the researcher used the raw data to train the model, to what extent could the resulting model be inaccurate, inappropriate, or dysfunctional? - Specifically, which actions have been taken by the research team in the process of cleaning the dataset and what potential consequences do these choices have on the predictions and/or findings made in the study? - How do the data cleaning actions normalize data and what are the potential consequences of taking out outliers in terms of minority representation in the model? - To what extent does the data cleaning process reflect the character of the data collected and the context in which it was provided? - What actions have been taken to anonymize/pseudonymize the data and to what extent is it possible to de-identify data subjects? Does the anonymization prevent certain types of analysis and what is the argument for the decisions taken? - How has the data been stored in order to safeguard the privacy of the data subjects? - If the research team consists of multiple parties and/or distributed calculations how has access to data been negotiated and established in a safe space solution for data subjects? *Model *The researcher’s model, based on cleaned training data, will likely have utility in predicting behaviours, or finding correlations in datasets. Such inferences may not be tailored to individuals or be based on anonymized data. Ethical issues may still remain, however, with regard to
1. the privacy considerations of groups on their collective behaviour and the resulting shifts of power balances,
1. the automation of inferences or decision-making, and
1. biases as well as errors in the output data. These issues may also arise if researchers choose to work with a pretrained model on different datasets, for instance open source models.Group Privacy and Power Dynamics - Can the knowledge that is generated and inferred from the model shift power balances with regard to specific communities and societies in the training data or as data subjects in terms of predictive power over their behaviour?- Could the increased power be operationalized maliciously if the model or inferred data was shared with a third-party, and how could such problems be mitigated? - Could the predictors identified by the model be operationalized maliciously by a third party when published and how could such use potentially be mitigated? - To what extent is the organization or the AI-system making decisions for data subjects?Automation - To what extent is human deliberation being replaced by automated systems and what consequences does it have for the research results? - Can the researcher override the automated inferences procedure, how will this be documented and justified for later reproducibility?- Are the automated inferences explainable? - Is there a strong incentive for the researcher to take the automated inferences as a base truth? How was the ground-truth identified and is this ground-truth adequate to predict the whole spectrum of the problem and/or population behaviour? - Can the data subjects influence the reach of the AI-system on their lives, their physical environments, and their own decisions? Should the researchers provide such functionality? *Biases and Errors *- To what extent has the researcher accounted for false positives and false negatives in the output of the model, and to what extent can the researcher mitigate their negative impacts? - Can the researchers use their model to infer social biases and communicate them? - How have steps of re-training the model to improve accuracy influenced the outcome and what considerations on representation/non-representation have been made in this practice? - If the research team uses a pretrained model, are the datasets well-documented and how can the character of the datasets influence the predictions of the research in question and the study of another context/practice?Model Training **- How many instances of re-training have taken place, what was the reason for each retraining and the result, what were the choices made for changing the settings, and what was the specific type of data added to the training loop? - How do the re-training choices align with the cultural and sociodemographic profile of the research group, and how does this affect the robustness/generalizability of the predictions and/or the findings of the study? - What would be the consequences of manually tweaking certain weights in the model and feeding the model with different training data? How would this affect the predictions of the model? "(franzke, et al., 2020 p.41-44)