project-dissemination reflection-questions
The scientific requirements of reproducibility and replicability demand from researchers to describe their experiment in such a way that another person could achieve at least similar results. For social science and humanities research using AI tools, this includes for instance making available the training data, the model and test prediction results if deemed safe for the data subjects (Zimmer, 2010).- Can the researcher make datasets available without violating the privacy of data subjects or revealing other sensitive information? - To what extent would rigorous anonymization of research data affect the utility of the data to allow for reproducibility and replicability? - What exact version of the model did the researcher use, was this model pre-trained and if so, what are the precise specifications of that particular dataset(s), what is the cultural and sociodemographic profile of the dataset?- Has the journal/conference in question established procedures for uploading material to safe space solution or other process for safe access for review purposes? Are there established repositories that offer sharing solutions appropriate for the specific material that might be used? - Does the journal/conference offer guidance for how to safely and adequately document AI based research? (The NeurIPS conference for example requires completion of a “reproducibility checklist,” https://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf) (franzke, 2020, p.44-45)