└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # Toward ethical, transparent and fair AI/ML: a critical reading list for engineers, designers, and policy makers 2 | 3 | 4 | In the past 5 years there’s been a lot of enthusiasm about AI and specifically machine learning and deep learning. As we continuously deploy AI models in the wild we are forced to re-examine what are the effects of knowledge symbolisation, generalisation and classification on the historical, political and social conditions of human life. We also need to remind ourselves that algorithms don’t exercise their power over us. People do. 5 | 6 | This reading list is made for engineers, scientists, designers, policy makers and those interested in machine learning and AI. It’s an open ended document that examines machine learning as a sociotechnical system and contextualises its critical discourse. For suggestions and comments please tweet @irinimalliaraki or drop me an email at e.malliaraki16@imperial.ac.uk 7 | 8 | These sections aren't in any particular order. There's overlap and interaction between these topics that you can jump around as much as you want; Reading "out of order" could lead to interesting connections. 9 | 10 | 11 | ## CRITICAL AI 12 | 13 | ![alt text](https://i.imgur.com/TTpIGvo.jpg) 14 | 15 | ##### Must 16 | * [Manufacturing an Artificial Intelligence Revolution](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3078224), Yarden Katz, 2017 *(paper)* 17 | * [The critical engineering manifesto](https://criticalengineering.org/), Julian Oliver, Gordan Savičić, Danja Vasiliev 18 | * [Resisting Reduction: Designing our Complex Future with Machines](https://pubpub.ito.com/pub/resisting-reduction), Joi Ito , 2017 *(article)* 19 | * [The Seven Deadly Sins of Predicting the Future of AI](http://rodneybrooks.com/the-seven-deadly-sins-of-predicting-the-future-of-ai/), Rodney Brooks, 2017 *(article)* 20 | * [Remarks on the Hole of Representation in Computer ‘Vision’](https://www.youtube.com/watch?v=VSAaGcPsim0&t=1361s), Benjamin Bratton, 2017 *(video)* 21 | ##### Optional 22 | * [Critical Studies in Machine Intelligence](http://kim.hfg-karlsruhe.de/), Karlsruhe University of Arts and Design 23 | * [Linguistic Capitalism and Algorithmic Mediation](http://infoscience.epfl.ch/record/200539/files/Kaplan_Representations.pdf), F Kaplan, 2014 *(article)* 24 | * [Defining Algorithmic Ideology: Using Ideology Critique to Scrutinize Corporate Search Engines](https://www.triple-c.at/index.php/tripleC/article/view/439/641), Astrid Mager *(paper)* 25 | * [Toward a Critical Technical Practice: Lessons Learned in Trying to Reform AI](http://polaris.gseis.ucla.edu/pagre/critical.html), Philip E. Agre, 1997 *(paper)* 26 | * [Don’t Call AI “Magic”](https://points.datasociety.net/dont-call-ai-magic-142da16db408), M.C.Elish, 2017 *(article)* 27 | * [Deep Learning: A Critical Appraisal](https://arxiv.org/pdf/1801.00631.pdf), Gary Marcus, 2017 *(paper)* 28 | * A list of lists and literature review on critical algorithm studies by Tarleton Gillespie and Nick Seaver: https://socialmediacollective.org/reading-lists/critical-algorithm-studies/#0.5 29 | * [AI and Alchemy](https://www.rand.org/content/dam/rand/pubs/papers/2006/P3244.pdf), Hubert Dreyfus, 1965 30 | 31 | 32 | 33 | ## AI ACCOUNTABILITY & GOVERNANCE 34 | ![alt text](https://cdn-images-1.medium.com/max/1600/1*QVbmegtTgAqYNphZik_pmw.jpeg) 35 | 36 | ##### Must 37 | * [Algorithmic Accountability: On the Investigation of Black Boxes](https://towcenter.org/research/algorithmic-accountability-on-the-investigation-of-black-boxes-2/), 2014, Nicholas Diakopoulos *(article)* 38 | * [Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability](http://journals.sagepub.com/doi/abs/10.1177/1461444816676645?journalCode=nmsa), Mike Ananny, Kate Crawford, 2016 *(paper)* 39 | * [The Scored Society: Due Process for Automated Predictions](https://papers.ssrn.com/abstract=2376209), 2014, Danielle Keats Citron & Frank Pasquale *(paper)* 40 | * [Computational Power and the Social Impact of Artificial Intelligence](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3147971), Tim Hwang, 2018 *(paper)* 41 | ##### Optional 42 | * [Mapping AI Governance](https://www.nesta.org.uk/data-visualisation-and-interactive/mapping-ai-governance/), NESTA, 2019 43 | * [Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (We Robot 2016)](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2757236), M. C. Elish, 2016 *(paper)* 44 | * [Transparent, Explainable, and Accountable AI for Robotics](http://discovery.ucl.ac.uk/10038294/1/Wachter_Transparent_explainable_accountable_AI.pdf), Sandra Wachter,Brent Mittelstadt, Luciano Floridi *(paper)* 45 | * [The Threat of Algocracy: Reality, Resistance and Accommodation](https://vimeo.com/144630904), John Danaher, 2016 *(paper)* 46 | * List of articles on [Fairness, Accountability, and Transparency in Machine Learning](https://www.fatml.org/resources/relevant-scholarship) 47 | 48 | 49 | 50 | 51 | ## AI TRANSPARENCY, EXPLAINABILITY & BIAS 52 | ![alt text](https://cdn-images-1.medium.com/max/1600/1*DnTZCa4fLh9iayRkGtZGLg.png) 53 | ### Bias 54 | 55 | * [The Trouble with Bias](https://www.youtube.com/watch?v=fMym_BKWQzk&t=698s) — NIPS 2017 Keynote — Kate Crawford *(video)* 56 | * [Technology Is Biased Too. How Do We Fix It?](https://fivethirtyeight.com/features/technology-is-biased-too-how-do-we-fix-it/), Laura Hudson, 2017 *(article)* 57 | * [Machine Bias: There’s software used across the country to predict future criminals. And it’s biased against blacks, julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing), 2016 *(article)* 58 | * [Artificial Intelligence’s White Guy Problem](https://www.nytimes.com/2016/06/26/opinion/sunday/artificial-intelligences-white-guy-problem.html?_r=0), Kate Crawford, 2016 *(article)* 59 | * [Why Stanford Researchers Tried to Create a ‘Gaydar’ Machine](https://www.nytimes.com/2017/10/09/science/stanford-sexual-orientation-study.html), Heather Murphy, 2017 *(article)* 60 | * [The Dark Secret at the Heart of AI](https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/?utm_source=MIT+Technology+Review&utm_campaign=ee53290ab6-weekly_roundup_2017-04-13_edit&utm_medium=email&utm_term=0_997ed6f472-ee53290ab6-154320925&goal=0_997ed6f472-ee53290ab6-154320925&mc_cid=ee53290ab6&mc_eid=2840907a15), Will Knight, 2017 *(article)* 61 | 62 | ### Transparency and Explainability 63 | 64 | #### Must 65 | * [Big Data’s Disparate Impact](http://www.californialawreview.org/wp-content/uploads/2016/06/2Barocas-Selbst.pdf), Solon Barocas & Andrew D. Selbst, 2016 *(paper)* 66 | * [Interpretable Machine Learning: A Guide for Making Black Box Models Explainable](https://christophm.github.io/interpretable-ml-book/), Christoph Molnar, 2018 *(book)* 67 | * [The Mythos of Model Interpretability](https://arxiv.org/pdf/1606.03490.pdf), Zachary C. Lipton, 2017 *(paper)* 68 | * [Towards A Rigorous Science of Interpretable Machine Learning](https://arxiv.org/pdf/1702.08608.pdf), Finale Doshi-Velez and Been Kim, 2017 *(paper)* 69 | 70 | #### Optional 71 | * [Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability](http://mike.ananny.org/papers/anannyCrawford_seeingWithoutKnowing_2016.pdf), Mike Ananny, Kate Crawford, 2016 *(paper)* 72 | * [Algorithmic Transparency for the Smart City](https://poseidon01.ssrn.com/delivery.php?ID=419112000078118093003011078127079067109042093053023044073119015124066086073010093023038097054120044049061091116025096081018116118073055047067113114102114091069028048042086071127081088091122007103075064094072106071108120085065024104020098084067105069&EXT=pdf), Robert Brauneis and Ellen P. Goodman *(paper)* 73 | * Weapons Of Math Destruction: How Big Data Increases Inequality And Threatens Democracy, Crown, 2016 74 | * [Digital Decisions](https://cdt.org/issue/privacy-data/digital-decisions/), CDT, 2017 75 | * [The Intuitive Appeal of Explainable Machines](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3126971), Andrew D. Selbst and Solon Barocas, 2017 (paper) 76 | 77 | 78 | 79 | 80 | ## AI FAIRNESS 81 | ![alt text](https://cdn-images-1.medium.com/max/1600/1*WRXeJzU72brhkiXkCu7F0w.gif) 82 | ##### Must 83 | * [Computer says no: why making AIs fair, accountable and transparent is crucial](https://www.theguardian.com/science/2017/nov/05/computer-says-no-why-making-ais-fair-accountable-and-transparent-is-crucial), Ian Sample, 2017 *(article)* 84 | * [On the (im)possibility of fairness](https://arxiv.org/pdf/1609.07236.pdf), Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian *(paper)* 85 | * [Algorithmic decision making and the cost of fairness](https://arxiv.org/pdf/1701.08230.pdf), Sam Corbe-Davies, Emma Pierson, Avi Feller, 2017 *(paper)* 86 | * [Algorithmic Justice League](https://www.ajlunited.org/) 87 | * [Understanding unintended sources of unfairness in data driven decision making](https://medium.com/@mrtz/how-big-data-is-unfair-9aa544d739de), Moritz Hardt, 2014 *(article)* 88 | * Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor, Virginia Eubanks, 2018 89 | * List of lists http://algorithmtips.org/resources/ 90 | 91 | ## AI OWNERSHIP & CONTROL 92 | ![alt text](https://cdn-images-1.medium.com/max/1600/1*l3T-ShUDS7bDrgBhbcTjHg.gif) 93 | 94 | * [Big data: Bringing competition policy to the digital era](http://www.oecd.org/competition/big-data-bringing-competition-policy-to-the-digital-era.htm), OECD *(paper)* 95 | * [Big Other: Surveillance Capitalism and the Prospects of an Information Civilization](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2594754), 2015, Shoshana Zuboff *(paper)* 96 | * [Why Big Tech Companies Are Open-Sourcing Their AI Systems](http://www.iflscience.com/technology/why-big-tech-companies-are-open-sourcing-their-ai-systems), Patrick Shafto *(article)* 97 | * [Artificial Intelligence Pushes the Antitrust Envelope](https://www.bna.com/artificial-intelligence-pushes-n57982087335/), Michaella Ross, 2017 *(article)* 98 | 99 | 100 | ## AI ETHICS 101 | ![alt text](https://cdn-images-1.medium.com/max/1600/1*B5R_5p8GNSQxBI_L3Tt1Kg.gif) 102 | 103 | ##### Must 104 | * [Why Artificial Intelligence Is Still Waiting For Its Ethics Transplant](https://www.wired.com/story/why-ai-is-still-waiting-for-its-ethics-transplant/), Scott Rosenberg, 2017 (article) 105 | * [Ethics in AI](https://www.youtube.com/watch?v=FcI_IYHjZYg&list=WL&index=10), debate by Imperial College London *(video)* 106 | * [The ethics of algorithms: Mapping the debate](http://journals.sagepub.com/doi/pdf/10.1177/2053951716679679), Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter and Luciano Floridi, 2016 *(paper)* 107 | * [ASILOMAR AI principles](https://futureoflife.org/ai-principles/?submitted=1#confirmation) *(article)* 108 | * [The Ethics of Artificial Intelligence](https://intelligence.org/files/EthicsofAI.pdf), Nick Bostrom and Eliezer Yudkowsky, Machine Intelligence Research Institute *(paper)* 109 | * [Toward an Ethics of Algorithms: Convening, Observation, Probability, and Timeliness](http://mike.ananny.org/papers/ananny_towardAnEthicsOfAlgorithms_2015.pdf), Mike Ananny, 2015 *(paper)* 110 | ##### Optional 111 | * [EthicsNet](https://www.ethicsnet.com/), a dataset for machine ethics 112 | * [Moral Machine by MIT](http://moralmachine.mit.edu/), a crowd-sourced picture of human opinion on how machines should make decisions when faced with moral dilemmas 113 | * [Crowdsourcing Empathy](https://deepempathy.mit.edu/) by MIT, 2017 114 | * [Ethically aligned design: A Vision for Prioritizing Human Wellbeing with Artificial Intelligence and Autonomous Systems](http://standards.ieee.org/develop/indconn/ec/ead_v1.pdf), IEEE, 2016 *(paper)* 115 | * [Towards a machine ethics](https://pacita.strast.cz/files/prezentace/session_xvii_bendel.pdf), Prof. Dr. Oliver Bendel *(paper)* 116 | * [Moral Decision Making Frameworks for Artificial Intelligence](https://users.cs.duke.edu/~conitzer/moralAAAI17.pdf), Vincent Conitzer, Walter Sinnott-Armstrong, Jana Schaich Borg, Yuan Deng, Max Kramer *(paper)* 117 | * [Toward Ensuring Ethical Behavior from Autonomous Systems: A Case-Supported Principle-Based Paradigm](https://aaai.org/ocs/index.php/WS/AAAIW15/paper/view/9976/10125), Michael Anderson, Susan Leigh Anderson, 2015 *(paper)* 118 | * [Artificial Rhetorical Agents and the Computing of Phronesis](http://computationalculture.net/artificial-rhetorical-agents-and-the-computing-of-phronesis/), Jennifer Maher, 2016 *(paper)* 119 | * [The Philosophy of Science and its relation to Machine Learning](https://pdfs.semanticscholar.org/3f3c/ef3028c3eaac6246f6855b4ff5916c0a96ae.pdf), Jon Williamson *(paper)* 120 | 121 | ## AI POLICY & LAW 122 | ![alt text](https://cdn-images-1.medium.com/max/1600/1*zsvvVD7a37fEArX3D936_w.gif) 123 | * [Is effective regulation of AI possible? Eight potential regulatory problems](http://philosophicaldisquisitions.blogspot.co.uk/2015/07/is-effective-regulation-of-ai-possible.html),John Danaher, 2015 *(articles)* 124 | * [THE NATIONAL ARTIFICIAL INTELLIGENCE RESEARCH AND DEVELOPMENT STRATEGIC PLAN](https://www.nitrd.gov/PUBS/national_ai_rd_strategic_plan.pdf), National Science and Technology Council, 2016 *(paper)* 125 | * [AI Investment Framework](https://medium.com/mmc-writes/the-mmc-ventures-ai-investment-framework-17-success-factors-for-the-age-of-ai-671722e7ecd2), David Kelnar, 2017 *(article)* 126 | * [Encoded laws, policies, and virtues: the offspring of artificial intelligence and public-policy collaboration](https://medium.com/artificial-intelligence-policy-laws-and-ethics/encoded-laws-policies-and-virtues-the-offspring-of-artificial-intelligence-and-public-policy-3dfb357faf9), Matt Chessen, 2017 *(article)* 127 | * [AI in the UK: ready, willing and able?](https://publications.parliament.uk/pa/ld201719/ldselect/ldai/100/100.pdf), HOUSE OF LORDS Select Committee on Artificial Intelligence, 2018, *(report)* 128 | 129 | ## AI & LABOUR 130 | ![alt text](https://cdn-images-1.medium.com/max/1600/1*eybzPhwyOhKXexz1HNTsFA.gif) 131 | ### Automation and inequality 132 | * [Prof. Michael Osborne — The Impact of Artificial Intelligence on Jobs](https://www.youtube.com/watch?v=-eQNObP6QrQ&list=WL&index=5), 2017 *(video)* 133 | * [The Relentless Pace of Automation](https://www.technologyreview.com/s/603465/the-relentless-pace-of-automation/), David Rodman, 2017 (article) 134 | * [Technological Unemployment and the Value of Work, John Danaher](http://philosophicaldisquisitions.blogspot.co.uk/2015/09/technological-unemployment-and-value-of.html), 2015 *(articles)* 135 | Optional 136 | * [The Economics of Artificial Intelligence: An Agenda](http://papers.nber.org/books/agra-1), Ajay K. Agrawal, Joshua Gans, and Avi Goldfarb, 2017 *(collection of papers)* 137 | * [Machine learning and the politics of data](https://vimeo.com/195218446), Amaro, 2017 *(video)* 138 | * [Theories of Technology and the Production of Value from Everyday Life](http://www.artandeducation.net/classroom/66046/theories-of-technology-and-the-production-of-value-from-everyday-life), 2015 *(videos)* 139 | 140 | ### Discrimination 141 | * [Understanding Fair Labor Practices in a Networked Age](https://datasociety.net/pubs/fow/FairLabor.pdf) by Tamara Kneese, Alex Rosenblat, and Danah Boyd, 2014 *(paper)* 142 | * [Networked Employment Discrimination, Alex Rosenblat](https://datasociety.net/pubs/fow/EmploymentDiscrimination.pdf), Tamara Kneese, and danah boyd, 2014 *(paper)* 143 | * [Economic Models of (Algorithmic) Discrimination](http://www.mlandthelaw.org/papers/goodman2.pdf), Bryce W. Goodman *(paper)* 144 | * [Data-Driven Discrimination at Work](http://scholarship.law.wm.edu/cgi/viewcontent.cgi?article=3680&context=wmlr), Pauline T. Kim, 2017 *(paper)* 145 | 146 | ## AI & SOCIAL IMPACT 147 | ##### Must 148 | * [AI Now 2017 — Experts Workshop](https://www.youtube.com/watch?v=5d4PBlG2P3w&t=1123s) *(video)* 149 | * [Artificial Intelligence for Social Good](https://cra.org/ccc/wp-content/uploads/sites/2/2016/04/AI-for-Social-Good-Workshop-Report.pdf), Gregory D. Hager, Ann Drobnis, Fei Fang, Rayid Ghani, Amy Greenwald, Terah Lyons, David C. Parkes, Jason Schultz, Suchi Saria, Stephen F. Smith, and Milind Tambe, 2017 *(paper)* 150 | * [AI for the Common Good: Sustainable, Inclusive and Trustworthy](https://weforum.ent.box.com/v/AI4Good?platform=hootsuite), Derek O’Halloran, 2017 *(paper)* 151 | * [Harnessing Artificial Intelligence for the Earth](https://preview.thenewsmarket.com/Previews/PWC/DocumentAssets/497923.pdf), PwC and the World Economic Forum, 2018 *(paper)* 152 | * [Principles for Accountable Algorithms and a Social Impact Statement for Algorithms](https://www.fatml.org/resources/principles-for-accountable-algorithms), FATML.org 153 | ##### Optional 154 | * [An AI pattern language](https://autonomy.datasociety.net/patternlanguage/), M.C. Elish and Tim Hwang, Data & Society *(paper)* 155 | * [Computational Power and the Social Impact of Artificial Intelligence](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3147971), Tim Hwang, 2018, *(paper)* 156 | * [AI Now, 2017 Report](https://assets.contentful.com/8wprhhvnpfc0/1A9c3ZTCZa2KEYM64Wsc2a/8636557c5fb14f2b74b2be64c3ce0c78/_AI_Now_Institute_2017_Report_.pdf), Alex Campolo, Madelyn Sanfilippo, Meredith Whittaker, Kate Crawford *(paper)* 157 | * Keeping track of what AI can do, and where it is being applied [https://deepindex.org/](https://deepindex.org/) 158 | * [AI Progress Measurement: Measuring the Progress of AI Research](https://www.eff.org/ai/metrics#Measuring-the-Progress-of-AI-Research), Electronic Frontier Foundation 159 | * [Exponential View by Azeem Azhar](https://us15.campaign-archive.com/home/?u=eee7b8043119f98544067854b&id=d581eab324), mailing list all things AI and society 160 | 161 | ## AI & PROPAGANDA 162 | ![alt text](https://cdn-images-1.medium.com/max/1600/1*9tjJH74yk7DMVlTAQgmv-Q.gif) 163 | * [Weapons of Mass Persuasion](https://motherboard.vice.com/en_us/article/kbzmvm/weapons-of-mass-persuasion), Tim Hwang, 2015 *(article)* 164 | * [The Observatory on Social Media](https://truthy.indiana.edu/) 165 | * [The Computational Propaganda Research Project](http://comprop.oii.ox.ac.uk/)(COMPROP) investigates the interaction of algorithms, automation and politics. 166 | * [Conversation AI](https://conversationai.github.io/) is a collaborative research effort exploring ML as a tool for better discussions online. 167 | * [Engineering the public: Big data, surveillance and computational politics](http://firstmonday.org/article/view/4901/4097), Tufekci, 2014 *(article)* 168 | 169 | ## AI & DESIGN 170 | * [Machine Learning for Designers](http://www.oreilly.com/design/free/files/machine-learning-for-designers.pdf), Patrick Hebron, 2016 171 | * [Experience Design in the Machine Learning Era](https://medium.com/@girardin/experience-design-in-the-machine-learning-era-e16c87f4f2e2), Fabien Girardin, 2016 *(article)* 172 | * [Design, Philosophy and A.I.](https://www.youtube.com/watch?v=4TIUVeOO5tk&t=177s), Benjamin Bratton. 2016 *(video)* 173 | * [The UX of AI](https://design.google/library/ux-ai/), Google 174 | * [Human-Centered Machine Learning](https://medium.com/google-design/human-centered-machine-learning-a770d10562cd), Jess Holbrook, 2017 *(article)* 175 | 176 | 177 | ## AI AUDITING & SECURITY 178 | ![alt text](https://cdn-images-1.medium.com/max/1600/1*c7Snu7beJLpE9S9XkGSBQg.jpeg) 179 | ##### Must 180 | * [Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms](http://www-personal.umich.edu/~csandvig/research/Auditing%20Algorithms%20--%20Sandvig%20--%20ICA%202014%20Data%20and%20Discrimination%20Preconference.pdf), Christian Sandvig , Kevin Hamilton , Karrie Karahalios , & Cedric Langbort, 2014 181 | * [Ten simple rules for responsible big data research](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005399), Matthew Zook ,Solon Barocas, danah boyd, Kate Crawford, Emily Keller, Seeta Peña Gangadharan, Alyssa Goodman, 2017 (article) 182 | * [The Malicious Use of Artificial Intelligence](https://img1.wsimg.com/blobby/go/3d82daa4-97fe-4096-9c6b-376b92c619de/downloads/1c6q2kc4v_50335.pdf), University of Oxford, Centre for the Study of Existential Risk, Future of Life Institute, OpenAI, Center for a New American Security, EFF, 2018 183 | ##### Optional 184 | * [https://pair-code.github.io/facets/](https://pair-code.github.io/facets/): Facets contains two robust visualizations to aid in understanding and analyzing machine learning datasets. 185 | * [Adversarial ML](https://media.ccc.de/v/34c3-8860-deep_learning_blindspots), Chaos Communication Congress 2017 (video) 186 | * [Machine Deception: Paper Roundup](https://medium.com/@timhwang/machine-deception-paper-roundup-7e9fee9771b9), Tim Hwang, 2018 (article) 187 | * [Practical Techniques for Interpreting Machine Learning Models: Introductory Open Source Examples Using Python, H2O, and XGBoost](https://fatconference.org/static/tutorials/hall_interpretable18.pdf), Patrick Hall, 2018 188 | * [Auditing Algorithms](http://auditingalgorithms.science/?page_id=89) 189 | * [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://arxiv.org/pdf/1712.03141.pdf), Battista Biggioa , Fabio Roli 190 | 191 | 192 | 193 | ## PEOPLE & ORGANISATIONS 194 | There are plenty of research groups and initiatives both in academia and in the industry start thinking about the relevance of ethics and safety in AI: 195 | * [Future of Life Institute](https://futureoflife.org/ai-news/)(Boston) 196 | * [AI4ALL](http://ai-4-all.org/) 197 | * [Centre for Data Ethics and Innovation Consultation](https://www.gov.uk/government/consultations/consultation-on-the-centre-for-data-ethics-and-innovation/centre-for-data-ethics-and-innovation-consultation) (UK) 198 | * [People for the Ethical Treatment of Reinforcement Learners](http://petrl.org/) 199 | * [Berkman Klein Center](https://cyber.harvard.edu/) (Boston) 200 | * [Alan Turing Institute Data Ethics Group](https://www.turing.ac.uk/news/alan-turing-institute-data-ethics-group/) 201 | * [Institute Electrical and Electronic Engineers — IEEE](http://standards.ieee.org/develop/indconn/ec/autonomous_systems.html) (Global) 202 | * [Centre for the study on existential risks](http://cser.org/) (Cambridge, UK) 203 | * [Association for the Advancement of Artificial Intelligence](http://www.aaai.org/) 204 | * [The Machine Intelligence Research Institute](https://intelligence.org/) 205 | * [Council on the Future of AI and Robotics](https://www.weforum.org/communities/the-future-of-artificial-intelligence-and-robotics) 206 | * [Data & Society’s Intelligence and Autonomy Initiative](http://autonomy.datasociety.net/) 207 | * [Facebook Artificial Intelligence Research](https://research.fb.com/category/facebook-ai-research-fair/) 208 | * [Ethics of Governance and Artificial Intelligence fund](https://cyber.harvard.edu/research/ai) 209 | * [Microsoft: AI research group](https://www.microsoft.com/en-us/research/research-area/artificial-intelligence/) 210 | * [K&L gates endowment for ethics](http://www.klgates.com/artificial-intelligence-practices/) (Global) 211 | * [Center for human-compatible AI](http://humancompatible.ai/) (Berkeley, CA) 212 | * [Machine Intelligence Research Institute](https://intelligence.org/) (Berkeley, CA) 213 | * [USC center for AI in society](https://www.cais.usc.edu/) (Los Angeles) 214 | * [Leverhulme center for future of intelligence](http://lcfi.ac.uk/) (Cambridge, UK) 215 | * [Partnership on AI](https://www.partnershiponai.org/) (Global) 216 | * [Future of Humanity Institute](https://www.fhi.ox.ac.uk/) (Oxford, UK) 217 | * [AI Austin](https://www.ai-austin.org/) (Austin, US) 218 | * [Open AI](https://openai.com/) (San Francisco, US) 219 | * [Campaign to Stop Killer Robots](https://www.stopkillerrobots.org/) (Global) 220 | * [AI, Ethics and Governance Project](https://www.media.mit.edu/projects/ai-ethics-and-governance/overview/) 221 | * [Campaign against Sex Robots](https://campaignagainstsexrobots.org/) (Global) 222 | * [PAIR | People+AI Research Initiative](https://ai.google/pair) 223 | * [Digital Ethics Lab- Oxford Internet Institute](https://www.oii.ox.ac.uk/research/digital-ethics-lab/) 224 | * [The MIT Laboratory for Social Machines](http://socialmachines.media.mit.edu/) 225 | * [Foundation for Responsible Robotics](http://responsiblerobotics.org/) (Global) 226 | * [Data & Society](https://datasociety.net/) (New York, US) 227 | * [World Economic Forum’s Council on the Future of AI and Robotics](https://www.weforum.org/communities/the-future-of-artificial-intelligence-and-robotics) (Global) 228 | * [AI Now Initiative](https://artificialintelligencenow.com/) (New York, US) 229 | * [Allen Institute for Artificial intelligence](http://allenai.org/) 230 | * [IEEE Global Initiative for Ethical Considerations in Artificial Intelligence and Autonomous Systems](https://standards.ieee.org/develop/indconn/ec/autonomous_systems.html) 231 | * [AI100](https://ai100.stanford.edu/) (Stanford, CA) 232 | * [Artificial Intelligence Committee](https://www.parliament.uk/ai-committee) — UK Parliament (AI Lords on Twitter) 233 | * [Open Roboethics Institute](http://www.openroboethics.org/) 234 | * [Ada Lovelace Institute](https://www.adalovelaceinstitute.org/) 235 | 236 | 237 | 238 | --------------------------------------------------------------------------------