Auditing Algorithms Lab | DSC 290 | UC San Diego | Fall 2021



This seminar is for students interested in empirically investigating the outputs of real-world algorithmic systems of all kinds, particularly those where the code and/or training data are not publicly available. The first few weeks of the class will include more readings and lectures, when we cover the history of auditing and legal/ethical issues it raises. This includes studying classic audits of non-algorithmic decision systems (e.g. equal opportunity hiring investigations) to contemporary issues around the Computer Fraud and Abuse Act and the IRB. We will learn various approaches to investigate such systems, including auditing via training datasets, code, user reports, API scraping, sockpuppet accounts, and headless browsers. We will read and discuss various algorithmic audits by researchers and regulators, which will be a mix of selected readings and readings students choose. The second half of the class will be more discussion- and activity-based, as we perform audits on several real-world models whose developers have encouraged public auditing (e.g. Wikipedia’s content moderation classifiers). Students will work towards a final project, where they will conduct their own audits and develop strategies for how systems can be designed for auditability.


There are no official prerequsites to register and students from all departments are welcome to enroll. The class will generally assume knowledge of:

Please get in touch if you have any doubts or concerns about the prerequsites.

Potential readings

Note that the final reading list and schedule has not yet been finalized. Please reach out to Stuart Geiger if you have any suggestions or ideas. And thanks to for many of these!

What is an audit?

Classic auditing in non-algorithmic systems

Algorithmic auditing frameworks and introductions

The UMN Linux kernel security audit

(A big TBD here!)

Cases of algorithmic audits

Online advertising and pricing

Facial, biometric, speech recognition

Recommender systems and search engine rankings

Social media and user-generated content (mostly NLP / sentiment analysis)

Hiring, admissions, and other social sorting

The ProPublica vs. COMPAS/Northpointe debate on COMPAS criminal risk scores