maribeth rauh

Research Engineer

Sociotechnical translation and evaluation

Few things fascinate me more than where digital technology meets the complexity of our analog world. I am motivated by how technology is being used here and now, for better and for worse.

Since 2021, I have focused on evaluations of generative AI models. Evaluation is a place where what we value is explicitly codified. In machine learning, this usually means translating vague notions of "good" into quantifiable numbers. Delving into the challenges that come with doing so endlessly intrigues me.

I am also passionate about bridging the gap between machine learning practitioners and social scientists, journalists, and others grappling with the societal impact of AI. This sociotechnical translation, in both directions, is critical for ensuring AI systems do not perpetuate or exacerbate existing inequalities. This will be the default without dedicated interdisciplinary work, and the central motivation of my career is to use translation and evaluation to resist that outcome.

Selected Publications

I have led research on how to improve safety evaluations of generative AI based on my experience evaluating DeepMind's LLMs:

Maribeth Rauh, Nahema Marchal, Arianna Manzini, Lisa Anne Hendricks, Ramona Comanescu, Canfer Akbulut, Tom Stepleton, Juan Mateos-Garcia, Stevie Bergman, Jackie Kay, Conor Griffin, Ben Bariach, Iason Gabriel, Verena Rieser, William Isaac, Laura Weidinger. 2024. Gaps in the Safety Evaluation of Generative AI. AIES 2024.

Maribeth Rauh, John Mellor, Jonathan Uesato, Po-Sen Huang, Johannes Welbl, Laura Weidinger, Sumanth Dathathri, Amelia Glaese, Geoffrey Irving, Iason Gabriel, William Isaac, Lisa Anne Hendricks. 2022. Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models. NeurIPS 2022 - Datasets and Benchmarks Track.

I conducted safety evaluations of LLMs at DeepMind, with a focus on fairness and toxicity:

Gemini: A Family of Highly Capable Multimodal Models, 2023

Improving alignment of dialogue agents via targeted human judgements, 2022

Scaling Language Models: Methods, Analysis & Insights from Training Gopher, 2021

I conducted sociotechnical research on the risks of generative AI and how to address that, as early as 2021:

Iason Gabriel, Arianna Manzini, Geoff Keeling, Lisa Anne Hendricks, Verena Rieser, Hasan Iqbal, Nenad Tomašev, Ira Ktena, Zachary Kenton, Mikel Rodriguez, Seliem El-Sayed, Sasha Brown, Canfer Akbulut, Andrew Trask, Edward Hughes, A. Stevie Bergman, Renee Shelby, Nahema Marchal, Conor Griffin, Juan Mateos-Garcia, Laura Weidinger, Winnie Street, Benjamin Lange, Alex Ingerman, Alison Lentz, Reed Enger, Andrew Barakat, Victoria Krakovna, John Oliver Siy, Zeb Kurth-Nelson, Amanda McCroskery, Vijay Bolina, Harry Law, Murray Shanahan, Lize Alberts, Borja Balle, Sarah de Haas, Yetunde Ibitoye, Allan Dafoe, Beth Goldberg, Sébastien Krier, Alexander Reese, Sims Witherspoon, Will Hawkins, Maribeth Rauh, Don Wallace, Matija Franklin, Josh A. Goldstein, Joel Lehman, Michael Klenk, Shannon Vallor, Courtney Biles, Meredith Ringel Morris, Helen King, Blaise Agüera y Arcas, William Isaac, James Manyika. 2024. The Ethics of Advanced AI Assistants. Arxiv.

Laura Weidinger, Maribeth Rauh, Nahema Marchal, Arianna Manzini, Lisa Anne Hendricks, Juan Mateos-Garcia, Stevie Bergman, Jackie Kay, Conor Griffin, Ben Bariach, Iason Gabriel, Verena Rieser, William Isaac. 2023. Sociotechnical Safety Evaluation of Generative AI Systems. Arxiv.

A. Stevie Bergman, Lisa Anne Hendricks, Maribeth Rauh, Boxi Wu, William Agnew, Markus Kunesch, Isabella Duan, Iason Gabriel, and William Isaac. 2023. Representation in AI Evaluations. ACM FAccT 2023.

Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2022. Taxonomy of Risks posed by Language Models. ACM FAccT 2022.

For a complete list of publications, take a peep at Google Scholar.

Talks

I love translating AI concepts and safety research to a variety of audiences and have given talks in corporate, academic, and public facing settings, such as,

Deep dives into ML fairness and GenAI evaluation at Deliveroo

Co-hosting the AIUK opening provocation

AI ethics workshop for the Royal Central School of Speech and Drama's Research Ethics and Integrity Committee

Seminar for The London Arts & Humanities Partnership post graduate students

Reach out to me about

Giving a talk or running a workshop

Grabbing a coffee

The best bothies in Scotland

via LinkedIn or email (last name @ first name . info)