Publications

Research on AI data, open models, evaluation, and public accountability.

A featured selection with emphasis on venue, awards, public infrastructure, media coverage, and reusable research artifacts.

Selected from 42 papers, 12k+ citations, and five best or outstanding paper awards.

Full Paper List

Featured

2026ICLR 2026

ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality

Shayne Longpre, Sneha Kudugunta, Niklas Muennighoff, I-Hung Hsu, Sandy Pentland, Chen-Yu Lee, Sercan Arik, Sayna Ebrahimi

Practical scaling laws for multilingual transfer across pretraining, finetuning, and decoding.

Featured in Google Research

Paper Google Research

Scaling-law plots from the ATLAS paper showing optimal model size trajectories for six languages. — Optimal multilingual scaling trajectories from ATLAS.

2025NeurIPS 2025

FlexOlmo: Open Language Models for Flexible Data Use

Weijia Shi, Akshita Bhagia, Kevin Farhat, Niklas Muennighoff, Pete Walsh, Jacob Morrison, Dustin Schwenk, Shayne Longpre, et al.

SpotlightFeatured in Wired

Paper Wired

Overview diagram from the FlexOlmo paper showing data-owner expert modules merged into a flexible language model. — FlexOlmo architecture overview from the paper.

2025NeurIPS 2025

Establishing Best Practices for Building Rigorous Agentic Benchmarks

Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, et al.

A practical framework for designing rigorous benchmarks for agentic systems.

1st Place at Berkeley AgentX Summit

Paper AgentX Summit

2025NeurIPS 2025

The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text

Nikhil Kandpal, Brian Lester, Colin Raffel, Sebastian Majstorovic, Stella Biderman, Baber Abbasi, Luca Soldaini, Shayne Longpre, et al.

An 8TB public-domain and openly licensed text corpus for rights-conscious model training.

Paper Washington Post

2025NeurIPS 2025

The Leaderboard Illusion

Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D'Souza, Sayash Kapoor, Ahmet Ustun, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, et al.

Featured in TechCrunch

Paper TechCrunch

Bubble chart from The Leaderboard Illusion comparing public models and maximum arena score by provider. — Provider score and model-count bubble chart from The Leaderboard Illusion.

2025ICML 2025

In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI

Shayne Longpre, Kevin Klyman, Ruth E. Appel, Sayash Kapoor, Rishi Bommasani, Michelle Sahar, Sean McGregor, et al.

SpotlightFeatured in Wired

Paper Project Wired

Diagram from the third-party AI evaluation paper showing coordinated flaw disclosure routes across the AI supply chain. — Coordinated flaw disclosure system diagram from the third-party evaluation paper.

2025NAACL 2025

The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models

Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, et al.

A fine-grained language generation benchmark built around principled model-based judging.

Best Paper Award

Paper

2025TMLR 2025

The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources

Shayne Longpre, Stella Biderman, Alon Albalak, Hailey Schoelkopf, Daniel McDuff, Sayash Kapoor, Kevin Klyman, et al.

A survey and practical map of tools for responsible foundation model development.

Outstanding Survey

Paper Cheatsheet

2025ICLR 2025

Bridging the Data Provenance Gap Across Text, Speech and Video

Shayne Longpre, Data Provenance Initiative contributors, Caiming Xiong, Luis Villa, Stella Biderman, Alex Pentland, Sara Hooker, Jad Kabbara

Featured in MIT Technology Review

Paper Data Provenance MIT Tech Review

Line charts from the Data Provenance multimodal audit showing source growth across text, speech, and video datasets. — Source growth across modalities, from the multimodal provenance paper.

2024NeurIPS 2024

Consent in Crisis: The Rapid Decline of the AI Data Commons

Shayne Longpre, Data Provenance Initiative contributors, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, Sandy Pentland

Featured in The New York Times

Paper Project NYT

Stacked temporal charts from Consent in Crisis showing changes in robots.txt, terms-of-service, and crawler restrictions. — Temporal web-consent signals from Consent in Crisis.

2024ICML 2024

A Safe Harbor for AI Evaluation and Red Teaming

Shayne Longpre, Sayash Kapoor, Kevin Klyman, Ashwin Ramaswami, Rishi Bommasani, Borhane Blili-Hamelin, et al.

Oral, 1.5%Featured in The Washington Post

Paper Open Letter Washington Post

Commitment table from the Safe Harbor paper outlining legal safe harbor, technical safe harbor, and researcher commitments. — Mutual safe-harbor commitments from the ICML paper.

2024Nature Machine Intelligence 2024

The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI

Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, et al.

10k+ downloadsFeatured in The Washington Post

Paper Explorer Washington Post

License distribution bar chart from the Data Provenance Initiative paper. — License distribution from the Data Provenance Initiative audit.

2024TMLR 2024

The Foundation Model Transparency Index

Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, Percy Liang

Featured Paper AwardFeatured in The New York Times

Paper Project NYT

Stacked bar chart from the Foundation Model Transparency Index showing scores by upstream, model, and downstream domains. — Domain scores from the Foundation Model Transparency Index.

2024NAACL 2024

A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity

Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, Daphne Ippolito

Outstanding Paper Award

Paper Talk

Pipeline figure from A Pretrainer's Guide to Training Data showing data selection, model pretraining, and downstream evaluation. — Experimental pretraining curation pipeline from the paper.

2023ICML 2023

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, Adam Roberts

1.5k+ GitHub stars100k+ downloads

Paper Google Blog Code

Timeline table from the Flan Collection paper showing public instruction tuning collections and training details. — Public instruction-tuning timeline from the Flan Collection paper.

For the complete bibliography and citation record.

Full Paper List