Research by AIST members

Representational Simplicity and Circuit Size Dissociate in a Threshold-Dependent Way: A Controlled Test via Adversarial TrainingAdam ElimadiSeptember 27 Tracing mechanisms of sycophantic agreement in language modelsSixing Chen, Zhuofan Josh Ying, Logan Riggs Smith, Jeremy Wertheimer, Natalie ShapiraSeptember 21 A Transformative AI Strategy for EuropeMonika Schnitzer (convenor), Daniel Privitera (editorial lead) and the Senior Expert Council, with research leads Ben Bucknall, Noemi Dreksler, Philip Fox, Anton Leicht, Conor McGlynn, Milo Rignell, Frauke StehrSeptember 14 Why Does Robustness Reduce Superposition?Adam Elimadi · COLM 2026 workshopAugust 23 Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across LanguagesIsabella Gidi, Antonio Almudévar, Core Francisco Park, Naomi Saphra, Ricard Marxer · COLM 2026August 19 Civil-Military Convergence in AI: Emerging Risks and Institutional ResponsibilitiesJean Dong, Riley Simmons-Edler, Jonathan Kwik, Paul Lushenko, Kanaka Rajan, Ryan P. BadmanAugust 12 The AI Whistleblower Protection Act: Suggested AmendmentsMark Gitau, Raqda Sayidali, Karl Koch, Abra GanzJuly 30 Language Models Embody and Amplify Human Cognitive Distortions: What Is to Be Done?Arnau Marín-Llobet, Steven A. Lehr, Mahzarin R. BanajiJuly 22 Optimizing Against Safety Representations: Activation-Guided Adversarial Suffixes and the Geometry of RefusalEge Çakar, Hannah Guan, Kayden Kehe · AAAI Summer Symposium 2026July 9 Individual Parameters in Weight-Sparse Transformers Appear InterpretableArnau Marín-Llobet, Stefan HeimersheimJuly 3 Natural Ungrokking: Asymmetric Control of Which Rules Survive PretrainingJuliana Li, Diya Sreedhar · ICML 2026 workshopJune 24 AI alignment is a human problemKonstantinos Voudouris, Xavier Roberts-Gaal, Marie Buhl, Geoffrey Irving, Christopher SummerfieldMay 30 Vision-Language Models Suppress Female Representations Under Ambiguous InputArnau Marín-Llobet, Simon Henniger, Mahzarin R. Banaji · EMNLP 2026May 29 Beyond Vibe Decision Theory: Asymmetric Manipulation Vulnerabilities in LLM Multi-Agent CoordinationSukanya Krishna, Tobin South · AAMAS 2026May 25 When Are Two Networks the Same? Tensor Similarity for Mechanistic InterpretabilityML Nissen Gonzalez, Melwina Albuquerque, Laurence Wroe, Jacob Meyer Cohen, Logan Riggs Smith, Thomas DoomsMay 14 Introspection Fine-Tuning (IFT): Training Small LLMs to IntrospectEly Hahami, Ishaan Sinha, Lavik JainMay 8 Automated Interpretability and Feature Discovery in Language Models with AgentsArnau Marín-Llobet, Javier FerrandoMay 2 Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm TypesHadas Orgad, Boyi Wei, Kaden Zheng, Martin Wattenberg, Peter Henderson, Seraphina Goldfarb-Tarrant, Yonatan BelinkovApril 10 RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift StudiesPatricia Paskov, Kevin Wei, Shen Zhou Hong, Dan Bateyko, Xavier Roberts-Gaal, Carson Ezell, Gailius Praninskas, Valerie Chen, Umang Bhatt, Ella GuestMarch 11 Characterizing the ability of LLMs to recapitulate Americans' distributional responses to public opinion polling questions across political issuesEric Gong, Nathan E. Sanders, Bruce SchneierMarch 6 Measuring Mid-2025 LLM-Assistance on Novice Performance in BiologyShen Zhou Hong, Alex Kleinman, Alyssa Mathiowetz, Adam Howes, Julian Cohen, Suveer Ganta, Alex Letizia, Dora Liao, Deepika Pahari, Xavier Roberts-Gaal, Luca Righetti, Joe TorresFebruary 18 Assessing the Case for Africa-Centric AI Safety EvaluationsGathoni Ireri, Cecil Abungu, Jean Cheptumo, Sienka Dounia, Mark Gitau, Stephanie Kasaon, Michael Michie, Chinasa T. Okolo, Jonathan ShockFebruary 14 Scaling Reward Modeling without Human SupervisionJingxuan Fan, Yueying Li, Zhenting Qi, Dinghuai Zhang, Kianté Brantley, Sham M. Kakade, Hanlin Zhang · ICLR 2026 workshopFebruary 11 Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game PerspectiveHaichuan Wang, Tao Lin, Lingkai Kong, Ce Li, Hezi Jiang, Milind Tambe · ICML 2026January 31 Circuit Mechanisms for Spatial Relation Generation in Diffusion TransformersBinxu Wang, Jingxuan Fan, Xu Pan · CVPR 2026January 9

Detecting the Disturbance: A Nuanced View of Introspective Abilities in LLMsEly Hahami, Ishaan Sinha, Lavik Jain, Josh Kaplan, Jon HahamiDecember 13 Decomposition of Small Transformer ModelsCasper L. Christensen, Logan Riggs · NeurIPS 2025 workshopNovember 12 Incident Analysis for AI AgentsCarson Ezell, Xavier Roberts-Gaal, Alan ChanAugust 19 User-Assistant Bias in LLMsXu Pan, Jingxuan Fan, Zidi Xiong, Ely Hahami, Jorin Overwiening, Ziqian XieAugust 17 How Do LLMs Persuade? Linear Probes Can Uncover Persuasion Dynamics in Multi-Turn ConversationsBrandon Jaipersaud, David Krueger, Ekdeep Singh LubanaAugust 7 Steering Out-of-Distribution Generalization with Concept Ablation Fine-TuningHelena Casademunt, Caden Juang, Adam Karvonen, Samuel Marks, Senthooran Rajamanoharan, Neel NandaJuly 22 Combining Cost-Constrained Runtime Monitors for AI SafetyTim Tian Hua, James Baskerville, Henri Lemoine, Mia Hopman, Aryan Bhatt, Tyler TracyJuly 19 Emergence of Hierarchical Emotion Organization in Large Language ModelsBo Zhao, Maya Okawa, Eric J. Bigelow, Rose Yu, Tomer Ullman, Ekdeep Singh Lubana, Hidenori TanakaJuly 12 Leveraging the Sequential Nature of Language for InterpretabilityUsha Bhalla, Alex Oesterling, Claudio Mayrink Verdun, Flavio P. Calmon, Himabindu LakkarajuJuly 10 Can Interpretability Predict Behavior on Unseen Data?Victoria R. Li, Jenny Kaufmann, Martin Wattenberg, David Alvarez-Melis, Naomi SaphraJuly 8 Persona Features Control Emergent MisalignmentMiles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A. Chi, Samuel Miserendino, Johannes Heidecke, Tejal Patwardhan, Dan MossingJune 24 Detecting High-Stakes Interactions with Activation ProbesAlex McKenzie, Urja Pawar, Phil Blandfort, William Bankes, David Krueger, Ekdeep Singh Lubana, Dmitrii KrasheninnikovJune 12 Recommendations and Reporting Checklist for Rigorous & Transparent Human Baselines in Model EvaluationsKevin Wei, Patricia Paskov, Sunishchal Dev, Michael J. Byun, Anka Reuel, Xavier Roberts-Gaal, Rachel Calcott, Evie Coxon, Chinmay DeshpandeJune 9 HeavyWater and SimplexWater: Watermarking Low-Entropy Text DistributionsDor Tsur, Carol Xuan Long, Claudio Mayrink Verdun, Hsiang Hsu, Chen-Fu Chen, Haim Permuter, Sajani Vithana, Flavio P. CalmonJune 6 Evaluating Sparse Autoencoders: From Shallow Design to Matching PursuitValérie Costa, Thomas Fel, Ekdeep Singh Lubana, Bahareh Tolooshams, Demba BaJune 5 Bridging the Artificial Intelligence Governance Gap: The United States’ and China’s Divergent Approaches to Governing General-Purpose Artificial IntelligenceOliver Guest, Kevin WeiJune 4 From Flat to Hierarchical: Extracting Sparse Representations with Matching PursuitValérie Costa, Thomas Fel, Ekdeep Singh Lubana, Bahareh Tolooshams, Demba BaJune 3 Noise Injection Systemically Degrades Large Language Model Safety GuardrailsPrithviraj Singh Shahani, Matthias ScheutzMay 16 Optimized Couplings for Watermarking Large Language ModelsDor Tsur, Carol Xuan Long, Claudio Mayrink Verdun, Hsiang Hsu, Haim Permuter, Flavio P. CalmonMay 13 Promising Topics for U.S.-China Dialogues on AI Risks and GovernanceSaad Siddiqui, Lujain Ibrahim, Kristy Loke, Stephen Clare, Marianne Lu, Aris Richardson, Conor McGlynn, Jeffrey DingMay 12 Soft Best-of-n Sampling for Model AlignmentClaudio Mayrink Verdun, Alex Oesterling, Himabindu Lakkaraju, Flavio P. CalmonMay 6 Third-Party Compliance Reviews for Frontier AI Safety FrameworksAidan Homewood et al. (Kevin Wei)May 3 Memorization and Knowledge Injection in Gated LLMsXu Pan, Ely Hahami, Zechen Zhang, Haim SompolinskyApril 30 Inference-Time Reward Hacking in Large Language ModelsHadi Khalaf, Claudio Mayrink Verdun, Alex Oesterling, Himabindu Lakkaraju, Flavio du Pin CalmonApril 24 Ctrl-Z: Controlling AI Agents via ResamplingAryan Bhatt, Cody Rushing, Adam Kaufman, Tyler Tracy, Vasil Georgiev, David Matolcsi, Akbir Khan, Buck ShlegerisApril 14 AI 2027Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, Romeo DeanApril 3 Towards Interpretable Soft PromptsOam Patel, Jason Wang, Nikhil Shivakumar Nayak, Suraj Srinivas, Himabindu LakkarajuApril 2 EconEvals: Benchmarks and Litmus Tests for LLM Agents in Unknown EnvironmentsSara Fish, Julia Shephard, Minkai Li, Ran I. Shorrer, Yannai A. GonczarowskiMarch 24 Measuring AI Ability to Complete Long TasksThomas Kwa et al. (Nikola Jurkovic)March 18 Human Baselines in Model Evaluations Need Rigor and TransparencyKevin Wei, Patricia Paskov, Sunishchal Dev, Michael Byun, Anka Reuel, Xavier Roberts-Gaal, Rachel Calcott, Evie Coxon, Chinmay DeshpandeMarch 5 Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept GeometrySai Sumedh R. Hindupur, Ekdeep Singh Lubana, Thomas Fel, Demba BaMarch 3 Multi-Agent Risks from Advanced AILewis Hammond et al. (Carson Ezell)February 19 Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision ModelsThomas Fel, Ekdeep Singh Lubana, Jacob S. Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Wattenberg, Demba Ba, Talia KonkleFebruary 18 Rule-Bottleneck Reinforcement Learning: Joint Explanation and Decision Optimization for Resource Allocation with Language AgentsMauricio Tec, Guojun Xiong, Haichuan Wang, Francesca Dominici, Milind TambeFebruary 15 AI Alignment at Your DiscretionMaarten Buyl, Hadi Khalaf, Claudio Mayrink Verdun, Lucas Monteiro Paes, Caio C. Vieira Machado, Flavio du Pin CalmonFebruary 10 The AI Agent IndexStephen Casper, Luke Bailey, Rosco Hunter and 12 others, including Luke Bailey and Carson Ezell and Kevin Wei and Nikola JurkovicFebruary 3

A Llama Sunk My Battleship! Asking Rational Questions with LLMs via Bayesian InferenceGabriel Grand, Valerio Pepe, Joshua Tenenbaum and 12 others, including Valerio PepeDecember 31 ICLR: In-Context Learning of RepresentationsCore Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, Hidenori TanakaDecember 29 Obfuscated Activations Bypass LLM Latent-Space DefensesLuke Bailey, Alex Serrano, Abhay Sheshadri, Mikhail Seleznyov, Jordan Taylor, Erik Jenner, Jacob Hilton, Stephen Casper, Carlos Guestrin, Scott EmmonsDecember 12 Noise Injection Reveals Hidden Capabilities of Sandbagging Language ModelsCameron Tice, Philipp Alexander Kreer, Nathan Helm-Burger, Prithviraj Singh Shahani, Fedor Ryzhenkov, Jacob Haimes, Felix Hofstätter, Teun van der WeijDecember 2 Competition Dynamics Shape Algorithmic Phases of In-Context LearningCore Francisco Park, Ekdeep Singh Lubana, Itamar Pres, Hidenori TanakaDecember 1 Evaluating Large Language Models’ Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human SubjectsFred Heiding, Simon Lermen, Andrew Kao, Bruce Schneier, Arun VishwanathNovember 30 Loss-to-Loss Prediction: Scaling Laws for All DatasetsDavid Brandfonbrener, Nikhil Anand, Nikhil Vyas, Eran Malach, Sham KakadeNovember 19 Alignment as JurisprudenceNicholas A. CaputoNovember 18 Abrupt Learning in Transformers: A Case Study on Matrix CompletionPulkit Gopalani, Ekdeep Singh Lubana, Wei HuOctober 29 Mixture of Parrots: Experts Improve Memorization More Than ReasoningSamy Jelassi, Clara Mohri, David Brandfonbrener, Alex Gu, Nikhil Vyas, Nikhil Anand, David Alvarez-Melis, Yuanzhi Li, Sham M. Kakade, Eran MalachOctober 24 Representation Shattering in Transformers: A Synthetic Study with Knowledge EditingKento Nishi, Maya Okawa, Rahul Ramesh, Mikail Khona, Hidenori Tanaka, Ekdeep Singh LubanaOctober 22 Estimating the Probabilities of Rare Outputs in Language ModelsGabriel Wu, Jacob HiltonOctober 17 How do AI Companies “Fine-Tune” Policy? Examining Regulatory Capture in AI GovernanceKevin Wei, Carson Ezell, Nick Gabrieli, Chinmay DeshpandeOctober 16 Analyzing (In)Abilities of SAEs Using Formal LanguagesAbhinav Menon, Manish Shrivastava, David Krueger, Ekdeep Singh LubanaOctober 15 Do Unlearning Methods Remove Information from Language Model Weights?Aghyad Deeb, Fabien RogerOctober 11 Dynamics of Concept Learning and Compositional GeneralizationYongyi Yang, Core Francisco Park, Ekdeep Singh Lubana, Maya Okawa, Wei Hu, Hidenori TanakaOctober 10 Causation Does Not Imply Correlation: A Study of Circuit Mechanisms and Model BehaviorsJenny Kaufmann, Victoria R. Li, Martin Wattenberg, David Alvarez-Melis, Naomi SaphraOctober 10 Endless Jailbreaks with Bijection LearningBrian R.Y. Huang, Maximilian Li, Leonard TangOctober 2 Optimal Ablation For InterpretabilityMaximilian Li, Lucas JansonSeptember 16 A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal LanguageEkdeep Singh Lubana, Kyogo Kawaguchi, Robert P. Dick, Hidenori TanakaAugust 22 The Quest for the Right Mediator: A History, Survey, and Theoretical Grounding of Causal InterpretabilityAaron Mueller, Jannik Brinkmann, Millicent Li and 10 others, including Samuel MarksAugust 2 An FDA for AI? Pitfalls and plausibility of Approval Regulation for Frontier Artificial IntelligenceDaniel Carpenter, Carson EzellAugust 1 Tamper-Resistant Safeguards for Open-Weight LLMsRishub Tamirisa, Bhrugu Bharathi, Long Phan and 12 others, including Rowan WangAugust 1 Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game ModelsAdam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, Claudio Mayrink Verdun, David Bau, Samuel MarksJuly 31 Planning in a Recurrent Neural Network That Plays SokobanMohammad Taufeeque, Philip Quirke, Maximilian Li, Chris Cundy, Aaron David Tucker, Adam Gleave, Adrià Garriga-AlonsoJuly 22 When Do Universal Image Jailbreaks Transfer Between Vision-Language Models?Rylan Schaeffer, Dan Valentine, Luke Bailey and 12 others, including Luke BaileyJuly 21 All Roads Lead to Rome? Exploring Representational Similarities Between Latent Spaces of Generative Image ModelsCharumathi Badrinath, Usha Bhalla, Alex Oesterling, Suraj Srinivas, Himabindu LakkarajuJuly 18 NNsight and NDIF: Democratizing Access to Foundation Model InternalsJaden Fiotto-Kaufman, Alexander Loftus, Eric Todd and 17 others, including Alexander Loftus and Samuel MarksJuly 18 What Makes and Breaks Safety Fine-tuning?Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip H.S. Torr, Amartya Sanyal, Puneet K. DokaniaJuly 14 Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept SpaceCore Francisco Park, Maya Okawa, Andrew Lee, Ekdeep Singh Lubana, Hidenori TanakaJune 27 Connecting the Dots: LLMs Can Infer and Verbalize Latent Structure from Disparate Training DataJohannes Treutlein, Dami Choi, Jan Betley, Cem Anil, Samuel Marks, Roger Baker Grosse, Owain EvansJune 20 Transcendence: Generative Models Can Outperform The Experts That Train ThemEdwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade, Eran MalachJune 17 Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language ModelsCarson Denison, Monte MacDiarmid, Fazl Barez and 11 others, including Samuel MarksJune 14 DafnyBench: A Benchmark for Formal Software VerificationChloe Loughridge, Qinyi Sun, Seth Ahrenbach, Federico Cassano, Chuyue Sun, Ying Sheng, Anish Mudide, Md Rakib Hossain Misu, Nada Amin, Max TegmarkJune 12 Designing a Dashboard for Transparency and Control of Conversational AIYida Chen, Aoyu Wu, Trevor DePodesta, Catherine Yeh, Kenneth Li, Nicholas Castillo Marin, Oam Patel, Jan Riecke, Shivam Raval, Olivia Seow, Martin Wattenberg, Fernanda ViégasJune 12 Improving Alignment and Robustness with Circuit BreakersAndy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, Dan HendrycksJune 6 Harvard Undergraduate Survey on Generative AIShikoh Hirabayashi, Rishab Jain, Nikola Jurković, Gabriel WuJune 2 WebSuite: Systematically Evaluating Why Web Agents FailEric Li, Jim WaldoJune 1 Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language ModelsSamuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, Aaron MuellerMarch 28 Safety Cases: How to Justify the Safety of Advanced AI SystemsJoshua Clymer, Nick Gabrieli, David Krueger, Thomas LarsenMarch 15 Governing Through the Cloud: The Intermediary Role of Compute Providers in AI RegulationLennart Heim, Tim Fist, Janet Egan, Sihao Huang, Stephen Zekany, Robert Trager, Michael A Osborne, Noa ZilbermanMarch 13 Defending Against Unforeseen Failure Modes with Latent Adversarial TrainingStephen Casper, Lennart Schulze, Oam Patel, Dylan Hadfield-MenellMarch 8 The Evolution of Statistical Induction Heads: In-Context Learning Markov ChainsBenjamin L. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, Nikolaos TsilivisFebruary 16 Measuring and Controlling Persona Drift in Language Model DialogsKenneth Li, Tianle Liu, Naomi Bashkansky, David Bau, Fernanda Viégas, Hanspeter Pfister, Martin WattenbergFebruary 13 What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental CausesVictor Lecomte, Kushal Thaman, Rylan Schaeffer, Naomi Bashkansky, Trevor Chow, Sanmi KoyejoFebruary 13 Distinguishing the Knowable from the Unknowable with Language ModelsGustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, Benjamin L. EdelmanFebruary 5 Black-Box Access is Insufficient for Rigorous AI AuditsStephen Casper, Carson Ezell, Charlotte Siegmann and 18 others, including Stephen Casper and Carson Ezell and Kevin Wei and Satyapriya KrishnaJanuary 25

Forbidden Facts: An Investigation of Competing Objectives in Llama 2Tony Wang*, Miles Wang*, Kaivalya Hariharan*, Nir ShavitDecember 31 Steering Llama 2 via Contrastive Activation AdditionNina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, Alexander TurnerDecember 31 Post-Deployment Regulatory Oversight for General-Purpose Language ModelsCarson Ezell, Abraham LoebDecember 8 Mechanistically Analyzing the Effects of Fine-Tuning on Procedurally Defined TasksSamyak Jain*, Robert Kirk*, Ekdeep Singh Lubana*, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, David KruegerNovember 21 Tensor Trust: Interpretable Prompt Injection Attacks from an Online GameSam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, Stuart RussellNovember 2 Inference-Time Intervention: Eliciting Truthful Answers from a Language ModelKenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, Martin WattenbergOctober 20 Image Hijacks: Adversarial Images can Control Generative Models at RuntimeLuke Bailey, Euan Ong, Stuart Russell, Scott EmmonsSeptember 1