Chen-Yu Lee

Google Scholar / Email

I am currently a Senior Staff Research Scientist and Manager at Google Cloud AI Research, driving innovation in machine learning and its real-world applications across diverse tasks and modalities.

Previously, I was at Apple where I published the Technology Development Group's inaugural research paper at CVPR and launched several key features in ARKit (now Vision Pro). I was also in the AI Research Group at Magic Leap with Andrew Rabinovich. I completed my PhD in deep learning, advised by Professors Zhuowen Tu and Pamela Cosman at UC San Diego, and mentored by Simon Osindero during my summer research. I am a recipient of the Test-of-Time Award for Deep Supervision at AISTATS 2025 and the Google Spotlight Award for Distilling Step-by-Step! at ACL 2023.

We are assembling a world-class team to explore the intersection of large AI models and high-value enterprise AI challenges. Contact me to learn more.

Publications

In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents

Zhen Tan, Jun Yan, I-Hung Hsu, Rujun Han, Zifeng Wang, Long T. Le, Yiwen Song, Yanfei Chen, Hamid Palangi, George Lee, Anand Iyer, Tianlong Chen, Huan Liu, Chen-Yu Lee, Tomas Pfister
ACL 2025 / Paper

Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence

Shangbin Feng, Zifeng Wang, Yike Wang, Sayna Ebrahimi, Hamid Palangi, Lesly Miculicich, Achin Kulshrestha, Nathalie Rauschmayr, Yejin Choi, Yulia Tsvetkov, Chen-Yu Lee, Tomas Pfister
ICML 2025 / Paper

Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling

Wenda Xu, Rujun Han, Zifeng Wang, Long T. Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yu Lee, Tomas Pfister
ICLR 2025 / Paper / Code

Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting

Zilong Wang, Zifeng Wang, Long Le, Huaixiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, Chen-Yu Lee, Tomas Pfister
ICLR 2025 / Paper / Google AI Blog

Reverse Thinking Makes LLMs Stronger Reasoners

Justin Chih-Yao Chen, Zifeng Wang, Hamid Palangi, Rujun Han, Sayna Ebrahimi, Long Le, Vincent Perot, Swaroop Mishra, Mohit Bansal, Chen-Yu Lee, Tomas Pfister
NAACL 2025 / Paper / Code

Where is the answer? An empirical study of positional bias for parametric knowledge extraction in language model

Kuniaki Saito, Chen-Yu Lee, Kihyuk Sohn, Yoshitaka Ushiku
NAACL 2025 / Paper (Oral Presentation)

TableRAG: Million-Token Tabular Reasoning with Large Language Models

Si-An Chen, Lesly Miculicich, Julian Martin Eisenschlos, Zifeng Wang, Zilong Wang, Yanfei Chen, Yasuhisa Fujii, Hsuan-Tien Lin, Chen-Yu Lee, Tomas Pfister
NeurIPS 2024 / Paper / Code

Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrieval

Yanfei Chen, Jinsung Yoon, Devendra Singh Sachan, Qingze Wang, Vincent Cohen-Addad, Mohammadhossein Bateni, Chen-Yu Lee, Tomas Pfister
EMNLP 2024 / Paper (Findings) / Google AI Blog

CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation

I-Hung Hsu, Zifeng Wang, Long T. Le, Lesly Miculicich, Nanyun Peng, Chen-Yu Lee, Tomas Pfister
ACL 2024 / Paper (Findings)

Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization

Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long Le, Abhishek Kumar, James R. Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, Tomas Pfister
ACL 2024 / Paper (Findings)

LMDX: Language Model-based Document Information Extraction and Localization

Vincent Perot, Kai Kang, Florian Luisier, Guolong Su, Xiaoyu Sun, Ramya Sree Boppana, Zilong Wang, Zifeng Wang, Jiaqi Mu, Hao Zhang, Chen-Yu Lee, Nan Hua
ACL 2024 / Paper (Findings)

CodecLM: Aligning Language Models with Tailored Synthetic Data

Zifeng Wang, Chun-Liang Li, Vincent Perot, Long T. Le, Jin Miao, Zizhao Zhang, Chen-Yu Lee, Tomas Pfister
NAACL 2024 / Paper (Findings) / Google AI Blog

Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding

Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, Tomas Pfister
ICLR 2024 / Paper / Code / Google AI Blog

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister
ACL 2023 / Paper (Findings) / Code / Google AI Blog (Google ACL 2023 Spotlight)

QueryForm: A Simple Zero-shot Form Entity Query Framework

Zifeng Wang, Zizhao Zhang, Jacob Devlin, Chen-Yu Lee, Guolong Su, Hao Zhang, Jennifer Dy, Vincent Perot, Tomas Pfister
ACL 2023 / Paper (Findings)

FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction

Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn, Nikolai Glushnev, Renshen Wang, Joshua Ainslie, Shangbang Long, Siyang Qin, Yasuhisa Fujii, Nan Hua, Tomas Pfister
ACL 2023 / Paper

VRDU: A Benchmark for Visually-rich Document Understanding

Zilong Wang, Yichao Zhou, Wei Wei, Chen-Yu Lee, Sandeep Tata
KDD 2023 / Paper / Code / Dataset

Multimodal Prompting with Missing Modalities for Visual Recognition

Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, Chen-Yu Lee
CVPR 2023 / Paper / Code

Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval

Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, Tomas Pfister
CVPR 2023 / Paper / Code / Google AI Blog

Prefix Conditioning Unifies Language and Label Supervision

Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, Tomas Pfister
CVPR 2023 / Paper / Google AI Blog

Neural Spline Search for Quantile Probabilistic Modeling

Ruoxi Sun, Chun-Liang Li, Sercan O. Arik, Michael W. Dusenberry, Chen-Yu Lee, Tomas Pfister
AAAI 2023 / Paper

Unifying Distribution Alignment as a Loss for Imbalanced Semi-Supervised Learning

Justin Lazarow, Kihyuk Sohn, Chen-Yu Lee, Chun-Liang Li, Zizhao Zhang, Tomas Pfister
WACV 2023 / Paper

Anomaly Clustering: Grouping Images into Coherent Clusters of Anomaly Types

Kihyuk Sohn, Jinsung Yoon, Chun-Liang Li, Chen-Yu Lee, Tomas Pfister
WACV 2023 / Paper

Self-supervise, Refine, Repeat: Improving Unsupervised Anomaly Detection

Jinsung Yoon, Kihyuk Sohn, Chun-Liang Li, Sercan O Arik, Chen-Yu Lee, Tomas Pfister
TMLR 2022 / Paper

DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning

Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, Tomas Pfister
ECCV 2022 / Paper / Code

FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction

Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, Tomas Pfister
ACL 2022 / Paper / Google AI Blog

Learning to Prompt for Continual Learning

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, Tomas Pfister
CVPR 2022 / Paper / Code / Google AI Blog

Learning from Weakly-labeled Web Videos via Exploring Sub-Concepts

Kunpeng Li, Zizhao Zhang, Guanhang Wu, Xuehan Xiong, Chen-Yu Lee, Zhichao Lu, Yun Fu, Tomas Pfister
AAAI 2022 / Paper / Google AI Blog

ROPE: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extraction

Chen-Yu Lee, Chun-Liang Li, Chu Wang, Renshen Wang, Yasuhisa Fujii, Siyang Qin, Ashok Popat, Tomas Pfister
ACL 2021 / Paper (Oral Presentation)

Learning to Branch for Multi-Task Learning

Pengsheng Guo, Chen-Yu Lee, Daniel Ulbricht
ICML 2020 / Paper

Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation

Chen-Yu Lee, Tanmay Batra, Mohammad Haris Baig, Daniel Ulbricht
CVPR 2019 / Paper / Code /  ML Journal

GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, Andrew Rabinovich
ICML 2018 / Paper

RoomNet: End-to-End Room Layout Estimation

Chen-Yu Lee, Vijay Badrinarayanan, Tomasz Malisiewicz, Andrew Rabinovich
ICCV 2017 / Paper

Generalizing Pooling Functions in CNNs: Mixed, Gated, and Tree

Chen-Yu Lee, Patrick Gallagher, Zhuowen Tu
TPAMI 2017 / Paper

Recursive Recurrent Nets with Attention Modeling for OCR in the Wild

Chen-Yu Lee and Simon Osindero
CVPR 2016 / Paper

Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree

Chen-Yu Lee, Patrick Gallagher, Zhuowen Tu
AISTATS 2016 / Paper / Code

Deeply-Supervised Nets

Chen-Yu Lee*, Saining Xie*, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu (*equal contribution)
AISTATS 2015 / Paper / Code (Test of Time Award @ AISTATS 2025, slides)
NIPS 2014 DL Workshop / Paper (Oral Presentation)

Region-based Discriminative Feature Pooling for Scene Text Recognition

Chen-Yu Lee, Anurag Bhardwaj, Wei Di, Vignesh Jagadeesh, Robinson Piramuthu
CVPR 2014 / Paper

Tracking Epithelial Cell Junctions in C. elegans Embryogenesis with Active Contours Guided by SIFT Flow

Sukryool Kang, Chen-Yu Lee, Monira Goncalves, Andrew Chisholm, Pamela Cosman
TBME 2014 / Paper

Automated Cell Junction Tracking with Modified Active Contours Guided by SIFT Flow

Chen-Yu Lee, Sukryool Kang, Andrew Chisholm, Pamela Cosman
ISBI 2014 / Paper / Demo

Smoke Detection Using Spatial and Temporal Analyses

Chen-Yu Lee, Chin-Teng Lin, Chao-Ting Hong, Miin-Tsair Su
IJICIC 2012 / Paper

Spatio‐temporal Analysis in Smoke Detection

Chen-Yu Lee, Chin‐Teng Lin, Chao‐Ting Hong
ICSIPA 2009 / Paper / Project Page

Pre-Prints & Tech Reports

When One LLM Drools, Multi-LLM Collaboration Rules

Shangbin Feng, Wenxuan Ding, Alisa Liu, Zifeng Wang, Weijia Shi, Yike Wang, Zejiang Shen, Xiaochuang Han, Hunter Lang, Chen-Yu Lee, Tomas Pfister, Yejin Choi, Yulia Tsvetkov
arXiv 2025 / Paper

Universal Model Routing for Efficient LLM Inference

Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeevesh Juneja, Zifeng Wang, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, Sanjiv Kumar
arXiv 2025 / Paper

Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems

Shangbin Feng, Zifeng Wang, Palash Goyal, Yike Wang, Weijia Shi, Huang Xia, Hamid Palangi, Luke Zettlemoyer, Yulia Tsvetkov, Chen-Yu Lee, Tomas Pfister
arXiv 2025 / Paper

Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models

Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, Tomas Pfister
arXiv 2023 / Paper

A Simple Semi-Supervised Learning Framework for Object Detection

Kihyuk Sohn*, Zizhao Zhang*, Chun-Liang Li, Han Zhang, Chen-Yu Lee, Tomas Pfister (*equal contribution)
arXiv 2020 / Paper / Code

Training Deeper Convolutional Networks with Deep Supervision

Liwei Wang, Chen-Yu Lee, Zhuowen Tu, Svetlana Lazebnik
arXiv 2015 / Paper / Code / Leaderboard of MIT Place205

Awards

Test-of-Time Award at AISTATS 2025
Oral at NAACL 2025
Google Spotlight Award at ACL 2023
Oral at ACL 2021
Oral at NIPS 2014 DL workshop

Academic Services

Area Chair: NeurIPS 2025, ICLR 2025, ICLR 2024
Reviewer: NeurIPS, ICML, ICLR, ACL, NAACL, EMNLP, COLM, CVPR, ICCV, ECCV, BMVC, PAMI, IJCV, JMLR
Outstanding Reviewer: CVPR 2019, BMVC 2019
Organizer / PC: SIGKDD 2025 AI4SupplyChain, CVPR 2020 CV4AR/VR