Chen-Yu Lee

Chen-Yu Lee










I am currently a Staff Research Scientist and Manager at Google Cloud AI Research, working on machine learning and its real-world applications to computer vision and natural language processing.

Previously I was at Apple where I published the first research paper from the Technology Development Group at CVPR and launched several key features in ARKit. I was also in the AI Research Group at Magic Leap with Andrew Rabinovich. I did my PhD in deep learning, advised by Professors Zhuowen Tu and Pamela Cosman at UC San Diego, and advised by Simon Osindero during the summer.

We are building a world-class team to explore the intersection of vision and language research. Contact me for more details.

Publications

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister
ACL 2023 / Paper (Findings)

QueryForm: A Simple Zero-shot Form Entity Query Framework

Zifeng Wang, Zizhao Zhang, Jacob Devlin, Chen-Yu Lee, Guolong Su, Hao Zhang, Jennifer Dy, Vincent Perot, Tomas Pfister
ACL 2023 / Paper (Findings)

FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction

Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn, Nikolai Glushnev, Renshen Wang, Joshua Ainslie, Shangbang Long, Siyang Qin, Yasuhisa Fujii, Nan Hua, Tomas Pfister
ACL 2023 / Paper

Multimodal Prompting with Missing Modalities for Visual Recognition

Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, Chen-Yu Lee
CVPR 2023 / Paper / Code

Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval

Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, Tomas Pfister
CVPR 2023 / Paper / Code

Prefix Conditioning Unifies Language and Label Supervision

Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, Tomas Pfister
CVPR 2023 / Paper

Neural Spline Search for Quantile Probabilistic Modeling

Ruoxi Sun, Chun-Liang Li, Sercan O. Arik, Michael W. Dusenberry, Chen-Yu Lee, Tomas Pfister
AAAI 2023 / Paper

Unifying Distribution Alignment as a Loss for Imbalanced Semi-Supervised Learning

Justin Lazarow, Kihyuk Sohn, Chen-Yu Lee, Chun-Liang Li, Zizhao Zhang, Tomas Pfister
WACV 2023 / Paper

Anomaly Clustering: Grouping Images into Coherent Clusters of Anomaly Types

Kihyuk Sohn, Jinsung Yoon, Chun-Liang Li, Chen-Yu Lee, Tomas Pfister
WACV 2023 / Paper

Self-supervise, Refine, Repeat: Improving Unsupervised Anomaly Detection

Jinsung Yoon, Kihyuk Sohn, Chun-Liang Li, Sercan O Arik, Chen-Yu Lee, Tomas Pfister
TMLR 2022 / Paper

DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning

Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, Tomas Pfister
ECCV 2022 / Paper / Code

FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction

Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, Tomas Pfister
ACL 2022 / Paper / Google AI Blog

Learning to Prompt for Continual Learning

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, Tomas Pfister
CVPR 2022 / Paper / Code / Google AI Blog

Learning from Weakly-labeled Web Videos via Exploring Sub-Concepts

Kunpeng Li, Zizhao Zhang, Guanhang Wu, Xuehan Xiong, Chen-Yu Lee, Zhichao Lu, Yun Fu, Tomas Pfister
AAAI 2022 / Paper / Google AI Blog

ROPE: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extraction

Chen-Yu Lee, Chun-Liang Li, Chu Wang, Renshen Wang, Yasuhisa Fujii, Siyang Qin, Ashok Popat, Tomas Pfister
ACL 2021 / Paper (Oral Presentation)

Learning to Branch for Multi-Task Learning

Pengsheng Guo, Chen-Yu Lee, Daniel Ulbricht
ICML 2020 / Paper

Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation

Chen-Yu Lee, Tanmay Batra, Mohammad Haris Baig, Daniel Ulbricht
CVPR 2019 / Paper / Code /  ML Journal

GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, Andrew Rabinovich
ICML 2018 / Paper

RoomNet: End-to-End Room Layout Estimation

Chen-Yu Lee, Vijay Badrinarayanan, Tomasz Malisiewicz, Andrew Rabinovich
ICCV 2017 / Paper

Generalizing Pooling Functions in CNNs: Mixed, Gated, and Tree

Chen-Yu Lee, Patrick Gallagher, Zhuowen Tu
TPAMI 2017 / Paper

Recursive Recurrent Nets with Attention Modeling for OCR in the Wild

Chen-Yu Lee and Simon Osindero
CVPR 2016 / Paper

Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree

Chen-Yu Lee, Patrick Gallagher, Zhuowen Tu
AISTATS 2016 / Paper / Code

Deeply-Supervised Nets

Chen-Yu Lee*, Saining Xie*, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu (*equal contribution)
AISTATS 2015 / Paper / Code
NIPS 2014 Workshop / Paper (Oral Presentation)

Region-based Discriminative Feature Pooling for Scene Text Recognition

Chen-Yu Lee, Anurag Bhardwaj, Wei Di, Vignesh Jagadeesh, Robinson Piramuthu
CVPR 2014 / Paper

Tracking Epithelial Cell Junctions in C. elegans Embryogenesis with Active Contours Guided by SIFT Flow

Sukryool Kang, Chen-Yu Lee, Monira Goncalves, Andrew Chisholm, Pamela Cosman
TBME 2014 / Paper

Automated Cell Junction Tracking with Modified Active Contours Guided by SIFT Flow

Chen-Yu Lee, Sukryool Kang, Andrew Chisholm, Pamela Cosman
ISBI 2014 / Paper / Demo

Smoke Detection Using Spatial and Temporal Analyses

Chen-Yu Lee, Chin-Teng Lin, Chao-Ting Hong, Miin-Tsair Su
IJICIC 2012 / Paper

Spatio‐temporal Analysis in Smoke Detection

Chen-Yu Lee, Chin‐Teng Lin, Chao‐Ting Hong
ICSIPA 2009 / Paper / Project Page

Tech Reports

A Simple Semi-Supervised Learning Framework for Object Detection

Kihyuk Sohn*, Zizhao Zhang*, Chun-Liang Li, Han Zhang, Chen-Yu Lee, Tomas Pfister (*equal contribution)
arXiv 2020 / Paper / Code

Training Deeper Convolutional Networks with Deep Supervision

Liwei Wang, Chen-Yu Lee, Zhuowen Tu, Svetlana Lazebnik
arXiv 2015 / Paper / Code / Leaderboard of MIT Place205

Patents

Meta-Learning for Multi-Task Learning for Neural Networks

Andrew Rabinovich, Vijay Badrinarayanan, Srivignesh Rajendran, Chen-Yu Lee
US Patent 11048978 (2021/6/29)

Room Layout Estimation Methods and Techniques

Chen-Yu Lee, Vijay Badrinarayanan, Tomasz Malisiewicz, Andrew Rabinovich
US Patent 10657376 (2020/5/19)

System and Method for Scene Text Recognition

Anurag Bhardwaj, Chen-Yu Lee, Robinson Piramuthu, Vignesh Jagadeesh, Wei Di
US Patent 9245191 (2016/1/26)

Academic Services

Reviewer

CVPR, ICCV, ECCV, NeurIPS, ICML, ICLR, BMVC, PAMI, IJCV, JMLR, TIP, TNNLS

Outstanding Reviewer

CVPR 2019, BMVC 2019

Program Committee Member

Computer Vision for AR/VR Workshop at CVPR 2020