Andy Zeng

Andy is a co-founder at Generalist. Before that, he was a research scientist and tech lead at Google DeepMind – and before that, a student at Princeton and UC Berkeley. Andy has worked on training large-scale robotics Foundation models, developing robots that write their own code, and inventing handheld data collection for robots.


X (Twitter)     G. Scholar     LinkedIn     Github

andyzeng at generalistai dot com

Video Language Planning

Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Kaelbling, Andy Zeng, Jonathan
Tompson
International Conference on Learning Representations (ICLR) 2024
Webpage  •   PDF  •   Code

PaLM-E: An Embodied Multimodal Language Model

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, Pete Florence
International Conference on Machine Learning (ICML) 2023
Webpage  •   PDF  •   Demo  •   Google AI Blog

Code as Policies: Language Model Programs for Embodied Control

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng
IEEE International Conference on Robotics and Automation (ICRA) 2023
★ Outstanding Learning Paper Award, ICRA ★
Webpage  •   PDF  •   Code  •   Colab  •   Google AI Blog  •   TechCrunch  •   AXIOS

Grasping in the Wild: Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations

Shuran Song, Andy Zeng, Johnny Lee, Thomas Funkhouser
IEEE International Conference on Intelligent Robots and Systems (IROS) 2020
IEEE Robotics and Automation Letters (RA-L) 2020
Webpage  •   PDF

TossingBot: Learning to Throw Arbitrary Objects with Residual Physics

Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, Thomas Funkhouser
Robotics: Science and Systems (RSS) 2019
IEEE Transactions on Robotics (T-RO) 2020
★ Best Paper Award, T-RO & Best Systems Paper Award, RSS ★
Webpage  •   PDF  •   Google AI Blog  •   New York Times  •   IEEE Spectrum

Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching

Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R. Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, Nima Fazeli, Ferran Alet, Nikhil Chavan Dafle, Rachel Holladay, Isabella Morona, Prem Qu Nair, Druck Green, Ian Taylor, Weber Liu, Thomas Funkhouser, Alberto Rodriguez
IEEE International Conference on Robotics and Automation (ICRA) 2018
The International Journal of Robotics Research (IJRR) 2019
★ 1st Place (Stow Task), Amazon Robotics Challenge 2017 ★
Webpage  •   PDF  •   Code  •   MIT News  •   Engadget

Show more publications

2024

UIUC CSL Student Conference

2023

EMNLP Workshop on RoboNLP

Cornell Robotics Seminar

MILA Robot Learning Seminar

UPenn GRASP SFI Seminar

RSS Workshop on Symmetries in Robot Learning (video)

RSS Workshop on Generalizable Manipulation Policy Learning: Paradigms and Debates

CVPR Workshop on 3D Scene Understanding for Vision, Graphics, and Robotics

ICRA Workshop on Embracing Contact: Making Robots Physically Interact with our World

ICRA Round-Table Discussion with Science Robotics

ICRA Workshop on Life-Long Learning with Human Help

Panel on Northwestern Grand Challenges in Robotics (video)

Guest Lecture at Columbia COMS 4733: Computational Aspects of Robotics

Stanford Vision and Learning Lab

2022

CoRL Workshop on Language and Robot Learning (video)

NeurIPS Workshop on Robot Learning (video)

NeurIPS Workshop on Habitat Rearrangement Challenge

Guest Lecture at Columbia IEOR E6617: Machine Learning and High-Dimensional Data

Princeton Robotics Seminar

RSS Workshop on Implicit Representations (video)

CVPR Tutorial on Vision-Based Robot Learning

MIT CSL Seminar (video)

Guest Lecture at Columbia COMS 4733: Computational Aspects of Robotics

2021

CVPR Workshop on 3D Vision and Robotics

Guest Lecture at Stony Brook University CSE525: Introduction to Robotics

Guest Lecture at NYU CSCI-UA 74: Big Ideas in Artificial Intelligence

2020

RSS Workshop on Self-Supervised Robot Learning

RSS Workshop on Robotics Retrospectives (video)

2019

Amazon Research Robotics Symposium

2018

RE·WORK Deep Learning for Robotics Summit

Guest Lecture at NCTU Robotics Seminar: Robotic Manipulation

2024

IEEE Robotics and Automation (RAS) Early Career Award

Human-Robot Interaction (HRI) Best Paper Award (Technical Track)

IEEE International Conference on Intelligent Robots and Systems (IROS) RoboCup Best Paper Award

2023

Conference on Robot Learning (CoRL) Best Student Paper Award

Associate Editor, IEEE Robotics and Automation Letters (RA-L)

Robotics: Science and Systems (RSS) Best Demo Paper Award Finalist

Google AI blog post "Modular Visual Question Answering via Code Generation"

Demo, Robotics: Science and Systems (RSS) - Large Language Models on Robots (memories)

Co-Organizer, RSS Workshop on Articulate Robots

IEEE International Conference on Robotics and Automation (ICRA) Outstanding Learning Paper Award

Associate Editor, IEEE International Conference on Robotics and Automation (ICRA)

Demo, Google I/O - Code as Policies with PaLI and PaLM 2 (memories)

Google AI blog post "Visual Language Maps for Robot Navigation"

Co-Organizer, AAAI Tutorial on Everything You Need to Know about Transformers

Google Research PaLM-E debut - "An Embodied Multimodal Language Model" (20+ derived YouTube videos)

Google AI blog post "PaLM-E: An Embodied Multimodal Language Model"

2022

Conference on Robot Learning (CoRL) Special Innovation Award

Area Chair, Conference on Robot Learning (CoRL)

Co-Organizer, CoRL Workshop on Pre-training Robot Learning

Co-Organizer, CoRL Workshop on Inductive Bias in Robot Learning

Google AI blog post "Robots That Write Their Own Code"

Google AI@ Event Code as Policies debut (memories) - AXIOS, TechCrunch, ZDNet (60+ articles)

Session Chair, Robotics: Science and Systems (RSS)

Co-Organizer, 2nd RSS Workshop on Scaling Robot Learning

Sponsor, Google Research Scholar Program

Co-Organizer, CVPR Tutorial on Vision-Based Robot Learning

Session Chair, IEEE International Conference on Robotics and Automation (ICRA)

Co-Organizer, ICRA Workshop on Scaling Robot Learning

AXIOS article "Unleash All This Creativity: Google AI's Breathtaking Potential"

CNET video "Google’s Most Advanced Robot Brain Just Got a Body"

Google Research PaLM-SayCan debut - Wired, Washington Post, The Verge (260+ articles)

Google AI blog post "Robot See, Robot Do"

Princeton News article "In Picking Up Trash, Robots Pick Up New Approaches to Work"

2021

Conference on Robot Learning (CoRL) Best Paper Award Finalist

Google AI blog post "Decisiveness in Imitation Learning for Robots"

Session Chair, IEEE International Conference on Robotics and Automation (ICRA)

Co-Organizer, RSS Workshop on Advancing AI and Manipulation for Robotics

Area Chair, Conference on Robot Learning (CoRL)

Google AI blog post "Learning to Manipulate Deformable Objects"

Sponsor, Google Research Scholar Program

Japan Factory Automation (FA) Foundation Paper Award

Google AI blog post "Rearranging the Visual World"

2020

IEEE Transactions on Robotics (T-RO) Best Paper Award

Conference on Robot Learning (CoRL) Best Paper Presentation Award Finalist

Google AI blog post "Visual Transfer Learning for Robotic Manipulation"

IEEE International Conference on Robotics and Automation (ICRA) Best Paper in Automation Award Finalist

Google AI blog post "Learning to See Transparent Objects"

Mentor, Google CS Research Mentorship Program

Sponsor, Google Faculty Research Awards

2019

New York Times article "A New Lab Full of Fast Learners" (100+ articles)

Google AI blog post "Learning to Assemble and to Generalize"

Robotics: Science and Systems (RSS) Best System Paper Award

Google AI blog post "Unifying Physics and Deep Learning with TossingBot"

2018

Honored to be a recipient of the Princeton SEAS Award for Excellence

Demo, Google TGIF - TossingBot (memories)

IEEE International Conference on Intelligent Robots and Systems (IROS) Best Cognitive Paper Award Finalist

Amazon Robotics Best System Paper Award

Honored to be a recipient of the NVIDIA Fellowship

MIT News article "Robo-Picker Grasps and Packs"

2017

1st place winners (stow) at the worldwide Amazon Picking Challenge 2017 with Team MIT-Princeton

2016

3rd place winners at the worldwide Amazon Picking Challenge 2016 with Team MIT-Princeton

Co-Organizer, CVPR Workshop on 3D Deep Learning

2015

Honored to be a recipient of the Gordon Y.S. Wu Fellowship in Engineering and Wu Prize

2015+

Reviewer, T-RO, RSS, CoRL, IJRR, RA-L, ICRA, NeurIPS, CVPR, IROS, ECCV, ICCV

Reviewer, SIGGRAPH, PR Journal, Eurographics, TMM, TIP, CASE, TCSVT

Member, IEEE