Interested in learning more about my Chinese name? Click 赵家樑.

Hi there!

I'm currently a PhD candidate with the Perceptual Science Group at MIT CSAIL. I'm advised by Prof. Edward H. Adelson, and I had the opportunity to collaborate with Prof. Russ Tedrake and Prof. Kaiming He. Below is a short summay of what I've been up to in recent years:

2022 - 2025🦾 (expected):

Cambridge, MA

Working towards my PhD at MIT CSAIL and MechE. I research contact-aware robotic manipulation and robot learning, with a focus on large behavior models and multi-modal sensing, especially tactile sensing.

2024 Summer⛱️:

Cambridge, MA

Worked as a research intern with the Large Behavior Models team at Toyota Research Institute. Had a great deal of satisfaction working with top researchers in the field on multi-modal sensing and cutting edge foundational models to scale robotics up.

2021 - 2022🚕:

Mountain View, CA

Worked on planning & prediction for self-driving cars at Nuro Inc. Very much enjoyed having real cars running on real streets in the city of Mountain View with my own algorithms. It felt crazy (and maybe a bit uneasy:) when I also sat in that car. Also enjoyed the California sunshine.

2018 - 2020🦿:

Pittsburgh, PA

Worked towards my Master in Robotics degree at The Robotics Institute of Carnegie Mellon University. Had the great pleasure to work with Prof. Oliver Kroemer at the IAM lab. Focused mostly on robot learning for robot manipulation.

2017 - 2018🤖:

Berkeley, CA

Worked on building & optimizing exoskeletons with Prof. Ruzena Bajcsy at UC Berkeley HART Lab as an undergrad researcher.

Out of work & school, I enjoy software and hardware robotics DIY. I recently started filming some of my hardware DIYs. Check out my YouTube channel Crafted by JZ or my BiliBili channel 试作间 if interested. Check out the #hobby section for released projects. Recently I'm working on an ambitious software project called tpvortex.org. Stay tuned.

News

2024 Sept: Heterogeneous Pre-trained Transformers (HPT) has been accepted to NeurIPS 24 as a spotlight paper!
2024 Sept: Our paper Transferable Tactile Transformers (T3) has been accepted to CoRL 24!
2024 Aug: Wrapped up my internship with the Large Behavior Models team at Toyota Research Institute! More details on my projects at TRI coming soon.
2024 May: We published a new paper on a foundational framework for camera-based tactile sensing and contact-aware manipulation T3!
2024 Jan: PoCo was accepted to RSS 2024! See it on MIT News.
2023 Oct: My paper on GelSight Svelte was nominated for the best paper award at IROS 2023! Also see it on MIT News.
2023 May: My paper FingerSLAM was presented at ICRA in London.
2023 May: I passed my qualification exam, so I'm officially a PhD candidate now:)

Research

Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers

A pre-trained framework to learn robotic manipulation through heterogeneous embodiments.

Lirui Wang, Xinlei Chen, Jialiang Zhao, Kaiming He. Accepted to NeurIPS 2024 (Spotlight).

Website arXiv Code Pretrained Weights HF Code

Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks

A tactile representation learned from multi-sensors and multi-tasks, and a tactile dataset that contains over 3M tactile images collected from 13 sensors and 11 tasks.

J. Zhao, Y. Ma, L. Wang, E. Adelson. Accepted to 2024 Conference on Robot Learning (CoRL).

Website arXiv Code Dataset and Weights Colab

Learning to insert electronics with submillimeter-level tolerance from tactile sensing and extrinsic contacts

J. Zhao as the leading contributor. In preparation.

more details coming soon.

PoCo: Policy Composition From and For Heterogeneous Robot Learning

A new fomulation for diffusion policies that can naturally be extented to multi-task, multi-embodiment settings.

L. Wang, J. Zhao, Y. Du, E. Adelson, R. Tedrake. Accepted to 2024 Robotics: Science and Systems (RSS).

Website arXiv YouTube Twitter

GelLink: A Compact Multi-phalanx Finger with Vision-based Tactile Sensing and Proprioception

A novel articulated tactile finger design that only requires one camera for tactile sensing.

Y. Ma, J. Zhao, E. Adelson. Accepted to the 2024 IEEE International Conference on Robotics and Automation (ICRA 2024).

arXiv

GelSight Svelte: A Human Finger-shaped Single-camera Tactile Robot Finger with Large Sensing Coverage and Proprioceptive Sensing

Think outside of the “box”: an optical design that unlocked the door of putting high-res tactile sensing in non-rectangular form factors.

J. Zhao and E. Adelson. Accepted to the 2023 International Conference on Intelligent Robots and Systems (IROS 2023)

Website arXiv YouTube MIT News Best Overall Paper Award Finalist

GelSight Svelte Hand: A Three-finger, Two-DoF, Low-cost, Tactile-rich Robot Hand for Dexterous Manipulation

A novel robot hand configuration that is dexterous and tactile-rich.

J. Zhao and E. Adelson. Submitted to the IROS workshop on Visuo-Tactile Perception, Learning, Control for Manipulation and HRI (IROS RoboTac 2023)

Website arXiv

FingerSLAM: Closed-loop Unknown Object Localization and Reconstruction from Visuo-tactile Feedback

Localizing and reconstructing unknown in-hand objects from vision and tactile with factor grasp-based optimization.

J. Zhao, M. Bauza, and E. Adelson. Accepted to the 2023 IEEE International Conference on Robotics and Automation (ICRA 2023)

arXiv YouTube

Causal Reasoning in Simulation for Structure and Transfer Learning of Robot Manipulation Policies

Improve data efficiency in RL by reducing observation and action spaces through causal reasoning.

T. E. Lee, J. Zhao, A. S. Sawhney, S. Girdhar, O. Kroemer. Accepted to the 2021 IEEE International Conference on Robotics and Automation (ICRA 2021)

arXiv Website

Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulation

Improve data efficiency in RL by devising and composing hierarchical controllers.

M. Sharma*, J. Liang*, J. Zhao, A. LaGrassa, O. Kroemer. Conference on Robot Learning (CoRL). November 2020.

arXiv Website YouTube Plenary Presentation

Towards Robotic Assembly by Predicting Robust, Precise and Task-oriented Grasps

Improve precision during different types of robotic assembly tasks with probabilistic grasp planning.

J. Zhao, D. Troniak, O.Kroemer. Submitted and accepted to 2020 Conference on Robot Learning (CoRL 2020). Oral presentation.

arXiv YouTube

Towards Precise Robotic Grasping by Probabilistic Post-grasp Displacement Estimation

Improve grasping precision by estimating a probabilistic distribution for post-grasp object displacements.

J. Zhao, J. Liang, O.Kroemer. Submitted and accepted to 12th Conference on Field and Service Robotics (FSR 2019). Oral presentation.

arXiv

...more on my Google Scholar page.

Hobby Corner

Alkaid Mount: Equatorial mount built with HarmonicDrive for astrophotography

A small yet sturdy equatorial mount built for astronomy/astrophotography, using real HarmonicDrive gearboxes. Fully open-sourced. See my blog post for more details.

GitHub YouTube BiliBili

Neptune MIDI

A 3d printed MIDI controller that can be used for video and photo editing (Lightroom, Photoshop, Premiere, DaVinci Resolve, etc.) It contains mechanical switches, rotary encoders, sliders, a joystick, and a small oled screen. Solidworks source files, STL files, BOM are fully open source.

GitHub YouTube BiliBili

RS-Slider

A mostly 3d printed, motorized, 2-axis camera slider that is Really Solid (R.S.). Tested with a 15lbs cinema camera setup. Software and hardware W.I.P. Details to be added.

RPI AirPlay HiFi

AirPlay + DAC + AMP, using a Raspberry Pi Zero W and HiFiBerry. Source files & instructins can be found in

GitHub: alanzjl/RPI_AirPlay_HiFi

2" Filter drawer for astrophotography

A filter drawer that houses a single 2" filter which be used between a telescope and a Nikon or Canon camera

GitHub: alanzjl/FilterDrawer

Resume

[Link]