Robotics
Where hardware, software and machine intelligence come together.
Our research is interdisciplinary and focuses on sensing, planning, reasoning, and control of single and multi-agent systems, including both manipulation and mobile robots. We strive to develop algorithms and methods for factory automation, smart building and transportation applications using machine learning, computer vision, RF/optical sensing, wireless communications, control theory and signal processing. Key research themes include bin picking and object manipulation, sensing and mapping of indoor areas, coordinated control of robot swarms, as well as robot learning and simulation.
Quick Links
-
Researchers
Devesh K.
Jha
Diego
Romeres
Daniel N.
Nikovski
Arvind
Raghunathan
Stefano
Di Cairano
Siddarth
Jain
Yebin
Wang
William S.
Yerazunis
Karl
Berntorp
Radu
Corcodel
Yuki
Shirai
Mouhacine
Benosman
Toshiaki
Koike-Akino
Tim K.
Marks
Scott A.
Bortoff
Abraham P.
Vinod
Avishai
Weiss
Ye
Wang
Matthew
Brand
Anoop
Cherian
Chiori
Hori
Jonathan
Le Roux
Philip V.
Orlik
Bingnan
Wang
Abraham
Goldsmith
Jianlin
Guo
Hassan
Mansour
Koon Hoo
Teo
Anthony
Vetro
Pedro
Miraldo
James
Queeney
-
Awards
-
AWARD Honorable Mention Award at NeurIPS 23 Instruction Workshop Date: December 15, 2023
Awarded to: Lingfeng Sun, Devesh K. Jha, Chiori Hori, Siddharth Jain, Radu Corcodel, Xinghao Zhu, Masayoshi Tomizuka and Diego Romeres
MERL Contacts: Radu Corcodel; Chiori Hori; Siddarth Jain; Devesh K. Jha; Diego Romeres
Research Areas: Artificial Intelligence, Machine Learning, RoboticsBrief- MERL Researchers received an "Honorable Mention award" at the Workshop on Instruction Tuning and Instruction Following at the NeurIPS 2023 conference in New Orleans. The workshop was on the topic of instruction tuning and Instruction following for Large Language Models (LLMs). MERL researchers presented their work on interactive planning using LLMs for partially observable robotic tasks during the oral presentation session at the workshop.
-
AWARD Joint University of Padua-MERL team wins Challenge 'AI Olympics With RealAIGym' Date: August 25, 2023
Awarded to: Alberto Dalla Libera, Niccolo' Turcato, Giulio Giacomuzzo, Ruggero Carli, Diego Romeres
MERL Contact: Diego Romeres
Research Areas: Artificial Intelligence, Machine Learning, RoboticsBrief- A joint team consisting of members of University of Padua and MERL ranked 1st in the IJCAI2023 Challenge "Al Olympics With RealAlGym: Is Al Ready for Athletic Intelligence in the Real World?". The team was composed by MERL researcher Diego Romeres and a team from University Padua (UniPD) consisting of Alberto Dalla Libera, Ph.D., Ph.D. Candidates: Niccolò Turcato, Giulio Giacomuzzo and Prof. Ruggero Carli from University of Padua.
The International Joint Conference on Artificial Intelligence (IJCAI) is a premier gathering for AI researchers and organizes several competitions. This year the competition CC7 "AI Olympics With RealAIGym: Is AI Ready for Athletic Intelligence in the Real World?" consisted of two stages: simulation and real-robot experiments on two under-actuated robotic systems. The two robotics systems were treated as separate tracks and one final winner was selected for each track based on specific performance criteria in the control tasks.
The UniPD-MERL team competed and won in both tracks. The team's system made strong use of a Model-based Reinforcement Learning algorithm called (MC-PILCO) that we recently published in the journal IEEE Transaction on Robotics.
- A joint team consisting of members of University of Padua and MERL ranked 1st in the IJCAI2023 Challenge "Al Olympics With RealAlGym: Is Al Ready for Athletic Intelligence in the Real World?". The team was composed by MERL researcher Diego Romeres and a team from University Padua (UniPD) consisting of Alberto Dalla Libera, Ph.D., Ph.D. Candidates: Niccolò Turcato, Giulio Giacomuzzo and Prof. Ruggero Carli from University of Padua.
-
AWARD MERL Researchers Win Best Workshop Poster Award at the 2023 IEEE International Conference on Robotics and Automation (ICRA) Date: June 2, 2023
Awarded to: Yuki Shirai, Devesh Jha, Arvind Raghunathan and Dennis Hong
MERL Contacts: Devesh K. Jha; Arvind Raghunathan
Research Areas: Artificial Intelligence, Optimization, RoboticsBrief- MERL's paper titled: "Closed-Loop Tactile Controller for Tool Manipulation" Won the Best Poster Award in the workshop on "Embracing contacts : Making robots physically interact with our world". First author and MERL intern, Yuki Shirai, was presented with the award at a ceremony held at ICRA in London. MERL researchers Devesh Jha, Principal Research Scientist, and Arvind Raghunathan, Senior Principal Research Scientist and Senior Team Leader as well as Prof. Dennis Hong of University of California, Los Angeles are also coauthors.
The paper presents a technique to manipulate an object using a tool in a closed-loop fashion using vision-based tactile sensors. More information about the workshop and the various speakers can be found here https://sites.google.com/view/icra2023embracingcontacts/home.
- MERL's paper titled: "Closed-Loop Tactile Controller for Tool Manipulation" Won the Best Poster Award in the workshop on "Embracing contacts : Making robots physically interact with our world". First author and MERL intern, Yuki Shirai, was presented with the award at a ceremony held at ICRA in London. MERL researchers Devesh Jha, Principal Research Scientist, and Arvind Raghunathan, Senior Principal Research Scientist and Senior Team Leader as well as Prof. Dennis Hong of University of California, Los Angeles are also coauthors.
See All Awards for Robotics -
-
News & Events
-
NEWS Diego Romeres gave an invited talk at the Padua University's Seminar series on "AI in Action" Date: April 9, 2024
MERL Contact: Diego Romeres
Research Areas: Artificial Intelligence, Dynamical Systems, Machine Learning, Optimization, RoboticsBrief- Diego Romeres, Principal Research Scientist and Team Leader in the Optimization and Robotics Team, was invited to speak as a guest lecturer in the seminar series on "AI in Action" in the Department of Management and Engineering, at the University of Padua.
The talk, entitled "Machine Learning for Robotics and Automation" described MERL's recent research on machine learning and model-based reinforcement learning applied to robotics and automation.
- Diego Romeres, Principal Research Scientist and Team Leader in the Optimization and Robotics Team, was invited to speak as a guest lecturer in the seminar series on "AI in Action" in the Department of Management and Engineering, at the University of Padua.
-
EVENT MERL Contributes to ICASSP 2024 Date: Sunday, April 14, 2024 - Friday, April 19, 2024
Location: Seoul, South Korea
MERL Contacts: Petros T. Boufounos; François Germain; Chiori Hori; Sameer Khurana; Toshiaki Koike-Akino; Jonathan Le Roux; Hassan Mansour; Zexu Pan; Kieran Parsons; Joshua Rapp; Anthony Vetro; Pu (Perry) Wang; Gordon Wichern; Ryoma Yataka
Research Areas: Artificial Intelligence, Computational Sensing, Machine Learning, Robotics, Signal Processing, Speech & AudioBrief- MERL has made numerous contributions to both the organization and technical program of ICASSP 2024, which is being held in Seoul, Korea from April 14-19, 2024.
Sponsorship and Awards
MERL is proud to be a Bronze Patron of the conference and will participate in the student job fair on Thursday, April 18. Please join this session to learn more about employment opportunities at MERL, including openings for research scientists, post-docs, and interns.
MERL is pleased to be the sponsor of two IEEE Awards that will be presented at the conference. We congratulate Prof. Stéphane G. Mallat, the recipient of the 2024 IEEE Fourier Award for Signal Processing, and Prof. Keiichi Tokuda, the recipient of the 2024 IEEE James L. Flanagan Speech and Audio Processing Award.
Jonathan Le Roux, MERL Speech and Audio Senior Team Leader, will also be recognized during the Awards Ceremony for his recent elevation to IEEE Fellow.
Technical Program
MERL will present 13 papers in the main conference on a wide range of topics including automated audio captioning, speech separation, audio generative models, speech and sound synthesis, spatial audio reproduction, multimodal indoor monitoring, radar imaging, depth estimation, physics-informed machine learning, and integrated sensing and communications (ISAC). Three workshop papers have also been accepted for presentation on audio-visual speaker diarization, music source separation, and music generative models.
Perry Wang is the co-organizer of the Workshop on Signal Processing and Machine Learning Advances in Automotive Radars (SPLAR), held on Sunday, April 14. It features keynote talks from leaders in both academia and industry, peer-reviewed workshop papers, and lightning talks from ICASSP regular tracks on signal processing and machine learning for automotive radar and, more generally, radar perception.
Gordon Wichern will present an invited keynote talk on analyzing and interpreting audio deep learning models at the Workshop on Explainable Machine Learning for Speech and Audio (XAI-SA), held on Monday, April 15. He will also appear in a panel discussion on interpretable audio AI at the workshop.
Perry Wang also co-organizes a two-part special session on Next-Generation Wi-Fi Sensing (SS-L9 and SS-L13) which will be held on Thursday afternoon, April 18. The special session includes papers on PHY-layer oriented signal processing and data-driven deep learning advances, and supports upcoming 802.11bf WLAN Sensing Standardization activities.
Petros Boufounos is participating as a mentor in ICASSP’s Micro-Mentoring Experience Program (MiME).
About ICASSP
ICASSP is the flagship conference of the IEEE Signal Processing Society, and the world's largest and most comprehensive technical conference focused on the research advances and latest technological development in signal and information processing. The event attracts more than 3000 participants.
- MERL has made numerous contributions to both the organization and technical program of ICASSP 2024, which is being held in Seoul, Korea from April 14-19, 2024.
See All News & Events for Robotics -
-
Internships
-
SA2181: Autonomous mobile robot data collection
MERL is seeking a highly motivated intern to collaborate in the collection of data for sensing, planning, and control methods in a robotic test-bed using Turtlebots at MERL. The ideal candidate is enrolled in a Masters/PhD program in Electrical, Mechanical, Aerospace Engineering, Robotics, Computer Science, or related program, with prior experience in motion planning, control, optimization, computer vision, and their application in mobile robots, including experimental validation. The candidate should be proficient in ROS, C/C++, and Python. The expected duration of the internship is 1-2 months, with a flexible start date in early summer to fall.
-
CI2075: Human-Machine Interface with Biosignal Processing
MERL is seeking an intern to work on research for human-machine interface with multi-modal bio-sensors. The ideal candidate is an experienced PhD student or post-graduate researcher having an excellent background in brain-machine interface (BMI), deep learning, mixed reality (XR), remote robot manipulation, bionics, and bio sensing. The expected duration of the internship is 3-6 months, with a flexible start date.
-
CA2132: Optimization Algorithms for Motion Planning and Predictive Control
MERL is looking for a highly motivated and qualified individual to work on tailored computational algorithms for optimization-based motion planning and predictive control applications in autonomous systems (vehicles, mobile robots). The ideal candidate should have experience in either one or multiple of the following topics: convex and non-convex optimization, stochastic predictive control (e.g., scenario trees), interaction-aware motion planning, machine learning, learning-based model predictive control, mathematical programs with complementarity constraints (MPCCs), optimal control, and real-time optimization. PhD students in engineering or mathematics, especially with a focus on research related to any of the above topics are encouraged to apply. Publication of relevant results in conference proceedings or journals is expected. Capability of implementing the designs and algorithms in MATLAB/Python is required; coding parts of the algorithms in C/C++ is a plus. The expected duration of the internship is 3 months, and the start date is flexible.
See All Internships for Robotics -
-
Recent Publications
- "Chance-Constrained Optimization for Contact-rich Systems using Mixed Integer Programming", Nonlinear Analysis: Hybrid Systems, December 2024.BibTeX TR2024-008 PDF
- @article{Shirai2024dec,
- author = {Shirai, Yuki and Jha, Devesh K. and Raghunathan, Arvind and Romeres, Diego},
- title = {Chance-Constrained Optimization for Contact-rich Systems using Mixed Integer Programming},
- journal = {Nonlinear Analysis: Hybrid Systems},
- year = 2024,
- month = dec,
- url = {https://www.merl.com/publications/TR2024-008}
- }
, - "Hierarchical planning for autonomous parking in dynamic environments", IEEE Transactions on Control Systems Technology, DOI: 10.1109/TCST.2024.3367468, March 2024.BibTeX TR2024-034 PDF
- @article{Wang2024mar2,
- author = {Wang, Yebin and Hansen, Emma and Ahn, Heejin},
- title = {Hierarchical planning for autonomous parking in dynamic environments},
- journal = {IEEE Transactions on Control Systems Technology},
- year = 2024,
- month = mar,
- doi = {10.1109/TCST.2024.3367468},
- issn = {1558-0865},
- url = {https://www.merl.com/publications/TR2024-034}
- }
, - "Multi-level Reasoning for Robotic Assembly: From Sequence Inference to Contact Selection", IEEE International Conference on Robotics and Automation (ICRA), March 2024.BibTeX TR2024-033 PDF Video
- @inproceedings{Zhu2024mar,
- author = {Zhu, Xinghao and Jha, Devesh K. and Romeres, Diego and Sun, Lingfeng and Tomizuka, Masayoshi and Cherian, Anoop},
- title = {Multi-level Reasoning for Robotic Assembly: From Sequence Inference to Contact Selection},
- booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
- year = 2024,
- month = mar,
- url = {https://www.merl.com/publications/TR2024-033}
- }
, - "Stochastic Learning Manipulation of Object Pose With Under-Actuated Impulse Generator Arrays", International Conference on Machine Learning and Applications (ICMLA), DOI: 10.1109/ICMLA58977.2023.00024, December 2023, pp. 112-119.BibTeX TR2023-151 PDF
- @inproceedings{Kong2023dec,
- author = {Kong, Chuizheng and Yerazunis, William S. and Nikovski, Daniel},
- title = {Stochastic Learning Manipulation of Object Pose With Under-Actuated Impulse Generator Arrays},
- booktitle = {International Conference on Machine Learning and Applications (ICMLA)},
- year = 2023,
- pages = {112--119},
- month = dec,
- doi = {10.1109/ICMLA58977.2023.00024},
- url = {https://www.merl.com/publications/TR2023-151}
- }
, - "Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks", Advances in Neural Information Processing Systems (NeurIPS) Workshop on Instruction Tuning and Instruction Following, December 2023.BibTeX TR2023-148 PDF
- @inproceedings{Sun2023dec,
- author = {Sun, Lingfeng and Jha, Devesh K. and Hori, Chiori and Jain, Siddarth and Corcodel, Radu and Zhu, Xinghao and Tomizuka, Masayoshi and Romeres, Diego},
- title = {Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks},
- booktitle = {Advances in Neural Information Processing Systems (NeurIPS) Workshop on Instruction Tuning and Instruction Following},
- year = 2023,
- month = dec,
- url = {https://www.merl.com/publications/TR2023-148}
- }
, - "Physics Informed Gaussian Process Regression Methods for Robot Inverse Dynamics Identification", Conferenza Italiana di Robotica e Macchine Intelligenti, October 2023.BibTeX TR2023-132 PDF
- @inproceedings{Giacomuzzo2023oct2,
- author = {Giacomuzzo, Giulio and Dalla Libera, Alberto and Romeres, Diego and Carli, Ruggero},
- title = {Physics Informed Gaussian Process Regression Methods for Robot Inverse Dynamics Identification},
- booktitle = {Conferenza Italiana di Robotica e Macchine Intelligenti},
- year = 2023,
- month = oct,
- url = {https://www.merl.com/publications/TR2023-132}
- }
, - "EARL: Eye-on-Hand Reinforcement Learner for Dynamic Grasping with Active Pose Estimation", 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), DOI: 10.1109/IROS55552.2023.10341988, October 2023, pp. 2963-2970.BibTeX TR2023-118 PDF Video
- @inproceedings{Huang2023oct,
- author = {Huang, Baichuan and Yu, Jingjin and Jain, Siddarth},
- title = {EARL: Eye-on-Hand Reinforcement Learner for Dynamic Grasping with Active Pose Estimation},
- booktitle = {2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
- year = 2023,
- pages = {2963--2970},
- month = oct,
- publisher = {IEEE},
- doi = {10.1109/IROS55552.2023.10341988},
- issn = {2153-0866},
- isbn = {978-1-6654-9190-7},
- url = {https://www.merl.com/publications/TR2023-118}
- }
, - "Constrained Dynamic Movement Primitives for Collision Avoidance in Novel Environments", IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), DOI: 10.1109/IROS55552.2023.10341839, September 2023, pp. 3672-3679.BibTeX TR2023-121 PDF Video
- @inproceedings{Shaw2023sep,
- author = {Shaw, Seiji and Jha, Devesh K. and Raghunathan, Arvind and Corcodel, Radu and Romeres, Diego and Konidaris, George and Nikovski, Daniel},
- title = {Constrained Dynamic Movement Primitives for Collision Avoidance in Novel Environments},
- booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
- year = 2023,
- pages = {3672--3679},
- month = sep,
- publisher = {IEEE},
- doi = {10.1109/IROS55552.2023.10341839},
- issn = {2153-0866},
- isbn = {978-1-6654-9190-7},
- url = {https://www.merl.com/publications/TR2023-121}
- }
,
- "Chance-Constrained Optimization for Contact-rich Systems using Mixed Integer Programming", Nonlinear Analysis: Hybrid Systems, December 2024.
-
Videos
-
Software & Data Downloads