Zifan Xu

🤖 RL for Robotics

1. Learning Agile Striker Skills for Humanoid Soccer Robots from Noisy Sensory Input

   Zifan Xu, Myoungkyu Seo , Dongmyeong Lee , Hao Fu , Jiaheng Hu , and 6 more authors

   In Proceedings of the 2026 IEEE International Conference on Robotics and Automation (ICRA) , May 2026

   Abs arXiv Bib PDF Code Website

   Learning fast and robust ball-kicking skills is a critical capability for humanoid soccer robots, yet it remains a challenging problem due to the need for rapid leg swings, postural stability on a single support foot, and robustness under noisy sensory input and external perturbations (e.g., opponents). This paper presents a reinforcement learning (RL)-based training pipeline that enables humanoid robots to execute robust continual ball-kicking with adaptability to different ball-goal configurations. The pipeline extends a typical teacher-student training framework—in which a teacher policy is trained with ground truth state information and the student learns to mimic it with noisy, imperfect sensing—by including four training stages: (1) long-distance ball chasing (teacher); (2) directional kicking (teacher); (3) teacher policy distillation (student); and (4) student adaptation and refinement (student). Key design elements—including tailored reward functions, realistic noise modeling, and online constrained RL for adaptation and refinement—are critical for closing the sim-to-real gap and sustaining performance under perceptual uncertainty.

   @inproceedings{ICRA2026-Xu,
     author = {Xu, Zifan and Seo, Myoungkyu and Lee, Dongmyeong and Fu, Hao and Hu, Jiaheng and Cui, Jiaxun and Jiang, Yuqian and Wang, Zhihan and Brund, Anastasiia and Biswas, Joydeep and Stone, Peter},
     title = {Learning Agile Striker Skills for Humanoid Soccer Robots from Noisy Sensory Input},
     booktitle = {Proceedings of the 2026 IEEE International Conference on Robotics and Automation (ICRA)},
     year = {2026},
     month = may,
   }

2. Dexterous Legged Locomotion in Confined 3D Spaces with Reinforcement Learning

   Zifan Xu, Amir Hossain Raj , Xuesu Xiao , and Peter Stone

   In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA) , May 2024

   Abs Bib PDF

   Recent advances of locomotion controllers utilizing deep reinforcement learning (RL) have yielded impressive results in terms of achieving rapid and robust locomotion across challenging terrain, such as rugged rocks, non-rigid ground, and slippery surfaces. However, while these controllers primarily address challenges underneath the robot, relatively little research has investigated legged mobility through confined 3D spaces, such as narrow tunnels or irregular voids, which impose all-around constraints. The cyclic gait patterns resulted from existing RL-based methods to learn parameterized locomotion skills characterized by motion parameters, such as velocity and body height, may not be adequate to navigate robots through challenging confined 3D spaces, requiring both agile 3D obstacle avoidance and robust legged locomotion. Instead, we propose to learn locomotion skills end-to-end from goal-oriented navigation in confined 3D spaces. To address the inefficiency of tracking distant navigation goals, we introduce a hierarchical locomotion controller that combines a classical planner tasked with planning waypoints to reach a faraway global goal location, and an RL-based policy trained to follow these waypoints by generating low-level motion commands. This approach allows the policy to explore its own locomotion skills within the entire solution space and facilitates smooth transitions between local goals, enabling long-term navigation towards distant goals. In simulation, our hierarchical approach succeeds at navigating through demanding confined 3D environments, outperforming both pure end-to-end learning approaches and parameterized locomotion skills. We further demonstrate the successful real-world deployment of our simulation-trained controller on a real robot.

   @inproceedings{ICRA2024-Xu,
     author = {Xu, Zifan and Raj, Amir Hossain and Xiao, Xuesu and Stone, Peter},
     title = {Dexterous Legged Locomotion in Confined 3D Spaces with Reinforcement Learning},
     booktitle = {Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA)},
     year = {2024},
     month = may,
     location = {Vienna, Austria},
   }

3. Benchmarking Reinforcement Learning Techniques for Autonomous Navigation

   Zifan Xu, Bo Liu , Anirudh Nair , Xuesu Xiao , and Peter Stone

   Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), May 2023

   Abs Bib PDF

   Deep reinforcement learning (RL) has brought many successes for autonomous robot navigation. However, there still exists important limitations that prevent real-world use of RL-based navigation systems. For example, most learning approaches lack safety guarantees; and learned navigation systems may not generalize well to unseen environments. Despite a variety of recent learning techniques to tackle these challenges in general, a lack of an open-source benchmark and reproducible learning methods specifically for autonomous navigation makes it difficult for roboticists to choose what learning methods to use for their mobile robots and for learning researchers to identify current shortcomings of general learning methods for autonomous navigation. In this paper, we identify four major desiderata of applying deep RL approaches for autonomous navigation: (D1) reasoning under uncertainty, (D2) safety, (D3) learning from limited trial-and-error data, and (D4) generalization to diverse and novel environments. Then, we explore four major classes of learning techniques with the purpose of achieving one or more of the four desiderata: memory-based neural network architectures (D1), safe RL (D2), model-based RL (D2, D3), and domain randomization (D4). By deploying these learning techniques in a new open-source large- scale navigation benchmark and real-world environments, we perform a comprehensive study aimed at establishing to what extent can these techniques achieve these desiderata for RL- based navigation systems

   @article{2023ICRA-Xu,
     title = {Benchmarking Reinforcement Learning Techniques for Autonomous Navigation},
     author = {Xu, Zifan and Liu, Bo and Nair, Anirudh and Xiao, Xuesu and Stone, Peter},
     journal = {Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA)},
     location = {London, England},
     month = may,
     year = {2023},
   }

4. APPLR: Adaptive Planner Parameter Learning from Reinforcement

   Zifan Xu, Gauraang Dhamankar , Anirudh Nair , Xuesu Xiao , Garrett Warnell , and 3 more authors

   In Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA) , Jun 2021

   Abs Bib PDF Video Code Slides Website

   Classical navigation systems typically operate using a fixed set of hand-picked parameters (e.g. maximum speed, sampling rate, inflation radius, etc.) and require heavy expert re-tuning in order to work in new environments. To mitigate this requirement, it has been proposed to learn parameters for different contexts in a new environment using human demonstrations collected via teleoperation. However, learning from human demonstration limits deployment to the training environment, and limits overall performance to that of a potentially-suboptimal demonstrator. In this paper, we introduce APPLR, Adaptive Planner Parameter Learning from Reinforcement, which allows existing navigation systems to adapt to new scenarios by using a parameter selection scheme discovered via reinforcement learning (RL) in a wide variety of simulation environments. We evaluate APPLR on a robot in both simulated and physical experiments, and show that it can outperform both a fixed set of hand-tuned parameters and also a dynamic parameter tuning scheme learned from human demonstration.

   @inproceedings{ICRA2021-Xu,
     author = {Xu, Zifan and Dhamankar, Gauraang and Nair, Anirudh and Xiao, Xuesu and Warnell, Garrett and Liu, Bo and Wang, Zizhao and Stone, Peter},
     title = {APPLR: Adaptive Planner Parameter Learning from Reinforcement},
     booktitle = {Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA)},
     location = {Xi'an, China},
     month = jun,
     year = {2021},
   }

📘 Large-Scale RL

1. ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priors

   Zifan Xu, Ran Gong , Maria Vittoria Minniti , Ahmet Salih Gundogdu , Eric Rosen , and 7 more authors

   In Proceedings of the 2026 Conference on Robot Learning (CoRL) , 2026

   Abs arXiv Bib PDF Website

   Learning generalizable and robust behavior cloning policies requires large volumes of high-quality robotics data. While human demonstrations (e.g., through teleoperation) serve as the standard source for expert behaviors, acquiring such data at scale in the real world is prohibitively expensive. This paper introduces ExpertGen, a framework that automates expert policy learning in simulation to enable scalable sim-to-real transfer. ExpertGen first initializes a behavior prior using a diffusion policy trained on imperfect demonstrations, which may be synthesized by large language models or provided by humans. Reinforcement learning is then used to steer this prior toward high task success by optimizing the diffusion model’s initial noise while keep original policy frozen. By keeping the pretrained diffusion policy frozen, ExpertGen regularizes exploration to remain within safe, human-like behavior manifolds, while also enabling effective learning with only sparse rewards. Empirical evaluations on challenging manipulation benchmarks demonstrate that ExpertGen reliably produces high-quality expert policies with no reward engineering. On industrial assembly tasks, ExpertGen achieves a 90.5% overall success rate, while on long-horizon manipulation tasks it attains 85% overall success, outperforming all baseline methods. The resulting policies exhibit dexterous control and remain robust across diverse initial configurations and failure states. To validate sim-to-real transfer, the learned state-based expert policies are further distilled into visuomotor policies via DAgger and successfully deployed on real robotic hardware.

   @inproceedings{ExpertGen2026-Xu,
     author = {Xu, Zifan and Gong, Ran and Minniti, Maria Vittoria and Gundogdu, Ahmet Salih and Rosen, Eric and Sivakumar, Kausik and Yan, Riedana and Wang, Zixing and Deng, Di and Stone, Peter and Zhang, Xiaohan and Schmeckpeper, Karl},
     title = {ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priors},
     booktitle = {Proceedings of the 2026 Conference on Robot Learning (CoRL)},
     year = {2026},
   }

2. Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks

   Ziping Xu , Zifan Xu, Runxuan Jiang , Peter Stone , and Ambuj Tewari

   In Proceedings of the International Conference on Learning Representations (ICLR) , May 2024

   Abs Bib PDF

   Multitask Reinforcement Learning (MTRL) approaches have gained increasing attention for its wide applications in many important Reinforcement Learning (RL) tasks. However, while recent advancements in MTRL theory have focused on the improved statistical efficiency by assuming a shared structure across tasks, exploration–a crucial aspect of RL–has been largely overlooked. This paper addresses this gap by showing that when an agent is trained on a sufficiently diverse set of tasks, a generic policy-sharing algorithm with myopic exploration design like ε-greedy that are inefficient in general can be sample-efficient for MTRL. To the best of our knowledge, this is the first theoretical demonstration of the "exploration benefits" of MTRL. It may also shed light on the enigmatic success of the wide applications of myopic exploration in practice. To validate the role of diversity, we conduct experiments on synthetic robotic control environments, where the diverse task set aligns with the task selection by automatic curriculum learning, which is empirically shown to improve sample-efficiency.

   @inproceedings{ICLR2024-Xu,
     author = {Xu, Ziping and Xu, Zifan and Jiang, Runxuan and Stone, Peter and Tewari, Ambuj},
     title = {<a href="https://openreview.net/forum?id=YZrg56G0JV">
                     <div class="link">
                       Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks
                     </div>
                   </a>
                   },
     booktitle = {Proceedings of the International Conference on Learning Representations (ICLR)},
     year = {2024},
     month = may,
     location = {Vienna, Austria},
   }

3. Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks

   Viraj Joshi^* , Zifan Xu^*, Bo Liu , Peter Stone , and Amy Zhang

   In Reinforcement Learning Conference (RLC) , Aug 2025

   Abs PDF Code

   Multi-task Reinforcement Learning (MTRL) has emerged as a critical training paradigm for applying reinforcement learning (RL) to a set of complex real-world robotic tasks, which demands a generalizable and robust policy. At the same time, massively parallelized training has gained popularity, not only for significantly accelerating data collection through GPU-accelerated simulation but also for enabling diverse data collection across multiple tasks by simulating heterogeneous scenes in parallel. However, existing MTRL research has largely been limited to off-policy methods like SAC in the low-parallelization regime. MTRL could capitalize on the higher asymptotic performance of on-policy algorithms, whose batches require data from the current policy, and as a result, take advantage of massive parallelization offered by GPU-accelerated simulation. To bridge this gap, we introduce a massively parallelized Multi-Task Benchmark for robotics (MTBench), an open-sourced benchmark featuring a broad distribution of 50 manipulation tasks and 20 locomotion tasks, implemented using the GPU-accelerated simulator IsaacGym. MTBench also includes four base RL algorithms combined with seven state-of-the-art MTRL algorithms and architectures, providing a unified framework for evaluating their performance. Our extensive experiments highlight the superior speed of evaluating MTRL approaches using MTBench, while also uncovering unique challenges that arise from combining massive parallelism with MTRL. Code is available at https://github.com/Viraj-Joshi/MTBench.

💭 LLMs Reasoning

1. LaRS: Latent Reasoning Skills for Chain-of-Thought Reasoning

   Zifan Xu, Haozhu Wang , Dmitriy Bespalov , Xian Wu , Peter Stone , and 1 more author

   In Findings of Empirical Methods in Natural Language Processing , Nov 2024

   Abs arXiv Bib

   Chain-of-thought (CoT) prompting is a popular in-context learning (ICL) approach for large language models (LLMs), especially when tackling complex reasoning tasks. Traditional ICL approaches construct prompts using examples that contain questions similar to the input question. However, CoT prompting, which includes crucial intermediate reasoning steps (rationales) within its examples, necessitates selecting examples based on these rationales rather than the questions themselves. Existing methods require human experts or pre-trained LLMs to describe the skill, a high-level abstraction of rationales, to guide the selection. These methods, however, are often costly and difficult to scale. Instead, this paper introduces a new approach named Latent Reasoning Skills (LaRS) that employs unsupervised learning to create a latent space representation of rationales, with a latent variable called a reasoning skill. Concurrently, LaRS learns a reasoning policy to determine the required reasoning skill for a given question. Then the ICL examples are selected by aligning the reasoning skills between past examples and the question. This approach is theoretically grounded and compute-efficient, eliminating the need for auxiliary LLM inference or manual prompt design. Empirical results demonstrate that LaRS consistently outperforms SOTA skill-based selection methods, processing example banks four times faster, reducing LLM inferences during the selection stage by half, and showing greater robustness to sub-optimal example banks.

   @inproceedings{Latent2023-Xu,
     title = {LaRS: Latent Reasoning Skills for Chain-of-Thought Reasoning},
     author = {Xu, Zifan and Wang, Haozhu and Bespalov, Dmitriy and Wu, Xian and Stone, Peter and Qi, Yanjun},
     booktitle = {Findings of Empirical Methods in Natural Language Processing},
     location = {Miami, Florida},
     month = nov,
     year = {2024},
   }