Separating where to go from how to walk let a simulated quadruped reach 94.2% navigation success, including 88.6% on map layouts it had never seen.

This is simulation research, presented as a conference paper at ICRoM 2025. A public proceedings record is not available yet. My M.Sc. thesis extends this work and is still in progress, and this page reports no hardware results.

Read the paper summary and original figures or download the complete conference manuscript (PDF).

The problem

A legged robot crossing cluttered, uneven ground has two jobs at once: choose a route around obstacles, and place its feet so it doesn’t fall. Training one “monolithic” policy to do both mixes very different objectives, and it tends to generalize poorly to new layouts.

The approach

A high-level navigation policy works on a 2D map and goal, and outputs a body-frame velocity command. That command is bounded, so the navigation policy can’t ask for motion the locomotion policy can’t deliver.

A low-level locomotion policy tracks the command in 3D. It uses 187 terrain height samples around the robot’s body and does no explicit footstep planning.

Both policies were trained and evaluated in GPU-parallel Isaac Gym simulation. In the reported setup, training took about five hours on a single GPU.

My role

I was second author on a three-person paper. I designed the hierarchical structure and implemented the reinforcement-learning path-planning component. In the lab, I also define requirements, evaluation criteria and success metrics for this line of experiments.

Results (simulation)

These numbers come from simulation under the paper’s evaluation protocol, and none of them were measured on a physical robot.

How this carries over to product work

Deciding what the upper layer may request, and what the lower layer guarantees, turned one hard problem into two testable ones, and each layer could be evaluated against its own success criteria. I use the same approach when I scope product features and split work between teams.

Thesis work, in progress

The thesis extends the architecture toward risk-aware navigation. It combines global graph search (Dijkstra) with sampling-based local planning (MPPI) and a distributional reinforcement-learning controller for locomotion. Results will be published once the thesis is complete.

Citation

Y. Ayoubi Rad, S. M. Sarfarazi and M. Shahbazi, “Hierarchical Deep Reinforcement Learning for Quadruped Navigation over Complex Terrains,” 13th RSI International Conference on Robotics and Mechatronics (ICRoM 2025), Tehran, December 2025. Author-hosted conference manuscript (PDF). A proceedings link will be added when the public record is available.

Read the full paper (PDF · 6 pages)

Authors: Yousef Ayoubi Rad, Seyyed Mahdi Sarfarazi and Mohammad Shahbazi.
Affiliation: AI & Dynamic Systems Lab, School of Mechanical Engineering, Iran University of Science and Technology.
Conference: 13th RSI International Conference on Robotics and Mechatronics (ICRoM 2025), December 16–18, 2025, Tehran, Iran.

The downloadable PDF is the conference manuscript supplied by the authors. All results below were obtained in simulation. This page does not report deployment on a physical robot.

Abstract

Quadruped robots are promising for tasks like disaster response and exploration, navigating cluttered spaces while maintaining stability over uneven terrains such as stairs, slopes, and obstacles. This paper presents a hierarchical deep reinforcement learning (HRL) framework that decouples high-level navigation from low-level locomotion via a compact velocity-command interface u = (vₓ, vᵧ, ωz). The high-level policy generates smooth body-frame velocity commands for goal-directed movement in 2D maps, while the low-level policy executes these commands using joint targets with PD control, enhanced by proprioceptive feedback and local terrain sampling. This modular approach mitigates reward conflicts, boosts training efficiency, and enables independent layer optimization in Isaac Gym. Evaluations on maze-like maps and varied terrains show reliable goal reaching, stable tracking, and improved robustness over monolithic baselines. The results underscore how hierarchical decomposition enhances generalization and stability, offering a promising path for intelligent quadruped mobility in real-world applications.

Why separate navigation from locomotion?

Choosing a route and keeping a legged robot balanced involve different observations, actions and time scales. Training a single policy to solve both can introduce competing rewards and make it harder to reuse a locomotion controller on a new map.

The proposed hierarchy connects the policies through a bounded body-frame velocity command. The navigation layer uses the robot’s pose, relative goal and obstacle information. The locomotion layer uses body and joint measurements together with a fixed pattern of 187 local terrain-height samples. Joint targets are tracked by a PD controller. Command saturation and smoothing keep the interface within the motion limits used in the simulation.

Simulation and terrain awareness

Experiments use Isaac Gym with a Legged-Gym-style task wrapper. The terrain suite includes stairs, ramps, discrete obstacles and sinusoidal surfaces. Navigation is trained in a 10 × 10 m workspace with procedurally generated obstacles; larger maps are normalized to the training domain.

The high-level planner is trained with Proximal Policy Optimization (PPO), using an MLP with two 256-neuron hidden layers. Training proceeds in stages: first train locomotion on procedural terrains, then hold that policy fixed while training navigation. The paper reports about five hours of combined training on an NVIDIA RTX 3080 Ti, with roughly 12,000 parallel simulation environments.

From a map to a velocity command

Input occupancy maps are converted into obstacle features for the navigation policy. The planner uses goal and obstacle information to produce commands, while the locomotion policy handles contact and balance in the physics simulation.

Reported results

The following values reproduce Table I of the paper. Navigation success, collision rate, path ratio and linear velocity-tracking error are distinct measures; navigation success alone does not describe end-to-end performance on uneven terrain.

The paper also reports 88.6% navigation success on unseen map layouts, without retraining, and 87.4% end-to-end success when obstacles and terrain variations are combined. Compared with the monolithic baseline, collisions per episode fall from 0.28 to 0.08, a 71.4% reduction. Removing terrain sensing raises the reported fall rate from 1.4% to 7.3%.

Scope and limitations

Evaluation uses simulation, static obstacles and nominal sensing. The local height representation does not provide semantic understanding or long-range terrain topology. Tight bends and small-platform sequences can slow progress, and freezing locomotion during navigation training limits how the two layers adapt together.

Online mapping, dynamic obstacles, selective co-training and hardware validation remain directions for further work. The reported simulation results are not evidence of real-world reliability.

For my contribution and the ongoing thesis work, see the related research case study.

Paper and citation

Download the complete conference manuscript (PDF).

Y. Ayoubi Rad, S. M. Sarfarazi and M. Shahbazi, “Hierarchical Deep Reinforcement Learning for Quadruped Navigation over Complex Terrains,” 13th RSI International Conference on Robotics and Mechatronics (ICRoM 2025), Tehran, Iran, December 16–18, 2025.

All five figures on this page were extracted from the supplied Word manuscript. The full PDF contains the equations, references and complete discussion.