NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidance

Wenzhe Cai  Jiaqi Peng  Yuqiang Yang  Yujian Zhang  Meng Wei 
Hanqing Wang  Yilun Chen  Tai Wang  Jiangmiao Pang 
Shanghai AI Laboratory  Tsinghua University 
Zhejiang University  The University of Hong Kong 

🔥 News

• We release the InternNav - an all-in-one open-source toolbox for embodied naivgation.
• We open source a comprehensive navigation system-1 benchmark, check here for more details!
• We release a high-quality Vision-Language-Navigation (VLN) dataset - InternData-N1!
• We release a new NavDP checkpoint - InternVLA-N1-S1.

🏡 Introduction

Navigation Diffusion Policy (NavDP) is an end-to-end mapless navigation model that can achieves cross-embodiment generalization without any real-world robot data. By building a highly efficient simulation data generation pipeline as well as the superior model design, NavDP achieves real-time path-planning and obstacle avoidance across various navigation tasks, including nogoal exploration, pointgoal navigation, imagegoal navigation.

💻 InternVLA-N1 System-1 Model

Please fill this form to access the link to download the latest model checkpoint.

🛠️ Installation

Please follow the instructions to config the environment for NavDP.

Step 0: Clone this repository

git clone https://github.com/InternRobotics/NavDP
cd NavDP/system1_baselines/navdp/

Step 1: Create conda environment and install the dependency

conda create -n navdp python=3.10
conda activate navdp
pip install -r requirements.txt

🤖 Run NavDP Model

Run the following line to start navdp server:

python navdp_server.py --port ${YOUR_PORT} --checkpoint ${SAVE_PTH_PATH}

Then, follow the subsequent tutorial to build the environment for IsaacSim and start the evaluation in simulation. By running with our benchmark, you should be able to replicate the navigation examples below:

NoGoal Exploration

PointGoal Navigation

ImageGoal Navigation

🎢 InternVLA-N1 System-1 Benchmark

🏠 Overview

This repository is a high-fidelity platform for benchmarking the visual navigation methods based on IsaacSim and IsaacLab. With realistic physics simulation and realistic scene assets, this repository aims to build an benchmark that can minimizing the sim-to-real gap in navigation system-1 evaluation.

Highlights

• ⭐ Decoupled Framework between Navigation Approaches and Evaluation Process

The evaluation is accomplished by calling navigation method api with HTTP requests. By decoupling the implementation of navigation model with the evaluation process, it is much easier for users to evaluate the performance of novel navigation methods.

• ⭐ Fully Asynchronous Framework between Trajectory Planning and Following

We implement a MPC-based controller to constantly track the planned trajectory. With the asynchronous framework, the evaluation metrics become related to the navigation approaches' decision frequency which help align with the real-world navigation performance.

• ⭐ High-Quality Scene Asset for Evaluation

Our benchmark supports evaluation in diverse scene assets, including random cluttered environments, realistic home scenarios and commercial scenarios.

• ⭐ Support Image-Goal, Point-Goal and No-Goal Navigation Tasks

Our benchmark supports multiple navigation tasks, including no-goal exploration, point-goal navigation as well as image-goal navigation.

📋 Table of Contents

• 🏠 Overview
• 🌆 Prepare Scene Asset
• 🔧 Installation of Benchmark
• ⚙️ Installation of Baseline Library
• 💻 Running Basline as Server
• 🕹️ Running Teleoperation
• 📊 Running Evaluation
• 🔗 Citation
• 📄 License
• 👏 Acknowledgements

🌆 Prepare Scene Asset

Please download the scene asset from InternScene-N1 at HuggingFace. The episodes information can be directly accessed in this repo. After downloading, please organize the structure as follows:

asset_scenes/
├── SkyTexture/
│   ├── belfast_sunset_puresky_4k.hdr
│   ├── citrus_orchard_road_puresky_4k.hdr
│   ├── ...
├── Materials/
│   ├── Carpet
│       ├── textures/
│       ├── Carpet_Woven.mdl
│       └── ...
│   ├── ...
├── cluttered_easy/
│   └── easy_0/
│       ├── cluttered-0.usd/
│       ├── imagegoal_start_goal_pairs.npy
│       └── pointgoal_start_goal_pairs.npy
│   ├── ...
├── cluttered_hard/
│   └── hard_0/
│       ├── cluttered-0.usd/
│       ├── imagegoal_start_goal_pairs.npy
│       └── pointgoal_start_goal_pairs.npy
│   ├── ...
├── internscenes_commercial/
│   └── MV4AFHQKTKJZ2AABAAAAADQ8_usd/
│       ├── models/
│       ├── Materials/
│       ├── metadata.json
│       ├── start_result_navigation.usd
│       ├── imagegoal_start_goal_pairs.npy
│       └── pointgoal_start_goal_pairs.npy
│   ├── ...
├── internscene_home/
│   └── MV4AFHQKTKJZ2AABAAAAADQ8_usd/
│       ├── models/
│       ├── Materials/
│       ├── metadata.json
│       ├── start_result_navigation.usd
│       ├── imagegoal_start_goal_pairs.npy
└──     └── pointgoal_start_goal_pairs.npy

Category	Download Asset	Episodes
SkyTexture	Link	-
Materials	Link	-
Cluttered-Easy	Link	Episodes
Cluttered-Hard	Link	Episodes
InternScenes-Home	Link	Episodes
InternScenes-Commercial	Link	Episodes

Note: The textures and dataset are still waiting for uploading to HuggingFace

🔧 Installation of Benchmark

Our framework is based on IsaacSim 4.2.0 and IsaacLab 1.2.0, you can follow the instructions to configure the conda environment.

# create the environment
conda create -n isaaclab python=3.10
conda activate isaaclab

# install IsaacSim 4.2
pip install --upgrade pip
pip install isaacsim==4.2.0.2 isaacsim-extscache-physics==4.2.0.2 isaacsim-extscache-kit==4.2.0.2 isaacsim-extscache-kit-sdk==4.2.0.2 --extra-index-url https://pypi.nvidia.com
# check the isaacsim installation
isaacsim omni.isaac.sim.python.kit

# install IsaacLab 1.2.0
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab/
git checkout tags/v1.2.0

# ignore the rsl-rl unavailable error
./isaaclab.sh -i 

# check the isaaclab installation
./isaaclab.sh -p source/standalone/tutorials/00_sim/create_empty.py

After preparing for the dependencies, please clone our project to get started.

git clone https://github.com/InternRobotics/NavDP.git
cd NavDP
git checkout navdp_benchmark
pip install -r requirements.txt

⚙️ Installation of Baseline Library

We collect the checkpoints for other navigation system-1 method from the corresponding respitory and organize their code to support the HTTP api calling for our benchmark. The links of paper, github codes as well as the pre-trained checkpoints are listed in the table below. Some of the baselines requires additional dependencies, and we provide the installation details below.

git clone https://github.com/InternRobotics/NavDP.git
cd NavDP
git checkout navdp_baseline
# you can rename the branch directory as nav_system1_baseline
mv ../NavDP ../nav_system1_baseline

Baseline	Paper	Repo	Checkpoint	Support Tasks
DD-PPO	Arxiv	GitHub	Checkpoint	PointNav
iPlanner	Arxiv	GitHub	Checkpoint	PointNav
ViPlanner	Arxiv	GitHub	Checkpoint Mask2Former	PointNav
GNM	Arxiv	GitHub	Checkpoint	ImageNav, NoGoal
ViNT	Arxiv	GitHub	Checkpoint	ImageNav, NoGoal
NoMad	Arxiv	GitHub	Checkpoint	ImageNav, NoGoal
NavDP	Arxiv	GitHub	Checkpoint	PointNav, ImageNav, NoGoal

DD-PPO

To verify the performance of DD-PPO with continuous action space, we interpolate the predicted discrete actions {Stop, Forward, TurnLeft, TurnRight} into a trajectory. To play with the DD-PPO in our benchmark, you need to install the habitat-lab and habitat-baselines. As Habitat only supports python <= 3.9, we recommand to create a new environment.

conda create -n habitat python=3.9 cmake=3.14.0
conda activate habitat
conda install habitat-sim withbullet -c conda-forge -c aihabitat
git clone --branch stable https://github.com/facebookresearch/habitat-lab.git
cd habitat-lab
pip install -e habitat-lab
pip install -e habitat-baselines

iPlanner

No addition dependencies are required if you have configured the environment for running the benchmark.

ViPlanner

For Viplanner, you need to install the mmcv and mmdet for Mask2Former. We recommand to create a new environment with torch 2.0.1 as backend.

pip install torch==2.0.1+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
pip install torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
pip install mmcv==2.0.0 -f https://download.openmmlab.com/mmcv/dist/cu118/torch2.0/index.html
pip install mmengine mmdet
pip install git+https://github.com/cocodataset/panopticapi.git

GNM, ViNT and NoMad

To play with GNM, ViNT and NoMad, you need to install the following dependencies:

pip install efficientnet_pytorch==0.7.1
pip install diffusers==0.33.1
pip install git+https://github.com/real-stanford/diffusion_policy.git

💻 Running Basline as Server

To install the dependencies for different baseline method, please refer to here. For each pre-built baseline methods, each contains a server.py file, just simply run server python script with parsing the server port as well as the checkpoint path. You can download the checkpoints from the baseline library table. Taking NavDP as an example:

cd nav_system1_baseline/navdp/
python navdp_server.py --port 8888 --checkpoint ./checkpoints/navdp_checkpoint.ckpt

Then, the server will run at backend waiting for RGB-D observations and generate the preferred navigation trajectories.

🕹️ Running Teleoperation

For quickstart or debug with your novel navigation approach, we provide a teleoperation script that the robot move according to your teleoperation command while outputs the predicted trajectory for visualization. With a running server, the teleoperation code can be directly started with one-line command:

# if the running server support no-goal task
python teleop_nogoal_wheeled.py
# if the running server support point-goal task
python teleop_pointgoal_wheeled.py
# if the running server support image-goal task
python teleop_imagegoal_wheeled.py 

Then, you can use 'w','a','s','d' on the keyboard to control the linear and anguler speed.

📊 Running Evaluation

With a running server, it is simple to start the evaluation as:

# if the running server support no-goal task, Please Parse the Absolute Path of the ASSET_SCENE
python eval_nogoal_wheeled.py --port {PORT} --scene_dir {ASSET_SCENE} --scene_index {INDEX}
# if the running server support point-goal task, Please Parse the Absolute Path of the ASSET_SCENE
python eval_pointgoal_wheeled.py --port {PORT} --scene_dir {ASSET_SCENE} --scene_index {INDEX}
# if the running server support image-goal task, Please Parse the Absolute Path of the ASSET_SCENE
python eval_imagegoal_wheeled.py --port {PORT} --scene_dir {ASSET_SCENE} --scene_index {INDEX}

Please parse the port to match the server port, and the scene asset as well as the scene index to decide the evaluate scenario. The evaluation results of the baseline methods will be released in a few days.

📄 License

The open-sourced code are under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License .

🔗 Citation

If you find our work helpful, please cite:

@misc{navdp,
    title = {NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidance},
    author = {Wenzhe Cai, Jiaqi Peng, Yuqiang Yang, Yujian Zhang, Meng Wei, Hanqing Wang, Yilun Chen, Tai Wang and Jiangmiao Pang},
    year = {2025},
    booktitle={arXiv},
}

👏 Acknowledgement

• InternUtopia (Previously GRUtopia): The closed-loop evaluation and GRScenes-100 data in this framework relies on the InternUtopia framework.
• InternNav: All-in-one open-source toolbox for embodied navigation based on PyTorch, Habitat and Isaac Sim.
• Diffusion Policy: Diffusion policy implementation.
• DepthAnything: The foundation representation for RGB image observations.
• ViPlanner: ViPlanner implementation.
• iPlanner: iPlanner implementation.
• visualnav-transformer: NoMad, ViNT, GNM implementation.