Guided Conditional Diffusion for Controllable Traffic Simulation

Ziyuan Zhong1,2 Davis Rempe1,3 Danfei Xu1,4 Yuxiao Chen1
Sushant Veer1 Tong Che1 Baishakhi Ray2 Marco Pavone1,3
1 NVIDIA2 Columbia University3 Stanford University4 Georgia Tech
International Conference on Robotics and Automation (ICRA) 2023


Controllable and realistic traffic simulation is critical for developing and verifying autonomous vehicles. Typical heuristic-based traffic models offer flexible control to make vehicles follow specific trajectories and traffic rules. On the other hand, data-driven approaches generate realistic and human-like behaviors, improving transfer from simulated to real-world traffic. However, to the best of our knowledge, no traffic model offers both controllability and realism. In this work, we develop a conditional diffusion model for controllable traffic generation (CTG) that allows users to control desired properties of trajectories at test time (e.g., reach a goal or follow a speed limit) while maintaining realism and physical feasibility through enforced dynamics. The key technical idea is to leverage recent advances from diffusion modeling and differentiable logic to guide generated trajectories to meet rules defined using signal temporal logic (STL). We further extend guidance to multi-agent settings and enable interaction-based rules like collision avoidance. CTG is extensively evaluated on the nuScenes dataset for diverse and composite rules, demonstrating improvement over strong baselines in terms of the controllability-realism tradeoff.

We present demos for five single-rule settings and two multi-rules settings in the following sections. For each setting, we compare CTG with the two strongest baselines in terms of rule violations for that setting (based on the quantitative results in the main paper). Note that all the videos are played at 5x speed.

Speed Limit

In this scene, the vehicles are supposed to drive under the specified speed limit. BC+opt generates trajectories with a vehicle (red vehicle) driving off-road. Compared with BC+opt and BITS+opt, CTG has the lowest speed limit violation (0.101m/s), and smoother and realistic trajectories.

BC+opt (speed limit violation: 0.224m/s)
BITS+opt (speed limit violation: 0.138m/s)
CTG (speed limit violation: 0.101m/s)

Target Speed

In this scene, the vehicles are supposed to drive following the corrsponding speed at each time step. Both TrafficSim+opt and BC+opt generate trajectories with vehicles (blue vehicle) going off-road. Compared with TrafficSim+opt and BC+opt, CTG has the lowest target speed violation (0.3m/s) and more realistic trajectories.

BC+opt (target speed violation: 1.3m/s)
TrafficSim+opt (target speed violation: 1.1m/s)
CTG (target speed violation: 0.3m/s)

Goal Waypoint

In this scene, the vehicles are supposed to reach the specified goal waypoints (plotted as circles on the map). Although BC+opt has lower goal waypoint violation (2.2m) than CTG (3.2m), both BC+opt and BITS+opt generate curvy, unrealistic trajectories and result in multiple collisions (e.g, purple vehicle and green vehicle). In contrast, CTG generates smooth and realistic trajectories with no collisions.

BC+opt (goal waypoint violation: 2.2m)
BITS+opt (goal waypoint violation: 6.3m)
CTG (goal waypoint violation: 3.2m)

No Off-road

In this scene, the vehicles are supposed to avoid going off-road. Although all the three methods generate trajectories without going off-road, CTG generates more realistic trajectories that are more likely to be observed in real-world traffic. In particular, BC+opt generates trajectories leading to multiple collisions (e.g., red vehicle and purple vehicle). TrafficSim+opt generates trajectories less smooth with one vehicle (blue vehicle) goes to a lane with wrong direction. In contrast, CTG produce smooth trajectories with vehicles all going along correct lanes and having no collisions.

BC+opt (Off-road violation: 0)
TrafficSim+opt (Off-road violation: 0)
CTG (Off-road violation: 0)

No Collision

In this scene, the vehicles are supposed to avoid collisions with each other. BITS generates trajectories leading to multiple vehicles collide with each other at the end (e.g., green vehicle and orange vehicle). Both CTG and BITS+opt do not have collisions. However, BITS+opt generates more curvy, less realistic trajectories (e.g., see the blue vehicle). In contrast, CTG generates smoother, more realistic trajectories.

BITS (Collision Violation: 0.238)
BITS+opt (Collision Violation: 0)
CTG (Collision Violation: 0)

StopSign + No Off-road

In this scene, the vehicles are supposed to stop if they enter the corresponding stop sign regions (denoted as colored square regions) and not go off-road. TrafficSim generates trajectories with vehicles keep driving within the corrsponding stop sign regions and going off road (blue vehicle), and leading to multiple collisions (e.g., green vehicle and orange vehicle). TrafficSim+opt generates trajectories with some vehicles not stopping within the stop regions (purple vehicle) and a collision between two vehicles (orange vehicle and yellow vehicle). In contrast, CTG produce trajectories without going off-road and all vehicles stop within the stop sign regions. CTG has acehieved much lower average minimum speed within the stop regions (0.001m/s VS 2.349m/s and 0.105m/s)

TrafficSim (stop sign violation: 2.349m/s, Off-road violation: 0.143)
TrafficSim+opt (stop sign violation: 0.105m/s, Off-road violation: 0)
CTG (stop sign violation: 0.001m/s, Off-road violation: 0)

Goal Waypoint + Target Speed

In this scene, the vehicles are supposed to reach goal waypoints (colored circles) with specified target speed.TrafficSim generates trajectories with one vehicle not reaching goal waypoint (gray vehicle). For BITS, while vehicles reach waypoints, the gray vehicle collides with the pink vehicle. In contrast, CTG produce trajectories reaching waypoints without collisions. Furthermore, CTG generated trajectories best satisfy the goal waypoint (with smallest goal waypoint violation 0.315m) and target speed (with smallest target speed violation 0.161m/s).

TrafficSim (goal waypoint violation: 1.015m, target speed violation: 0.312m/s)
BITS (goal waypoint violation: 0.391m, target speed violation: 0.285m/s)
CTG (goal waypoint violation: 0.315m, target speed violation: 0.161m/s)

This project page template is based on this page.
For any questions, please contact Ziyuan Zhong at