Be it on real roads or in the world of simulations, one natural goal of creating an AV/ADAS system is to avoid collisions and close-calls. At evaluation time, an important consideration becomes clear: The behaviors of other agents around the System-Under-Test (SUT) vehicles matter a lot.

Unsurprisingly, models struggle to avoid collisions when other vehicles behave as though the SUT vehicle doesn’t exist. Unreactive, log replay non-playable character vehicles (NPC) following their ground truth trajectories often create situations where it is impossible for the SUT to react appropriately. This makes sense. How is an SUT supposed to learn to avoid collisions when all negotiations are unilateral and the log replay NPCs will simply follow set trajectories regardless of SUT behavior? One training strategy would be to only focus on collisions that indicate an issue with the SUT and fix how it negotiates that particular scenario. However, this sorting process must be done tediously by hand due to the high number of false-positive collisions that indicate nothing about the SUT, which consumes precious resources.

At Inverted AI, we take pride in generating safer, realistic driving simulations where both the SUT and the other background NPCs can negotiate intelligent and safe decisions back and forth between each other. This involves reactive NPCs that will change their behavior based on the actions of the ego vehicle and vice versa. As an example, highways are a typical entry point for testing AV and ADAS systems. We found that running reactive NPCs reduces collisions by a factor of 3 to 8 compared to replaying the NPCs with real driving log data.

	Log replay - Total collision rate	Reactive NPCs - Total collision rate	Reactive NPCs - Ego fault collision rate
Highway merge	0.0112	0.0014	0.0008
Highway lane change	0.0068	0.0023	0.0011

Below is a selection of side-by-side comparisons between log replay (left) and reactive NPCs (right):

• For sparse or dense traffic, reactive NPCs significantly reduce rear-ending incidents.

• Reactive NPCs make courtesy yields for the merging SUT.

• Reactive NPCs are more aware of surroundings and exhibit behavior like making way for faster traffic, and delaying lane changes for safety.
  

 

These are just a few of the diverse range of scenarios in which our NPCs are reacting to the behavior of an SUT and allowing it the opportunity to learn real-world, realistic negotiation capabilities. Please get in touch to learn more about our reactive AV/ADAS evaluation pipeline through our realistic, reactive, and diverse NPC agents.