You might not be seeing much overspray with the primer because it is heavier and doesn't aerosolize as much, but your fan is important to evacuate VOC (especially isocyanate), which are the hazard to you and explosions if they reach a critical mass. No amount of efficiency of your gun can prevent VOC, both as it's being sprayed and as it's flashing off.
You will find clear will make a fog almost instantaneously, whereas heavy primers don't. Regardless of that, you still want to get all the bad joojoo out of the air.