I like poking fun at game mechanics as much as the next guy... Okay, maybe a little bit more than most. But realism has always taken a back seat to enjoyment here.
The shotgun as it exists now is a crucial tool for many early game builds and tactics. Changes to that mechanic, especially the auto-hit property, are unlikely. Recoil itself is already on par with other damage sources--totally unrealistic but fun. There is no tracking of mass in DoomRL, and 'flying' enemies don't really exist in DoomRL either; in theory we could slap another var and another being flag into the engine for that, but in practice I don't see that happening.
Your gameplay physics need a little retuning. In the real world momentum is more readily conserved than energy, and the recoil felt with a shotgun is minimal. I've yet to find any weapon, shotgun or otherwise, that I cannot absorb the recoil from, let alone one that actually knocks me back far enough to justify our level of knockback. Given that our knockback is unrealistic, trying to account for the potential loss in kinetic energy seems a little silly. It's not an issue in real life; trying to apply it here contradicts the Rule of Cool.
That leaves two ideas that may gain traction: Center of cone damage bonuses (The shotgun cones themselves can be different sizes, but there's no damage dropoff for the edge) and beings soaking up some of the damage when they are in front of other beings (the being that is further away will take less damage due to distance dropoff, but the being in front doesn't block anything). Of the two, the first one is *probably* the easier one to implement, but the harder one to evaluate balance-wise. The second one I wouldn't sweat balance-wise.
