I think I know the answer :)
Its because they are two different weapons, and they apply the damage very differently.
Assuming you fire a DSG at 4 sq range, it should take average 2 shots to kill a Demon.
A RL, should take 1 shot (as long as its a direct hit). With a normal SG, its 2-3 at point blank (so I don't know why its 2-3 point blank for your DSG).
Ok so you're wondering why 4d4 is doing more damage than 6d6? This is because they deal damage
differently.
If you fire the DSG at 1 sq range, you're still not hitting the demon with all of its firepower. You will notice because things behind the demon will take damage as well. So you are dealing 6d6, but not all in the same place. The demon won't take all 100% of the shotguns damage (shotguns have a very wide "inaccurate" spread, with a 100% chance to hit something within that spread, but individual pellets can hit targets behind)
Rockets work differently too. 4d4 is probably an approximation of the splash damage of a rocket, taking one in the chest means you take respectively more damage. A direct hit with a rocket launcher deals 100% of the rockets damage to that target.
I don't know about the damage and knockback thing. I know that knockback is highly related to damage dealt, this works in both Berserk! and DoomRL. The more damage dealt, the more knockback a target receives. Different monsters (and the player) have different weights which modify the final knockback value.