I don't recall how tab-aiming works, but I can imagine it's one of two ways.
1) Monsters are probably stored in a list. If you press tab, the program checks if the next monster in the list is visible (in LOS). If so, the monster is targeted; if not, the next monster is checked. The order of targeting would depend on the order of monsters in the list. As the list is probably unordered, the selection appears random.
2) The program checks an LOS vector in a specific direction for presence of monsters. If a monster is located, it is targeted. If no monster is located, the vector is rotated. This is repeated until a 360 degree circle has been checked. Target selection in this case is clockwise or ccw. This method would be tedious though.
Are you suggesting auto-aim should be smarter? E.g. always pick the closest monster, or the strongest?
Edit: I don't really mind which monster auto-aim picks, as long as it remembers my previous target if it's still alive. Also, you can hit monsters 'partially covered by walls' by targeting past them - actually I consider it to be a bug if a monster is in LOS but the line of fire is blocked, but you get used to it.