Meanwhile there are gigantic holes in the modding documentation including Generator, AI, User Interface, Core, etc. Some stuff can be reverse engineered but not easily.
This is part of the work I'm doing for modding in 0.9.9.6: getting stuff properly documented. Besides the lua code, there's a lot to sift through and understand on the FPC/valkyrie side, and so it's a pretty big effort. I will REALLY try to accomplish this over the next couple of weeks (last week got bogged down by some feature additions and bug fixes).
The changelog refers to two additional module types "Episode" and "Total conversion". What are the details of these?
Total conversions aren't actually available yet: it'll take some pretty huge rewrites in the way that DoomRL loads mods (currently it's all game-side: TCs will let you customize all the menu options as well).
Episodes are basically a way of chaining levels together to form a group of levels, much in the way that DoomRL plays now (you're placed at Phobos Base Entry, then move to Phobos 2, then Phobos 3, etc). I will explain the specifics when I update the Modding:Module page. To be honest I don't know enough off-hand to explain it here, but I will make that page top priority.
Is it possible to make a mod which is simply DoomRL + added features? For example a new challenge; or smarter enemies; or additional badges; etc.
This is doable but not easy, at least at the moment. If you know how to make an episode then literally recreating DoomRL is real quick: as for the rest, in essence you're just altering hooks and/or prototypes, but doing this is often a pain. I don't think it'd be too hard to write some helper functions that make this more manageable for players. The exception right now is stuff like literally adding a challenge to the challenge list or badges/medals to their respective lists. Since mods don't reload DoomRL in its entirety, these cannot be altered: however, including challenge-like properties to an episode and granting the player badges/medals (which will appear in the mortem) can work.
What are the plans for modding in 0.9.9.6 and beyond?
For 0.9.9.6, not a lot! This is because the graphical version is at the highest priority for Kornel, and he often takes care any and all API. Still, I think something I can do is provide a better ease of use for the modding system (such as the aforementioned helper functions). Since there aren't many changes, too, this gives me the opportunity to actually update the modding docs and tutorials, which (as you pointed out) are pretty badly outdated.
After that, I'm pretty sure 0.9.9.7 should have total conversions ready. I'm not entirely sure other than this, but I'll bet the other developers have some ideas.