Disclaimer: I don't have a Mac, and I've never installed DoomRL on a mac. I'll try to do me best to help you out though.
I'm going to be basing my instructions off this text from the download page:
Note about the Mac OS X version — coding for this platform is still relatively new for me, so this time the release requires some degree of manual installation. You can only reliably run it from the terminal, and need to manually unpack it to some folder. You’ll also need to install SDL bundles — SDL , SDL_image , and SDL_mixer — if you have any problem installing, feel free to ask at the forums!
You'll first need to download and install the SDL bundles (if you haven't already):
SDLSDL_imageSDL_mixerThese bundles are provided as .dmg files. On Mac OS, you should be able to double click and open these up. They will look like folders with files (and more folders) inside. Each of these packages has a readme file that explains its installation, but I can repeat it here.
To install each of the SDL bundles, you just need to copy the folder called *something*.framework into a folder on your machine called
/Library/Frameworks
I expect the /Library folder will already exist on your machine. If the Frameworks subfolder doesn't exist, you can create it. (There should be an option in one of the Finder's drop-down menus to create a new folder. To rename a file or folder, click on the name in the Finder.) You should be able to copy the *something*.framework folder by drag-and-drop, or by using the copy and paste options in one of the Finder's drop-down menus.
I'll assume you've already found and downloaded the DoomRL package (called doomrl-osx-x64-0996.tar.gz). I believe the default action when double-clicking on a .tar.gz is to extract the contents to the current folder, so do that. This will create a folder named doomrl-osx-x64-0996. It doesn't really matter where this folder is, but you might want to move it to some other folder on your system where you put applications or games. That is all there is to installation of DoomRL itself.
The last point mentioned in the instructions from the download page is that it works better to run DoomRL from the terminal. (You can try double clicking on the doomrl executable in the doomrl-osx-x64-0996 folder if you aren't up to using the terminal, but I can't guarantee it will work.) If you type "Terminal" into the spotlight, it should reveal the terminal application that is already installed on your machine. Run it. This will open a little text-based window you will have to use navigate to the folder with the doomrl executable file.
To execute a command in the terminal, you just type it in and then press the enter or return key. The main command you need is cd (short for "change directory"). Typing the name of a folder after cd will take your terminal into that folder. Other commands that might be useful are pwd ("print working directory") which shows where you are now and ls ("list") which displays a list of the files and directories in your current directory. So, if you moved the DoomRL folder to the desktop, you can just type:
cd ~/Desktop/doomrl-osx-x64-0996
After you are in the doomrl folder, all you have to do is type
doomrl
to run the game.
If something I said doesn't make sense or doesn't work, or if you'd like more elaboration on any point, please don't hesitate to ask!