Couldn't agree more.
The academic debate of which is the better car totally ignores the fact that these are not purchased to be "street raced" and that tenths of a second in acceleration spreadsheets do not a good car make. Sorry, but that's for kids in Acura Integras. Naturally, there's some intangible bragging/marketing rights to having the quicker car. Ultimately though, it comes down to personal preferance of different brands of style, performance, feel and practical need or financial options.
The RS4, no matter how you slice it, is not as track-balanced as an M3 or 911, awesome as it is. If you like your track days and have a nice tire budget the other two are probably more fun. But if you live in the snowbelt, have a kid in a child-seat and enjoy keeping a slightly lower profile in the neighborhood, well then the RS4 choice is easy.
Personally (and this is just me), I'd opt for the M3 if I were in the market. You can't beat the combo of track-ready spec with the practical packaging for everyday driving.