The reflection of the bright space behind the viewer overpowers the image of the dim space on the other side.

This is how it’s not just the “dark hole” effect you’re referring to. It’s a dark hole with a bright image superimposed over it. Your eyes can’t adapt to the dark hole because of the reflected image.