Why does the villain always win right before the hero defeats him?
One common setup for a story goes like this:
- We have the heroes on one side
- We have the villains on the other side
- The bad guy has an evil plot that will cause some undesirable result
- The heroes know about the plot and are trying to stop it
Very frequently in such stories--in almost all of them, in fact--the villain either actually succeeds in accomplishing their goal or comes within a hair's breadth of it, with the heroes utterly defeated, before some thing happens and the good guys are finally able to improbably snatch victory from the jaws of defeat. A few examples:
Star Wars: The Death Star is literally seconds away from blowing up the rebels' base when Luke scores the decisive shot.
Thor: The Destroyer rampages throughout the town, defeats all of Thor's companions, and blasts Thor almost to death before he suddenly regains his worthiness and manages to turn things around.
Aladdin: The only way to stop Jafar is to literally let him win and become an all-powerful GENIEEEE!!!
Harry Potter: It's only possible to defeat Voldemort after Voldemort kills Harry and the Death Eaters overrun both the Wizarding government and (most of) Hogwarts.
The Order of the Stick: Not finished yet, but the story is clearly approaching the end, and there are currently two different major villains, each with their own distinct plan to effectively ruin the world as we know it. Both are currently only one step away from total victory.
I could go on (and on and on...) but you get the point.
At first glance you'd imagine the purpose of this device is to build tension, but that only works until you've seen it enough times to be able to predict that this is exactly what's going to happen, again. So, I have to ask:
Is there any other reason to use this device in a narrative, beyond "to build tension"? And what good alternatives are there to it (that don't involve the bad guy actually emerging victorious)?