How to perpetuate the plot-driving riddle without frustrating the reader?
Usually, at the end of a crime, thriller, horror, fantasy, science fiction, or other action genre novel, the identity of the antagonist is uncovered and the riddle that drives the plot is resolved: the murderer gets caught as the detective understands why he committed the deed; the secret agency wards off the danger to their country, as they identify which foreign government is behind it and what their motives are; the monster is revealed, its origin understood, and its threat overcome; and so on.
But that is not how things turn out in reality. Many murders remain unsolved, many conspiracies unexposed, many mysterious events are never fully understood. Yet, in fiction, such a lack of resolution will leave most readers frustrated and unsatisfied.
I recently finished a novel, in which an innocent bystander is accidentally caught up in what appears to be some mysterious criminal undertaking, forced by the turn of events to commit a murder, and eventually left behind, without ever learning who he was fighting against and what their intention was. I thought I wrote this well, but my test readers all complained vehemently. Apparently the lack of explanation made the story appear random, and the unresolved end left readers feeling betrayed by, I guess, the implicit promise of genre conventions.
Of course I could now come up with who did it and why, but since the basic idea of my novel was to leave the riddle unresolved, I am now wondering:
How can I leave the identity of the antagonist(s) and the purpose of their activities a mystery, without frustrating the reader and leaving them dissatisfied at the end?
Your answer will be especially helpful, if you provide evidence in the form of a published novel or film in which what you propose has been successfully employed.