Great Literature and literary fiction are not necessarily the same thing. Shakespeare's work, while certainly great literature, was genre material produced for general consumption. Both general and literary fiction can find its way into canon; what distinguishes literary fiction is not so much quality or endurance, but idiosyncrasy.
General fiction relies on convention: structured plot, near-journalistic (or alternately, florid) prose, easily recognized character relationships, and typically a third-person omniscient or roving third-person limited perspective. Literary fiction may discard an advancing plot (The Sound and the Fury), specific characters (If on a Winter's Night a Traveler) or even spelling and syntax (Finnegan's Wake), operating by its own rules. It requires more effort of interpretation from the reader, but it can convey thought-structures beyond the means of conventional storytelling.
Most other people don't have a definition of literary fiction, but just a vague sense that it's something like a classic, or that it uses lots of big words, or that it's anything that doesn't fit another category. Do you object to either my description of the qualities of general fiction, my description of the qualities of literary fiction, or the distinctions I've drawn between the two? If so, on what grounds?
I've read a considerable amount of literary fiction, classics, general fiction, and genre fiction, as well as having been exposed to a great deal of fiction aspiring to be literary and received instruction on the qualities of literary fiction, in the course of pursuing a Bachelor's Degree of Fine Arts in the English Language. My opinion is certainly not authoritative, but being conversant in all the sorts of literature under discussion and having intimate knowledge of literary fiction, those who read it, and those who produce it, I'm confident those last two groups of people, at least, would assent to most or all of my description.
"Literary fiction may discard an advancing plot..., specific characters..."
So may genre fiction. A great example would be any of the myriad stories that take place within dreams, and more specifically, stories that leave you wondering if what you just read was a psychological event; a spiritual event; or a material world event based on the science of many worlds, parallel universes, and hidden dimensions. You'll find most of these in the fantasy and science fiction categories.
"...or even spelling and syntax... operating by its own rules."
Many genre specific stories do the same, or would anyone really not consider Lord of the Rings to be fantasy? And Tolkien was not alone when it came to making-up his own language.
"It requires more effort of interpretation from the reader."
This is generally only true because these authors do their very best to use the most obscure and uncommon words they can find. They dig up old lost and forgotten words more often than not, and do so for all the wrong reasons. Why would I use a word like "threadbare" in this day and age when almost no one under thirty would know what it meant? when I can be more effective by speaking in modern language and say "wearing thin" or "getting old" etc. Most authors that get pegged with the LF tag are simply poor communicators who come off sounding like a 12-year old trying to impress their English teacher and failing miserably at it. The one and only reason to use uncommon and obscure words is if they are the only ones available to convey the meaning you need. I fully expect to come upon obscurity when reading something like Chris Langan's Cognitive Theoretical Model of the Universe where he had to actually invent terms where none existed to convey his thoughts. The same would have been true during the fifties when David Bohm was inventing new concepts like nonlocality and quantum potential. I also expect to see uncommon terms when reading good sci-fi. (I don't think I ever read an Author C. Clarke novel without learning some new ones.) This simply isn't the case with hardly any literary fiction I know of. Their obscurity is generally just bad writing.
"...but it can convey thought-structures beyond the means of conventional storytelling."
Name one that I can't find the equivalent of in a genre category.
"Most other people don't have a definition of literary fiction, but just a vague sense...."
That's because it doesn't have one except that of being very focussed on words even at the expense of the story. LF is by its very nature incredibly subjective, and most people realize that.
Where most LF authors miss the boat is that they tend to concern themselves more with these uncommon and obscure words as a way of making themselves appear smarter than they really are. But this isn't what happens. Most readers will simply be irritated because they realize they're reading an author who has a thesaurus open in front of them. They come off sounding more like precocious children than adults. What they should be concerned with is learning to turn a phrase. That, along with having good stories, is exactly what makes a great author a great author. Few were better at it than Mark Twain and GK Chesterton. Consider Twain's describing a child's birthday party as a "pleasant turmoil." No uncommon or obscure words there. Good authors don't need them. Take any number of quotes from Chesterton:
"The true soldier fights not because he hates what is in front of him, but because he loves what is behind him."
"...feminism is mixed up with a muddled idea that women are free when they serve their employers but slaves when they help their husbands."
"It is absurd for the Evolutionist to complain that it is unthinkable for an admittedly unthinkable God to make everything out of nothing, and then pretend that it is more thinkable that nothing should turn itself into everything."
"Impartiality is a pompous name for indifference which is an elegant name for ignorance."
Or how about Ray Chandler:
"To say goodbye is to die a little."
Uncommon and obscure words are almost always an excuse for a lack of talent, and this is exactly why LF doesn't sell. Not because people don't understand it, but because they do.
"I've read a considerable amount of literary fiction, classics, general fiction, and genre fiction, as well as having been exposed to a great deal of fiction aspiring to be literary and received instruction on the qualities of literary fiction, in the course of pursuing a Bachelor's Degree of Fine Arts in the English Language."
Hmm—well:
"Without education, we are in a horrible and deadly danger of taking educated people seriously." ~ GK Chesterton
