Eating one you've found to be already even a little bit open (and therefore dead) is pretty much booking a two-night stay at the nearest bathroom, where you'll have plenty of time to wonder why it was you thought that it'd be a good idea to eat a rock.
So you can see why the abundance of oysters in the 19th century led to their being mostly eaten by the working poor in the U.S. and the U.K. (also, they're not very nutritious). Dickens even mentions them derisively in The Pickwick Papers, making them certified riff-raff food even in a literary universe where people are willing to start shit over gruel.
Then How Did It Get So Fancy?
The industrial age brought a population boom and many, many oysters were eaten. Coupled with the kind of pollution you normally only see in dystopian 80s movies about 1999, many of the oysters were killed off, driving up price and demand, and therefore catching the interest of rich people.
To fix this problem, foreign oysters were brought in to replenish the population. Unfortunately, 19th century medical science was still in its Flintstone's car stage, and nobody thought to point out that the local and foreign oysters might carry diseases that the other might not be immune to, leading to tons of oysters eventually becoming rocks for real.
Thus oysters were made permanently scarce and pricey, and as Beanie Babies proved, rich people will spend money on anything if doing so means that you can't have it.