Many of these bands almost made history for a different reason -- by producing albums with covers that would have traumatized (or at least confused) a generation.
With an industry full of people getting paid loads of money to make decisions that don't matter, people are going to play it safe. Which is where you get trends like these.
Even if you love their music, often musicians are doing what they're doing only because they realized at some point in the past that making any money was going to require them to radically shift gears.
Most musicians have to put up with record company executives being interfering dicks. Not all bands let them get away with it, though. Every now and then, through luck, opportunism or having balls of brass, some get to fight back.
Every so often a song explodes into the mainstream completely by accident, tearing up the charts to the surprise (and sometimes chagrin) of the musicians responsible.