Since "villains" came from the poorer parts of society, and since the word was mainly used by people on the opposite end, it soon came to refer to anyone who was "base or lowborn." In other words, criminals. Fair or not, it's much easier to blame crime on the servants or other commoners than on people who bathe regularly. Still, the fact that the word gained a negative connotation says more about the upper class people using it than the "villains" themselves.
At some point "villain" came to mean "mean country fellow," which apparently means that people in the 14th century had no word for a country fellow who wasn't mean, since they couldn't conceive of such a thing being possible. The word "villain" just simmered for hundreds of years in a stew of elitism and prejudice, and as a result, today we're using it to define the likes of the Joker, Voldemort, and Darth Vader.
And Chad Laptoprage.
But it all started with some innocent farmhands. So if your gardeners ever tie you to the tracks and cackle maniacally, you can have a good laugh at the irony. Considering that we've been using their office as an insult for centuries, who can blame them?