You've seen this on the signs of little shops that are trying to be quaint and old-fashioned, usually phrased as "Ye Olde _____ Shoppe":
"People couldn't spell for shit before they invented dictionaries."
So that's the way people talked back then, right? Sentences were full of "thou"s and "thee"s and "ye"s. Go to any Renaissance Fair and you'll find as many "ye"s as leather corsets.
The Y at the beginning of "Ye" isn't actually a Y at all. It's an Old English -- excuse us, Olde English -- letter called thorn:
Oh, and it's pronounced "th." So when you read "Ye Olde Dildo Shoppe," you're really reading "The Old Dildo Shop." Now don't you feel stupid for buying old dildos?
So how did we get duped into putting "ye" in front of our 20th century American English pubs? When the printing press first took off in the 15th century, German typesetters were the ones making the letters, and they didn't have a symbol for thorn because, well, they were German, not English. So the first English printers had to improvise with the letters they were given. This included the letter Y, which in the handwritten form looked a lot like thorn.
"Fuck it, I'm not figuring out how to carve that on a block."