Joker alludes to being cast out like a leper (possibly by the so-called distinguished gentleman), just like Bruce will be, says that he and Batman are both "freaks," and that they are not like "civilized people" who live by a code. Funny thing is that Bruce Wayne almost did break this particular code in the first movie. It wasn't until later that a Dawson's Creek cast member reminded him about his father's morals.
In that very small bit of dialogue, the Joker is basically telling Batman, "Your father is a liar and a hypocrite." And while it's mostly just metaphors being chucked at the audience's heads, it also showcases the disgust that Joker has for a very particular type of person.
Another underlying theme in The Dark Knight is family. Harvey Dent literally calls Rachel "his family." So when Harvey becomes Two-Face, he goes after Commissioner Gordon's family. The cops who betray Gordon? They were trying to protect their families. The only two characters who are "abandoned" and without family? Batman and Joker.
The Joker's extravagant stories for his facial scars lose all bravado if you take into account that they may actually be from a birth defect. Being mentally (and physically) defective would actually explain why he would be "abandoned" by his own father. Like a leper.
Despite all his seemingly mindless violence, the Joker doesn't want to kill Batman. He's trying to protect him, because he sees them both as being loners. He says this multiple times. Maybe, just maybe ... they could be family. And then the very last scene of the film is Batman being chased off. Like a leper.
One of Joker's men in the film is a schizophrenic from Arkham Asylum. He must have broken out in the previous movie, Batman Begins. Where could they have possibly met? Maybe at some kind of Young Anti-Batman Professionals mixer? Or maybe Joker was in Arkham in Batman Begins. What most people don't seem to notice is that Batman Begins is actually the Joker's origin story, too. The Waynes' deaths led to the crumbling of Gotham, the rise of a hero, the freaking breakout of Arkham Asylum, and further escalation. And the last scene is Gordon finding the Joker's playing card. He was in this shit the whole time, probably practicing monologues about scars.
In Dark Knight, when the cops run Joker's DNA, they can't find a criminal record. Something is fishy. Arkham is for the criminally insane, right? So they'd have his DNA if he were ever arrested, that's for sure. Well, not if the Joker was locked in Arkham as a minor and kept there until the breakout. If he was abandoned by his father at Arkham, it must have been at a very young age.