#2. The Child Actors Are Growing Up Too Fast
There are a ton of child characters in Game of Thrones who, according to George R.R. Martin, were exceptionally difficult to cast because of how much bullshit those characters are going to be subjected to over the course of the series. In the first season, everything was fine, but as we already mentioned, each subsequent season has taken a year to produce, which basically means we're watching these kids age dramatically over the course of what is supposed to be a few months of story. Isaac Hempstead-Wright, who plays Bran, a character who is 7 years old in the novels, is currently deep in the trenches of puberty, making him an ever-growing weight on Hodor's tired shoulders.
The inevitable "Winterfell Prom" episode will be awkward as shit.
Bran can't walk, so they can hide his monster legs underneath a blanket and pretend that his voice isn't getting freakishly deeper each episode; otherwise, they'd have to send out casting calls for sickly bowl-cut British children every single year. The character of Arya Stark is facing a similar problem, being that she is a 14-year-old girl played by a 17-year-old actress. The show's producers have a very short window of time before they're forced to either recast her or make some bold costume and makeup decisions. But it's not like they're going to have to stop the show for an extended period of time or anything, right? (See: "They're totally going to have to do that exact thing," above.)
#1. The Fans and Actors Might Start Dictating the Plot
While the average Game of Thrones fan has learned to accept heartbreaking death with the curdled stoicism of a Russian grandmother, it would be a completely different beast if a fan favorite character or storyline suddenly disappeared for an entire season. And while Martin can kill Tyrion Lannister any time he wants in the books, HBO would be very, very reluctant to suddenly have to continue the show without Peter Dinklage.
"King Kong ain't got shit on me!"
This is exactly what happened when the producers were faced with adapting Book 4 of the series, which focuses mostly on completely new characters and plot lines while leaving the familiar characters from the first three books on hold until Book 5. The result was the producers deciding to fold the two stories together to focus on the characters and storylines that fans of the show already knew and loved, simplifying both actor contracts and production time constraints while pruning some of the seemingly inconsequential characters and side stories Martin decided to write into his novel for whatever reason.
The thing is, the producers of the show have absolutely no idea how important those seemingly inconsequential characters and storylines are going to be, as they've only been told how the overall story is eventually going to end. Anything between now and then is a total fucking mystery to them. They just made the best decision for their television show without any consideration for how it might affect the story they're adapting (which, as we cannot stress enough, is still being written).
"He's back ... with a vengeance. Season 6, only on HBO."
Basically, eventually there will be two entirely different timelines for Game of Thrones fans -- the timeline in the show, and the one in Martin's novels. That's not a new thing, necessarily -- adaptations diverge from their source material all the time. But here, the source material is still being written. There are (at least) two novels left in the series that haven't come out yet. And if the show's production ends up catching up to Martin before the sixth book comes out, it's likely that they will go completely off the source material and write whatever they want, rather than wait five or 10 more years to complete the eight seasons of television they've already committed to.
That's going to create a huge clusterfuck for the fans (particularly if, for instance, Tyrion and Jaime team up to save the kingdom in the final episode of the television show, whereas in Martin's eventual final book they both get eaten by fucking dragons or something) that could potentially sabotage the reception of Martin's yet-to-be-released novels. That man is in his 60s and is currently in the middle of producing the one definitive work of his lifetime that will be remembered long after he succumbs to elderly fatness. When the time comes for the producers to either wait for him to put out another book or just take control of the Game of Thrones story themselves, they will almost certainly go for the latter, and things between them and Martin will get seriously ugly. Like Red Wedding ugly.