In reality, this man would barely be past the beer bong stage of his life.
Yes, it's just TV fantasy, and yes, our entertainment will always involve action heroes who are impossibly strong or wise guys who are impossibly witty. What bugs us about this is the implication that if you are born with these supernatural smart people genes, time and effort become meaningless concepts.
This lets us to drag out our favorite and most often-quoted statistic: acquiring mastery in just one field takes approximately 10,000 hours, or ten years... even if you're really smart. Some shows get around this by simply saying the character has eidetic (or a "photographic") memory, but that's a skill so rare that it's doubtful that it even exists, and there is no recorded example of a single human using the ability to become a genius in every subject they touch.
Muscle memory matters less than your ability to hallucinate in shades of Math.
So it's annoying because these characters are kind of an insult to everyone who's ever actually become a genius in a field through 30 years of laborious study. They create the impression that a talented person can soak up perfect knowledge of ancient Chinese literature in six months or so, allowing them to move right on to molecular physics or Jiu Jitsu. No big deal -- in the shows they honestly don't seem to work any harder than the average Joe.