On the Nazi Bullshit Plausibility Scale, which ranges from "could actually happen on planet Earth" to "not even in an Indiana Jones movie," this one at least makes some sense. Yeah, it's a mostly uninhabitable wasteland, but if you're the kind of asshole who wants to eat the whole pie before anyone else gets a piece, you don't leave the crumbs behind.
Plus, in theory, it could be a useful staging location for operations in the southern hemisphere (Added bonus: strategic, militarized penguins sound like an excellent use of Nazi resources). According to legend, the Germans built a base so strong that, after the fall of Berlin, they were able to retreat there and fend off attacks from the British and American armies for years. In fact, they purportedly hung on until the 1950s, when they were blown up with tactical nukes.
And we never had to worry about evil ever again.
There's even a nugget of truth to the stories. The Nazis did stage a few expeditions to Antarctica, but so did the U.S., and the only thing we're hiding down there is the answer to who the alien was at the end of The Thing. It was ever so briefly hypothesized that our frozenest continent might really be useful in some way to either side, but common sense prevailed when they realized how stupidly cold it gets down there.