If we hadn't shown up, every one of those medals would have been up for grabs. If all the countries that threatened to boycott had followed through (Sweden, Great Britain, France, Czechoslovakia [ask your parents], and the Netherlands), the Nazis would have had a shot at a whopping 137 additional medals.
Let's say that, theoretically, Germany had won every single one of those boycotted medals. Then the only Olympic Games that featured the Nazis, a self-proclaimed superior race of people, would be in the Guinness Book of World Records today as second on the list of countries to win the most Olympic medals in a single games. Hitler would have been able to use that utter domination to his advantage, and the Nazis would be remembered as, at the very least, being quite good at sports. But because we showed up, the narrative was about the Nazis losing to a black man. Jesse Owens showing up in Berlin and kicking some Aryan ass is a lot better than Jesse Owens sitting at home being oppressed in his own country and never making the history books.
The New Yorker
Here's a "Jewish stereotype" beating the blond Germans. Because we were the tolerant ones.