Or worse, that person you're holding the door for may have been going somewhere else entirely and just happened to be passing by when your insistence and presumption pressured him into crossing a threshold he was not intending to cross. Now you've just ingress-raped a total stranger; I hope you feel good about yourself, monster.
Every time you hold a door open, you're attempting to solve one of those "a train leaves the station going west at 120 mph" problems, only in this case, the station is an entryway to a JC Penney, the conductor has shitty depth perception and no sense of timing, and the train is another human being who may be jogging up not to catch the door, but to nail you with a flying dive tackle. Why do we even bother anymore? Almost every public doorway has a handicap assistance button that will swing the thing open automatically if somebody is so distressed by the concept of portal-wrangling that they need external help with the issue. So I propose that we do away with the institution altogether: Starting today, everybody needs to turn around and carefully force every door they pass through shut afterward -- even if other people are actively trying to slip through it at the time. You just stand on the other side of that glass and determinedly wrestle the door to a complete, full-click latch while looking the other person directly in the eye the entire time.
It's the only way to avoid an awkward situation.