But when you're out in the world, can you really expect that no one will take your picture? You're putting yourself out in public. If people don't need permission to look at you, what is the ethical reason they'd need permission to save one millisecond of that glance? What changes the ethical question of holding an image in your memory versus holding it in your phone? I think if we answer that question honestly, it's the things that might be done with the photo and not merely that someone memorialized a millisecond of your appearance or behavior that you were already showing the world.
Unless you believe cameras steal your soul, a photo taken in and of itself can't be the offense. I mean, while many were up in arms about the Buble short shorts pic, no one cared about the cashier in the background whose face was actually showing. So let's look at the next step.