There's only one problem with that: that last one isn't true. To explain why you think diamonds are scarce, let's rewind about a hundred years.
Via Wikimedia Commons
That's one diamond mine. That thing to the left is a town.
For much of human history, diamonds were valuable because they could only be found in a select few places in the world. In the 19th century, however, that actually changed as people started to find massive deposits, most notably in Africa. Those deposits were large enough that the rarity of diamonds was quickly becoming a thing of the past.
"Sir, if you continue dropping these on the ground, I'll have to give you a ticket for littering."