They called it flyting, and the basic idea was to mock and deride your opponent in the crudest possible fashion ... while rhyming. So, yeah: rap. Only the insults these guys churned out were vile enough to make Eminem shit his entire mansion.
Speaking of shit, the first example of that word being used as a personal insult comes from "The Flyting of Dunbar and Kennedy." The record of a purported verbal duel in 1503, academics describe it as "just over 500 lines of filth." Dunbar and Kennedy (D-Money and K-Hole) exchanged insults like this:
Here's a hint: "Gluntoch" means "dirty knees," and "giltin hips" means "shitty arse." Dunbar's claiming that his rhymes hit Kennedy so hard, the other man shit right onto his own knees.
Translation: You think you're king, but your balls are showing.
The best kind of academics suspect that Scottish slave holders took their penchant for flyting to the New World, where it merged with existing African musical traditions and laid the groundwork for what would eventually become rap. So the next time you're pumping some Biggie, say a little "thank you" to Scotland for contributing something besides bagpipes to music history.
Quit while you're even, guys.