The first farmers soon found out that although agriculture did provide food, manual labor was far more grueling than the relatively bohemian lifestyle hunter-gatherers enjoyed back when food was abundant. This showed in their build: Compared to the big, meat-fed hunter types, agricultural people were a small and bony folk. It wasn't just because of all the hard work, either: Early farmed food was the kind of muck Taco Bell would hesitate to offer its customers, since early herders had no goddamned idea what they were doing in terms of breeding. This, combined with the fact that livestock lived practically under the same roof as their human owners, led to a number of animal diseases becoming more prevalent and figuring out how to jump from animals to people.
"Uh, STDs too ..."
The food the farmers were growing wasn't much better: The sugary grains agricultural societies fed on started decaying their teeth.
Incidentally, the dawn of farming also messed up our relationship with our fellow man: It marked the start of social inequality. Tribal hunter-gatherers had to work together in order to obtain food, so they were all more or less equal. This egalitarian attitude went right down the toilet the second one farmer had enough surplus crop to hire others to do the bullshit manual labor for him. As this trend continued and societies evolved into larger and larger groups, these boss/subordinate roles escalated to the point where we suddenly had kings and slaves.