They won't kill you outright. They'll just coordinate to gradually increase the length of your commute 30 seconds more each day.
The eventual takeover of self-driving vehicles is going to outright change the world. There are countless benefits, such as increased efficiency and the end of Fast And Furious movies, but by far the greatest result would be a near-elimination of vehicle-related deaths. Of course, there are some downsides, too, like the near-elimination of vehicle-related deaths.
As it turns out, there are some folks out there who rely on car accidents to get by, and we're not just talking about the douchebags who make those Faces Of Death videos. A huge percentage of organ donations come from accident fatalities, so lowering accident rates will remove hospitals' largest supplier of fresh body parts this side of backyard wrestling. The availability of all vital organ donations is already critically low, with waiting lists unmanageably long, so a big dip in second-hand lungs and livers is going to kick things into crisis mode. That's right, people not dying is going to lead to other people dying.
They won't even be able to rely on the occasional weird driver murdering their passenger anymore.
Not having any organs available is actually a more terrifying future than a robot apocalypse, because it results in a massive demand for the kinds of body parts that people can't quite live without. We've seen black markets pop up for Beanie Babies, so there's no way the theft and distribution of organs won't grow out of control. One possible solution is the legalization of selling one's own organs, like a kidney or a hunk of one's intestine, for cold, hard cash. In summary: Robot cars will kill us by saving us from each other. It's true, despite making even less sense than Terminator: Genisys.