Ian Fleming wasn't great at writing characters who weren't James Bond, James Bond's bosses, and the guys who wanted to kill James Bond. And he especially wasn't great with people who apparently don't talk like Bond does. In You Only Live Twice, the novel in which Bond dyes his skin, cuts his hair, and dresses to resemble a Japanese person, the results are mixed. You can tell that Fleming definitely had an appreciation for Japanese culture and liked the beauty of the region that he set the novel in, but everything he describes sounds far goofier than he intended. In Live And Let Die, Bond travels to Harlem, and the journey is excruciating.
From the outset, you definitely get the feeling that Fleming is trying to compensate for something. Every sentence in the first half of Live And Let Die may as well end with "... but I'm not racist." For example, if Fleming was to write a piece of dialogue like "They won't even let a white man in there ..." it would end with "... but they do enjoy jazz." Fleming stumbles over himself to make sure that he is as ignorantly nice about things as possible. At one point, a black woman is driving a car, and Bond feels the need to remark about how amazing it is that a black woman is driving a car, and driving it well.
"'Smashing,' said James, as he was able to sleep with a black woman, and not get Sickle Cell of the Ebola."
It would be different if this kind of thing didn't happen repeatedly. When he's not out narrowly avoiding death, Bond spends most of his time in the company of rich white people in '50s/'60s England. He's not worldly so much as he just has access to a lot of train tickets. One would logically expect him to go to Harlem and be taken aback somewhat. But he's never this grating in any other book, or when dealing with any other situation.