It would be different if this kind of thing didn't happen repeatedly. When he's not out narrowly avoiding death, Bond spends most of his time in the company of rich white people in '50s/'60s England. He's not worldly so much as he just has access to a lot of train tickets. One would logically expect him to go to Harlem and be taken aback somewhat. But he's never this grating in any other book, or when dealing with any other situation.
Again, this continues for 50 percent of Live And Let Die, with Fleming using his characters to make awkward commentary and then doubling back on himself to try to prove that he's not being racist. Would a racist talk negatively about an entire neighborhood of black people, only to say that some of them are not bad? That's the question that Fleming asks you over and over. It's obvious that he really wants to approach the issue of race relations, but has absolutely no idea as to how to do it properly. He says nice things, but they're all obliterated by the fact that the shit that he wrote just before the nice thing was way more awful than the nice thing was nice.
"For someone dumb enough even amongst his own race to lose his hand, he was smart enough to figure out the replacement."
It doesn't help that he often goes out of his way to try and write black peoples' dialogue phonetically, so that you'll get a conversation between two people that looks like:
Bond: "So, you expect me to talk?"
Villain: "No, sir' eee'. Ah do in'a'deed' nah be expectin' dat, Mist' uh' Bond. Ah is expectin' you'se ta mebbe die."
What kind of exotic dialect did Fleming imagine when he wrote it out? Did he write it normally and figure that no one was going to believe him? Was no one going to take him seriously if he didn't take the extra measure to ensure "authenticity" and write every word as if the characters had to struggle to remember what they were saying right in the middle of saying it?
"I do declare tis a pleasa to meet ya, mista ... uh, Band."
It also doesn't help that Fleming has created the antagonist, Mr. Big, and rather than put him in some sort of fantastical or "classy" location like he did with every other criminal mastermind, sets him in Harlem, where apparently people painstakingly separate each syllable when they attempt their native form of speech. I guess it's cool that Fleming decided to make a black man (in what would become a hugely popular series) such an important character in 1954, but he managed to sabotage all of it. The Live And Let Die movie would go on to have the same kind of undertones, but never to the degree of the novel, which makes a shift into the uncomfortable within the first few pages, and then wallows in it.