Evidently this was a B-side, so I never heard it back in the long, long ago. This may actually be the first time that I've heard it. Very good, in my humble opinion.
I heard it first as the cover by the Moody Blues (see below). I loved the song, and still do. Probably a character defect, I'm sure that my old hipster friends would still find my appreciation something to criticize. But with music, as with love, the heart wants what the heart wants.
First listen? That's probably it. Racism? Probably not, I loved the Drifters. Just an accident of timing and fate. Let's just say it's one of those things.