I think the distinction is made for the same reason a bunch of people don't like "Disabled Person": People don't like being defined by their disability, so suggust "Person with Disability" instead so they're a person first. Maybe similar concept?
I had always thought that came about because Disabled Person sounds like someone who Can't Do, whereas Person with Disability is someone who Can Do Except For One Certain Thing.
Edit: I guess that is basically the same thing that you're saying.
I'd probably avoid any use of the word "color" to describe a black person. Anything other than "black person" sounds strange to me when I say it.
"Which barista are referring to?"
"The person of color."
See, it's legit but who says that.