First, I'm not denying anybody their choice, I'm saying what annoys me. People have a perfect right to do things that annoy me. Voting Republican annoys me, but people do it all the time.

Second, I specified "a specific known person." Different thing from an unspecified "a person" or "anybody." The mind, or at least my mind, doesn't process them the same way.

Third, neither of these types of usages goes back a long time in English. A book called What's Your Pronoun? by Dennis Baron attempted to prove that singular "they" had a long history in English, but all its premodern examples were for ambiguous usages like "everyone" or "no one." Which is yet a third type of usage.