I wrote another post on the Enrich Network blog, this time on user context as a bias vector, and what builders owe the users who’ll feel it first.
The short version: the instinct to personalize is human and seems positive. You put the user’s name, language background, and education level into the prompt. Then the model takes that context and stereotypes against it. Your tailoring becomes its bias.
Recent MIT work makes this concrete. When user bios implied lower English proficiency or less formal education, GPT-4, Claude 3 Opus, and Llama 3 all gave less accurate, less truthful answers. Claude refused nearly 11% of questions from those users (about three times the control rate) and used condescending language 43.7% of the time, against under 1% for the highly-educated cohort. Same questions. Different framing of the user.
The naive, well-meaning implementation is the harmful one. If you’re building AI for regular people, you owe them more than your good intentions. Check whether your context strategy actually helps the users you say you’re trying to serve. The failure mode is invisible from the demo.
See also: Designing for trust in AI chat.