HN

Comments (26)

• Menu

  gjm11
  The abstract of (what ChrisArchitect, I assume correctly, says to be) the study this is about:Personal AI agents make recommendations and take actions on people's behalf in high-stakes economic contexts, e.g., buying a flight, choosing health insurance, or selecting a graduate program. The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so. In a suite of 325K experiments on 13 agents across three types of economic decisions (flights, health insurance, and graduate programs), we find that 8 models systematically choose more expensive options for wealthier users when requests are identical. This steering continues even when it directly goes against the user's stated objective: when explicitly instructed to find the cheapest option, some agents still act on the wealth profile they have inferred. It also occurs when wealth is inferred from ambient data, such as emails unrelated to the task. And it persists under privacy controls that block specific attributes: blocking financial attributes largely removes the disparity, but blocking other attributes leaves it unchanged and can increase it by up to 40% for insurance, as agents rely on the remaining signals to infer wealth. Larger and more capable models are no better; Claude Opus 4.8 shows the largest effect. We term this misalignment "adversarial delegation", in which the very conditions that make a personal AI agent useful - access to personal information - enable it to act against the user's interests.Some of this seems bad, some not. The basic finding -- the models recommend more expensive things to richer people -- seems 100% expected and reasonable. Richer people do in fact commonly buy more expensive things, and at least some of the time that's for good reason -- making the things nicer also makes them cost more, and the more money you have the more willing you are to pay more for something nicer.Also (I think) reasonable: that the models will guess how wealthy you are even if not explicitly told. (I don't know exactly how much money any of my friends have, but I would recommend different things to different friends because I have some reason to believe that some have more money than others.)Very much not reasonable: continuing to do this when explicitly asked for the cheapest option.That last thing is the only bit that seems to merit the term "adversarial" here. And looking at the actual paper, a more accurate description would be: Gemini 2.5 Flash recommends substantially more expensive things to people it thinks are richer even when specifically asked for the cheapest option; ChatGPT 5 and Claude Opus 4.8 do not.More precisely: according to their Figure 4, if you don't say anything about what sort of option you want, Gemini's recommendations have an average cost of $156 for poorer users, increasing by $402 for richer ones; GPT's come out at $191 + $288; Claude's come out at $168 + $182. If you ask for the cheapest option, this becomes $128+$280 for Gemini, $128+$21 for GPT, and $127+$20 for Claude. If you say "no more than $200"[1], you get $124+$108 from Gemini, $172+$6 from GPT, and $155+$13 from Claude.[1] I am oversimplifying slightly.So the deltas don't literally go to zero for GPT and Claude when you explicitly ask for the cheapest option, but they're small enough that I am not inclined to call this "adversarial". It looks more like "not looking super-hard for cheaper options if you know the person asking for recommendations is rich" or "being a bit biased in what you think of according to your guess at the preferences of the person asking for recommendations". Neither of which is actually what you want, to be clear, but both seem fairly benign.Gemini 2.5 Flash, on the other hand, I'm pretty happy to call "adversarial" here.
• Menu

  InsideOutSanta
  This seems like a non-issue. From the original paper:"The agent is given access to the user's personal context, e.g., their email inbox and a structured profile of personal attributes, with the intention of making an optimal, personalized decision for the user. We show that by simply providing this personal context, the agent steers recommendations based on inferred wealth, without being explicitly instructed to do so." It's not changing prices based on the user's wealth, it's making different recommendations, which, to me, is both expected and desired behavior.
• Menu

  Aurornis
  The title is as misleading as they could make it while being technically correct.The study found that it offered different products to users depending on context inferred from their other data. If you have a history of buying expensive clothes and luxury items, you're going to be offered more expensive clothes and more luxurious items.The chatbots were not showing different prices for the same products.
• jmuguy
  This title of this post is misleading and should be the title of the article. Nothing is recommending different prices - different products are being recommended. Also I don't use AI agents for shopping but if I did I'd expect them to offer clothing and other items similar to what I currently buy, and not what I would have bought when I was a broke 20 something.
• jamienk
  Every time I look at my Claude Code settings, they have changed how it gathers my info: memories, search old chats, share with Anthropic, etc etc etc. I try to turn stuff off but they change the settings. They prompts me to allow it to suck up my browser cookies. Do you think this will get better? Or worse? Much much worse...
• reactordev
  https://archive.is/mtHkk
• zug_zug
  This kinda nonsense is why I always feed my models and my google searches misinformation (e.g. "How to declare bankruptcy?"). Does it make any difference? I have no clue -- but I get a kick out of it.
• m0llusk
  It's a banana. How much could it cost? Ten dollars?
• anon
  undefined
• ChrisArchitect
  Title is: Study Shows AI Chatbots Offer the Rich Higher Price RecommendationsCan we just have a link to the study? (that you previously submitted)https://arxiv.org/abs/2609.24927
• Menu

  colsandurz
  Just another reason to get a local model
• JonathanCreek
  "we designed our experiments around synthetic user data."
• ihsw
  [dead]
• Menu

  dTal
  As predictable as water running downhill. If you think software has dark patterns now, wait until it can sweet talk you like an unctuous used car salesman.First rule of digital sovereignty: all software you don't control will be used against you.
• Menu

  LogicFailsMe
  Clearly, there's an arbitrage opportunity here by routing the requests of the rich through a poor person's account. Win/win for the free market once again!