HN

Comments (124)

• Menu

  joshkel
  I'm not clear on what the claim is here.The article faults LLMs for not being more creative (riding in the city in the rain, in East Asia, in outer space, etc.). But, if prompted, "Generate an SVG of a pelican riding a bicycle," with no additional details or specification, then I would _expect_ something generic: a side view, nondescript terrain or perhaps terrain associated with a pelican (i.e., beach), little clothing (because birds don't wear clothes) with the possible exception of something strongly associated with bicyclists (a helmet).The article talks about "the LLM diversity collapse problem," which makes it sound like LLMs are getting less diverse over time, but the linked benchmarks suggest that newer, more capable models are _more_ creative.Out of curiosity, I tried a couple of prompt variations in Opus 5.5. "Be creative" did a beach scene but added fish to the bike basket and a lighthouse in the background - not too creative. "Pick a unique and creative variation of this concept" resulted in a "midnight circus act" of a pelican riding a penny-farthing on a tightrope while balancing a fishing pole with fish (and, this time, two lighthouses - the tightrope is implausibly strung between them).
• Menu

  orbital-decay
  The lack of variety is probably the most underrated issue, and the most likely cause of annoying patterns you all hate. All current post-trained LLMs are mode-collapsed compared to the post-training set distribution, and this doesn't get nearly enough attention from the labs and independent researchers. It's far from being a transformer-only problem, it's a common failure mode in all kinds of models. It's not a fundamental limitation either, unsupervised models have no problem with being as diverse as the training set allows.
• Menu

  jvanderbot
  It is fairly easy to go ask for a differently styled pelican, environment, background, etc. But that's not the point.I was thinking just today about how, if I could hire my competition's lead designer/programmers, have them design a system for me cleanroom, that I'd effectively know everything there is to know about my competition's limitations. This is trivial to do with LLMs."Make me a thing that does X" is going to come out implemented basically the same way most of the time, as a kind of stack-overflow, well commented, internship project. (or, if undirected enough, as a collection of those based on a bog-standard architecture)Same-y enough that you could train an adversary on it? Or hack it afterward? I am beginning to think so!
• Menu

  ed
  A good prompt will shift the distribution away from average eg: https://john.hartnup.uk/2026/06/07/ai-event-posters.htmlThis is a feature, not a bug.
• Menu

  randomblock1
  The example just doesn't support the claim.If I ask for a pelican riding a bicycle, I am expecting a relatively boring image of a pelican riding a bicycle, not a pelican flying through a thunderstorm while on a bicycle. That's just straight up not what I asked for. I didn't ask it to be creative, when I imagine myself riding a bicycle, I'm not flying! If I added "be creative and unique, you can add or change anything as long as there is still a pelican on a bicycle", then sure.Suppose I want an SVG of a laptop. It should probably look like a Macbook Air or something, neutral colors. Easily recognizable as a laptop. If it chose to be creative (unprompted) and gave me a futuristic or retro laptop, or one that's super wacky, there's a high chance of it looking bad, and a high chance I say "no that's not what i wanted". So during the training step, it probably learns to just follow instructions and be less creative unless asked.Sounds to me like LLMs are just getting better at following instructions and doing what they're told. Simple prompt, simple output.
• Menu

  spuz
  And funnily even in the diverse set, most of the images show the pelican riding from left-to-right. There's a ton of bias in our data and even in our brains that you would need to erase to get true diversity.
• ozgung
  > This is the LLM diversity collapse problem no one is talking aboutWe’ve been trying to tell this on every pelican thread. All models were recreating the same composition. The sun, the road, the green, the scarf, going right. Even the style. They’re not obvious associations one would make given the prompt.
• Menu

  aabajian
  More importantly, why is photorealism not implied: https://www.reddit.com/r/GeminiAI/comments/1x54q78/argons_sv...Why does an SVG pelican imply that it should look polygonal?I ran into this issue recently when asking Claude to design a logo for my van. It made a polygonal SVG. I asked it to make it more photorealistic and it looked way better, still an SVG though.
• Menu

  electroweak
  Would any teacher of, say, grade 4 students, reading this ask them to draw the same prompt and let us know the result? It would be so interesting to see how we differ from the machines, if at all.
• tetraodonpuffer
  is this a case where alignment/rlhf collapses diversity same as image models going from "base" to "turbo" does (where the base model is typically way more influenced by seed, while distilled/turbo models aren't as much)? I am also wondering if part of this is also prompting, maybe we should have a second simonw benchmark that appends "be original" to the prompt and see if all models try to be original the same way too.After all if you ask 100 people to draw a horse, it is quite likely it will be a lot more similar to a "here's a stock horse photograph" than a "here's what picasso would draw when asked to draw a horse"
• Menu

  fenderog
  Related, I have been working on evaluating creative output of different models here https://fenderog.com/art-crit/.
• Menu

  vatsachak
  Thanks for pointing this out.The reason why LLMs cannot be purely* pre-trained on LLM outputs is because LLM output diversity is low.This implies two things:1) There is going be no functional difference between hypothetical Qwen 4.5 and Fable 5.52) LLMs are not AGI
• armanj
  This part of a recent talk points out exactly this problem, and (likely) why human expertise will remain important in the age of LLMs doing everything: https://youtu.be/X7vf7koPxKg?t=2806
• Menu

  seidleroni
  I think this may be close to what Ethan Mollick has discussed regarding the AI's having a specific "taste". This is how they draw pelicans but obviously if you asked them to make different versions that are more creative, they could do that. For now, this is one way humans can provide a lot of input and direction.https://www.oneusefulthing.org/p/the-overhang
• xtracto
  Tangential to this, but I've found asking models to generate Origami instructions way more interesting:"Write an svg with graphical instructions on how to make a pelican riding a bicycle in Origami using any number of A4 paper sheets."All models struggle a lot. Most current models all get a "basic airplane that really floats" pretty well.
• phil-martin
  Oh this took me far too long to realise - this is a clever ad for their paid AI design product https://kombai.com/features/design/
• ks2048
  Interesting thing I notice: in 10/13 of the examples, there are 3 small horizontal lines behind the bike meant to convey motion.These look surprisingly similar and not necessarily something that directly follows from the prompt.
• wolttam
  I disagree with this framing entirely.It's unsurprising and perhaps even desirable that different LLMs return similar results to an identical prompt.If you want variation, prompt for it.
• jeanpah
  I think this behaviour Will kill the leftovers of innovation we still have. If if happens with the pelican, it Will happen to everything else.
• Menu

  riebschlager
  Why would I want an LLM to be creative? I want to use these tools to express my creativity, not its own.If I'm asking simply and directly for a pelican riding a bicycle, then please honor my intent and gimme that pelican. I don't need it on the moon. I do not need a top hat. I just want what I asked for.Fulfilling a simple request simply seems like a feature, not a bug.
• dzonga
  all A.I will do as it expands the 'work' it touches - is to expose those with taste vs those without taste.tho who know to refine, say NO. & do more with less even in the age of too cheap to meter tokens.that has always been the work that produced every delightful product.
• SirMaster
  The real question is why are the spokes always wrong?
• tonyedgecombe
  Perhaps all the designers will end up telling us programmers “just learn to draw”.
• tamimio
  I would be curious to see if you would ask few designers to draw the pelicans with the same prompt to them, no further instructions given, and they should not know about the pelican AI benchmark thing, how that will turn out?!
• Menu

  moffkalast
  They say the pelican isn't benchmaxxed, but at this point how can it possibly not be?
• senko
  Models do seem to have a preferred style, see for example what Opus likes to do: https://s3.us-east-1.amazonaws.com/vault.senko.net/blog/quic...The pelican from the post can be changed by prompting, as other commenters show. But I found it pretty hard to guide Opus away from the above.Other models' results look different, but also show style convergence.You can also see in "creative" game output (Astra loves 3d), or text output.It's not all bad - this shows we should treat AI as a tool and assume any creativity (or lack of it) as coming from the human.This nicely redefines slop as work you slapped together without much care, irrespective of what the tool was.
• mochacosine1206
  Because they use the same prompt?
• minimaxir
  As someone who has spent an inordinate time researching prompt engineering technique for producing more interesting images, it's a good thing the base case with an underspecified prompt is boring as it gives the model more malleability for more interesting prompts.One of the reasons image AI slop has been so prevalent is that the slop is the base case (as derived from tons of RLHF because the median user has no taste) and it is becoming increasingly difficult to steer those models even when the prompt is more specified.
• azrazalea_debt
  I mean in a lot of cases it seems like the leadership people see this as a win not a lose.Lost count of the number of times a requirement is given in terms of how another product is doing it. "make chips like gmail uses for tags", "let's copy anthropics LLM chat UI", "we should try and make it look like X competitor"."Looking the same as everyone else" seems to be a bonus in their mind
• Menu

  0xbadcafebee
  They're asking why a predictive text model with a fixed set of data, all of which are trained on almost the same open datasets, isn't being more random in its output?People weren't happy with boring answers, so they added randomness to the output. The randomness led to hallucination, so they made it less random. Now you want it to give you a unique artistic impression for every prompt? It's a spell checker, not an art student.
• pstuart
  Maybe its just the simplest image to make and each model follows that path because they too find it the simplest.Perhaps the prompt should include what the scene should contain.
• hn_throwaway_99
  This is inherently an optimization problem, and it affects a lot more than LLMs. With so much of the world being instantly connected, things converge to the "best" style at the loss of diversity.For example, a lot has been written about how all corporate logos look the same now, https://news.ycombinator.com/item?id=32040506 , and also about how so much interior design and architecture has converged to the same look. In the 70s there may have been tons of styles people derided as ugly, but at least there was colorful diversity. Now so many higher end (and "aspirational high end") interior design has the same minimalist, gray/white/tan, "look you can't see my refrigerator or dishwasher" look. I was recently looking for an apartment and all of the "luxury" apartments I looked at (obviously take that adjective with a heap of salt) had nearly the exact same layout and style.
• Menu

  micromacrofoot
  People aren't that creative either, ask someone "imagine a pelican on a bicycle" and then ask them "where are they" and you're going to either get the generic beach locale, or something familiar to the responder (i.e., where they live or where they ride a bike).You'd probably need to ask thousands of people (or some children) before you heard "a pelican on the moon" or "a pelican in neo-tokyo"
• aaroninsf
  Hot take: the problem here is not the one implied or AFAICT assumed by the author.The problem is not that LLM or, say, generative AI tools such as Midjourney, "have no imagination" and collapse upon the same solution. Anyone who has spent any serious time exploring what was originally called the "latent space" Midjourney provided a view into, is aware that the combinatorics around what these systems have ingested is inconceivably large and full of wonders and surprises... what we might well describe as unexpected, novel, dare I say creative, imagery [etc.]The problem AFAICT is asking for examples of something without providing any process or instruction.Vibe-coded single-shot prompts are inevitably (unless this is patched) going to converge on statistically-likely or contextually hinted examplars.And this is true all the way down. It's true of secondary and tertiary features (what camera angle do we have on the pelican? Is it a juvenile or injured? Is it riding to the left or right? etc.)The fact that humans are a heterogeneous pool with different experiences and capabilities, means if you sample humans, you get high variability. If you ask contemporary LLM, they are converging, indeed, but not in the way implied; they are converging because the corpus they are trained on is comprehensive and coextensive.What you get is the centroids in the space.Until you simply ask for something else.If you want more "creative" examples, of pelicans or anything else, it's more or less trivial to get them. And if you detect durable secondary or tertiary qualities you want to see varied, you ask.More seriously,if what you want is to interact with large capable models/systems which by default exhibit variability , you do the (minimal) work to make that happen.Midjourney has from very early on exposed explicit dials to turn in exactly this way.I am surprised the author does not know this.
• lofaszvanitt
  I haven't read the paper yet but I think the keyword here is SVG. Which is a 2D format, usually used for icons. So that already defines how it would look like in the end.
• dvl82
  Now draw me an elephant in the room
• jdw64
  But is Pelican Bench really that helpful? I'm not sure. Why is this even trending? It seems like a kind of meta joke — is there anyone who can explain in what sense it became a kind of community meme? I understand it became famous because a well-known person named Simon tried it. But I feel like there should be more coding benchmarks, honestly...
• sleepysloppy7
  [dead]
• Menu

  user43928
  So, how were the more creative pelicans created? I assume by changing the prompt from the default.I don't see why you couldn't do the same in real projects.If the design looks too generic, you can just ask for a change, no?