HN

Comments (132)

• Menu

  snitty
  This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero.Just use the LLM to make a thing that's actually good at what you need it to do.
• Menu

  tasuki
  > I compared to a 10-year old CNN and to my surprise found that this was superior.I'm surprised he's surprised.> A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs.Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).
• Menu

  Legend2440
  > I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU.>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
• Menu

  jonex
  As people have mentioned, it isn't that surprising that a specially trained model beats a general purpose model.However, I think it kinda misses the potential of llms for diagnosis. What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those? What if the images aren't perfect examples and happen to contain out of distribution content like a tattoo?
• Menu

  MitPitt
  And a calculator outperforms LLMs at multiplication and division...
• jacobgorm
  The model architecture might be ten old, but it seems it was trained recently, so the title is a bit misleading.
• Menu

  lukehandcool
  I found SkinGPT-4 when I was looking into this article. From the nature paper "among the 150 cases, a significant percentage of SkinGPT-4’s diagnoses (80.63%) were evaluated as correct or relevant by board-certified dermatologists." Which does outperform the CNN in this article.I remember being blown away by CNNs and GANs when they first gained popularity. I do think, with skin cancer diagnosis, a transformer architecture is going to be the better bet.
• armcat
  The same holds true for most classification tasks - if you have sufficient samples and high quality labels, training an xgboost or a deep learning model with a classification head is the way to go. You can use a frontier LLM on high-effort to help you with classification reasoning and explainability.
• arfelious
  LLMs shouldn't be treated as if they were purpose-built for specific cases just because they generally output a plausible-looking answer, and this is not something that was discovered only recently
• Menu

  chilmers
  Here's a good heuristic for predicting the ability of a modern frontier LLM at any non-automatable task: Is it hugely overrepresented in their pre-training dataset (e.g. code), and were they specifically post-trained on it using reinforcement learning (e.g. coding)? If the answer to one or both is yes, you should expect them to be very good at it. Otherwise, you should expect them to be, at best, as good as an average human who's read a few books on the subject and is good at bullshitting.
• anon
  undefined
• Menu

  everfree
  My dog outperforms the world's top researchers in an unassisted 100 meter sprint.
• Menu

  podgietaru
  This doesn't seem that surprising to me? One is tasked with a simple very narrow task. The other has a corpus including billions of other images and pieces of information. It'd seem to me that instinctively one would outperform the other, regardless of age.
• amelius
  Read this as: AI beaten by a 10yo.
• fastball
  This isn't really surprising. LLMs are intended to be generally intelligent. They are spiky on the things frontier labs tend to benchmark against (software/math) but are not intended or expected to be great at everything. The general intelligence unlocks new workflows, it does not perfect existing ones.That being said, I'm sure a frontier LLM could make some improvements to your CNN.
• Menu

  jonatron
  It's cool that there's an in-browser skin classifier...but it says "It is not a state-of-the-art model". The obvious question is, is there a free in browser state-of-the-art model?
• lynndotpy
  This isn't surprising but it's good to have reporting to point to. The only way a general purpose LLM like these will match the performance of a bespoke classifier is if it has access to such a model.
• Menu

  callamdelaney
  Yeah obviously. Xgboost outperforms llms in algorithmic trading, even applying the token prediction method to markets xgboost is effectively better.Llms have their applications. Other models still do.
• c0rruptbytes
  did you ask the LLM to make a CNN instead haha, might’ve won
• Kalanos
  this should not be surprising. alphafold is supervised learning.
• Kuyawa
  I'd like to see it compared to chinese models like DeepSeek, Kimi, GLM, etc just for the sake of research
• thomasjeff1
  An example of "Dumb it down"
• simianwords
  Couldn't you just train a general purpose LLM on the output of this CNN?Anyone wanna take a bet that the general purpose model will beat these specific models in a year or two?
• j45
  It seems one is more specialized for the purpose and compare it against a general purpose LLM instead of a specalized/focused LLM?This is like saying I compared this against photos for sure stored in a home swimming pool (CNN), compared to photos presumably stored in an ocean (generalized LLM).I use LLMs where I can, but I'm no automatic fan of LLMs, a healthier comparison here seems to be what if a general purpose LLM was provided those CNN images, and perhaps what does a fine tuned local model with the CNN images perform like?It still might not be better, but I would suspect it's closer, and a more realistic apples to apples comparison.General assistant models compared to task specific models seem to be a well known spot where things don't always fit, but might be improving.
• anon
  undefined
• Menu

  spwa4
  Which CNN? A fine-tuned version of Inception v3, fine-tuned on 68,011 dermatology imagesWhich LLM? GPT-6 Astra and Claude Fable 5.1, not specified exactly which versionObviously this is a bit like comparing an 100 iq radiologist, 20 years experience, with a just-tested-at-iq-130 6 year old and observing the radiologist beat the crap out of the kid. Was anybody really expecting a different outcome? Let's see after fine tuning.Btw: the actual lesson from that network when it came out, which is really important is that Inception v6, the Google trained version (ie. can recognize cats, halloween masks and airplanes) beat, with a wide margin, the version that was trained from scratch on clinical images (and still didn't do so well, 62%. Not even really good enough for a first triage).
• Menu

  ChrisArchitect
  fyi CNN: convolutional neural network
• scottchiefbaker
  [dead]
• tourist2d
  [dead]
• Menu

  howunfortunate
  I'm sorry but as an MLE this is not a remotely useful article, and does not belong on the front page.It's comparing a fine-tuned CNN to general purpose models on a tiny dataset.