HN

Need help?
<- Back
Evidence of inconsistencies in evaluation process and selection of winners

Evidence of inconsistencies in evaluation process and selection of winners

twerkmeister

Comments (240)

  • ecshafer
    AI is useful. But the amount of people that are simply offloading all of their thinking to AI and blindly accepting the answer is absurd. Kaggle is most likely using ai to assess the submissions and are not using any common sense by blindly accepting the results.
  • hoppp
    I don't know about this exact competition but overall fair hackathons have been killed by AI.It all seems fine from the outside but all the code is generated in all the projects and judging happens via AI, I have seen projects win because they prompt inject that they are the winners.It used to be about human skill, now it's about ideas and of course insiders are the main winners.
  • mr_toad
    People have been using brute force methods to win Kaggle competitions since the beginning (and other people have been complaining about it just as long).At its core, ML is all about computer generated models (automated feature selection, hyper parameter tuning). Many (most?) of the models produced in Kaggle are already black boxes and have been for a long time. The model that won the Netflix prize was never used in production for that reason.Using an LLM to generate code to generate a black box is pretty much par for the course.
  • throwfaraway135
    AI submissions and AI judges a match made in (AI) heaven.
  • ryukoposting
    I thought Kaggle was a website where you download dubious CSV files of annualized bean consumption in Bolivia, or whatever.Was Kaggle ever a reputable source of original research, or a source of anything with any provenance at all? That would be news to me. The fact that 25 grand was involved this time is unique, I guess.
  • apwheele
    I think this is a good meta-lesson for Kaggle. When you have objective metrics to hill-climb towards, AI can do quite well. When you just phone it in and rely on LLM as a Judge, the results are not so great.
  • irasigman
    It’s a shame that Arvix (and once thoughtful places like Kaggle) are used for self-promotion.I get people want to work at an AI lab but slopping it in public in this manner is counterproductive to the original intended purpose of these places.
  • sublimefire
    Went through the comments here and there and one thing to note is that there was a question about who do you think should have won instead. This is a good question because it is possible that all submissions were like this or there were ones that looked just worse. It would be quite useful to know who came close as well in this case. If you knew which submissions were good you could have a process to revoke the prize and give it to someone else in case of fraud or negligence or similar.Having said that it is also possible that the mistakes and claims were a human error, sure a lot gets ai generated these days but there is a chance in which case the accusation does not look so severe anymore.
  • ablation
    "I think you just need to accept the results of the competition. The winning submissions clearly provide value and had a lot of effort invested in them. I'm not really worried about a few inconsistencies or mistakes if the value is still there. Did you think another submission deserved to win over these?"That comment is gold. Yeah, I'm not worried about hallucinated slop, just accept it was the winner folks.
  • boringg
    Best question - how many tokens in dollars were spent to win the comp?
  • liveoneggs
    <obama medal meme>
  • anon
    undefined
  • greenleafone7
    I think there is a saying about having a million monkeys with typewriters... or something of that sort.
  • GodelNumbering
    > "Finding 1: Scale Buys Evaluation, Not Control"The attached paper's (https://arxiv.org/pdf/2604.16009) title is "MEDLEY-BENCH: Scale Buys Evaluation but Not Control in AI Metacognition"This is the most blatant Claude line, or as Claude would put it, the smoking gun.
  • SwellJoe
    This is why I'm not wasting my time on these things (either in participating or in paying attention to the results). The noise is just too high. I'd be furious if I'd spent a bunch of time on doing actual work for something like this only to have slop win. Hopefully, the other contestants worked on stuff that is transferable to other purposes.
  • nsagent
    Sadly, the major ML/AI/NLP conferences are being inundated with AI slop papers. That will arguably have a bigger impact on the quality of research moving forward.
  • 27183
    What's up with all the AI generated responses on that page?
  • anon
    undefined
  • josefritzishere
    Gross
  • tantalor
    Flagged, editorialized title
  • twerkmeister
    Please note this Post was just renamed without my involvement from:Blatant AI slop just won a 25K USD Deepmind Kaggle Grand Prizeinto"Evidence of inconsistencies in evaluate process and selection of winners"
  • htrp
    deepmind using AI to evaluate submissions?
  • blueTiger33
    overall, the quality of products has been going downhill.AI is not there yet, instead of working hard, everyone is choosing the easy way out.AI slop wins prize, I wonder if Ai slop read it also. would not be surprised. however not to judge anyone, I think we are seeing slop everywhere, hope some things still require hard blocks for low quality.its difficult to justify lack of attention and details
  • darkxanthos
    The real story here is the judging potentially being AI slop.
  • esafak
    I think the heyday of Kaggle is in the past. I would not lose sleep over this site.
  • jgilias
    It was probably scored by AI too. Same reason why slop-filled resumes apparently work better these days.
  • voxleone
    [dead]
  • supersoftware
    [dead]
  • polterguy-hyper
    [dead]
  • subwaydash
    [dead]
  • ndbe
    The amount of slop in the replies is just sad.
  • animanoir
    [dead]
  • onesandofgrain
    AI is 95% useless. Not quite worth the trillion dollar market cap lol.* The AI bots are downvoting me * hooray
  • biosboiii
    LLM-as-a-judge?Given that LLMs are trained with RL && LLM-as-a-judge, is it really cheating if real competitions use the same?Maybe the real alignment is the slop we decoded along the way
  • blacklimetea
    What a whiner!The problem with removing bullying from the upbringing process is you get insufferable twats like this who can't take "No" for an answer and who can't take a loss.His mom told him "Everything you do is art!"
  • Zsfe510asG
    All AI companies have slop press sites that hype them up. What we saw since 2020 is the largest industrial propaganda campaign in history. It started with Lex Fridman planted interviews to make AI researchers appear human and ends with AI awarding AI prizes.Mainstream journalists didn't know any better and thought they were reading secret inside information and parroted it - until now when the house of cards is collapsing.Notice that the defense in the comment section is the Silicon Valley platitude that "it provides value". No sane person believes that any longer, only the financially invested and some SciFi trash addicts.
  • fg137
    I always find it interesting when I see posts here around "LLMs are just fancy autocompletion machines" and there are 100 comments below it.
  • jesse_dot_id
    I think that a lot of software engineers are using LLMs and a lot of very popular tools are developed by, or are assisted by, LLMs. Is this not just going to be a thing going forward?This feels akin to traditional artists getting angry at digital art winning competitions when that was a new concept.We're simply in the early stages of a paradigm shift, no?