HN

Need help?
<- Back
Remove pgrust: the entry is cheating (per-query kernels and result caching)

Remove pgrust: the entry is cheating (per-query kernels and result caching)

f311a

Comments (118)

  • malisper
    Hi! I'm the creator of pgrust. Let me explain what happened.For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview. We just open-sourced a newer version called pgrust 0.4-beta yesterday, and you can fully audit it here[0]. Rather than contest the accusations on GitHub, we figured it would be easier to complete pgrust 0.4 and open-source it to show that we're being honest. You can also verify what I'm saying yourself against the 0.4-preview binary we submitted to ClickBench here[1]. I don't have a way to prove this is the exact binary submitted, but if ClickHouse provided the SHA of the binary they tested against, it would match.The three main accusations boil down to: 1) We used handwritten code optimized against each query in the benchmark, 2) We cached the results between query runs, and 3) We didn't support queries outside of the 43 ClickBench queries.First, we did not use handwritten queries. The code Alexey pointed out does not run during the benchmark. The design of 0.4 relies on a JIT compiler that compiles your SQL to a Lisp-based intermediate representation that we call Gravity IR that we then convert to x86 and ARM assembly. While developing 0.4, I wrote handwritten code against the IR for benchmarking purposes. The idea was that if queries written in our IR are fast, and we can compile good enough IR fast, then our new query engine will be fast. This is also the reason the directory those "kernels" are in is called "irprobe" - it was a probe of our IR to make sure it was fast. Our 0.4-preview submission went through the pipeline: SQL query -> Gravity IR -> assembly.Second, the "result-cache" Alexey observed is the result of a cache we use to cache which predicates match which strings in our dictionaries we use for dictionary compression. Alexey wasn't aware of this cache and assumed we were just caching the query results. On ClickBench you are allowed to use caches like this that hold intermediate data[2]. ClickHouse has its own cache similar to this called the Condition Cache[3]. In our newest submission, we were upfront about all the caching we do and how to disable each cache[4].The third claim that we didn't support other queries outside of the 43 ClickBench queries is partially false, but does have some ground to it. As I said before, pgrust 0.4-preview was an early version of pgrust 0.4. Once our new JIT compiler supported enough SQL to handle the ClickBench queries, we decided to submit it to ClickBench. AFAIK, ClickBench has no rules about how much SQL you need to support. On top of that, we did ship that we did ship 0.4-preview with our old query engine. It would have been a two-line change to fall back to it and support a much broader set of queries outside what our new engine could support. Those queries would still be fast, but not quite as fast as our new engine was. Our new 0.4-beta release does now support all Postgres SQL (minus some bugs around CTEs and MERGE)To sum up, pgrust 0.4-preview was an early build of pgrust 0.4 while it was in development. It was a real query engine and actually executed queries. I acknowledge that we submitted it prematurely, when it supported a narrow set of SQL, and we should have ensured broader SQL coverage before submitting it. With our new submission that solves these problems, and we're hoping for pgrust to be reinstated. If there's anything off about the submission, please point it out and we're happy to fix it.[0] https://github.com/malisper/pgrust/releases/tag/v0.4-beta[1] https://pgrust.com/downloads/archive/2026-09-clickbench-snap...[2] https://github.com/ClickHouse/ClickBench/blob/main/README.md[3] https://clickhouse.com/docs/concepts/features/performance/ca...[4] https://github.com/ClickHouse/ClickBench/pull/2751
  • whinvik
    Wow the mood is aggressive here.Its like a lot of people really want pgrust to fail.There maybe a chance this was cheating or it may have been a misunderstanding. But now since there is an open source version, why is there still this strong sense of negativity.
  • drchaim
  • torben-friis
    This idea of "here's the beta, I'll release the code in a few weeks" sounds counter intuitive to me. is it normal in open source? Any reason for it?
  • bmacho
    ai;dr: a bench result is not randomized, and it returns cached result somewhere.My opinion: (some) bench results should run on randomized data.
  • stemchar
    Oh lord. I thought pgrust was a serious project.
  • frevib
    These massive PR summaries. Is anyone really reading them?
  • dheatov
    That's one high profile instance of human vs robot. Surreal.
  • teiferer
    For anybody not familiar with pgrust or ClickBench:Pgrust is a re-implememtation of postgresql in rust.ClickBench is a benchmark suite for databases.The link goes to a removal request of pgrust fromthat benchmark because of its alleged cheating.
  • HackerThemAll
    I’ll sit out this drama until pgrust 0.5.
  • verdverm
    Sounds kind of like Claude was "cheating" and the humans are engaged in eyes-free coding. Diff so big github refuses to render. I suspect they are reaching the limits of what this new vibe "paradigm" of programming can do. If you haven't pushed a vibe project far, eventually progress becomes near impossible because the slop has grown and the conditions are overly complex, locally focused. Ai is not good at writing reusable software (on their own).stay in the loop friends
  • mschuster91
    > The public v0.3 source confirms the design > The hot runs are a result cacheFunny, the report alleging cheating reeks of LLM usage just as much as the commit diff [1] of the next attempt:> # The standard ClickBench driver over the step scripts in this directory; ./install fetches the dataset.[1] https://github.com/ClickHouse/ClickBench/pull/2751/files
  • gourneau
    [flagged]
  • chelseahermes
    [flagged]
  • giozadi
    [flagged]
  • NoDodgeQuestion
    [flagged]
  • acalvino4
    Even before the ai age, people published inaccurate/misleading benchmarks all the time. Sometimes those were actual cheating, and other times an honest mistake. What matters is that the author acknowledges the issue and attempts to correct the mistake, which clearly seems to have happened here: https://github.com/ClickHouse/ClickBench/pull/2751Most of this thread doesn't seem to acknowledge that, and feels more like anti-ai psychosis using this incident as an anecdote to justify the pre-existing hatred of ai-generated projects.