HN

Comments (45)

• Menu

  Xiaoher-C
  The compile-time lineage part is the most interesting bit to me. A lot of “data lineage” tools feel like archaeology after the fact: parse logs, reconstruct what probably happened, then hope it matches reality.Having the compiler know “this column flows into these downstream models” before execution changes the workflow quite a bit. It makes refactors and masking policies much less scary.Do you expose any kind of “lineage diff” between branches? For example: this PR changes the downstream impact of `customer.email` from A/B/C to A/B/D. That would be useful in code review.
• Menu

  ramon156
  If your introduction message already includes a bunch of uncurated claims and LLM smells, then what does that say about the code I'm about to run?
• Menu

  jtagliabuetooso
  Cool release.IMO, "Why it's distinctive" is a bit misleading on a few points: certainly the dbt and DX folks can add their POV, but even considering stuff I know / authored ;-), https://arxiv.org/pdf/2308.05368 from 2023 (and following releases) cover branches in a native way (no clone), immutability (re-run), and lineage.Extensions to be considered are different languages (what about Python), and branch semantics. Two immediate questions would be: can you nest branches? How does merge works across systems if you don't control compute?
• Menu

  mollerhoj
  Its a bit confusing to claim that "The things your current stack can't give you because it doesn't own the DAG" and use DataBricks as your example: DataBricks includes jobs and pipelines, so it very much owns the DAG, no?
• Menu

  data_ders
  hiya, anders from dbt here. cool project -- I especially love the branching and budgeting options you've built in. both are things that I'd love for the dbt standard to include one day. was it dbt's lack of those feature that inspired you to start this project? It also seems you have an aversion to Jinja, which, believe me, I get!FYI dbt-fusion [1] is going GA next week (though GA for Databricks will come later) Most of it is source-available and ELv2-licensed, but there's a number of crates that are Apache 2.0, namely: dbt-xdbc, dbt-adapter, dbt-auth, dbt-jinja, dbt-agate. We also have plans to OSS more as time goes on (stay tuned).I just wanted to call out the OSS crates in case you'd rather focus on "making your beer taste better" than have to re-build foundations. I'd love to hear if any of those crates come in handy for you (even more so if they don't work for you).Feel free to reach out on LinkedIn or dbt community Slack if you ever want to chat more![1]: https://github.com/dbt-labs/dbt-fusion
• Menu

  PeterWhittaker
  Congrats on the work, but have you considered another name? Naming is hard and always will be: When I first scanned the headline, my initial thought was "that's an interesting area for the Rocky Linux team to explore". After a moment, "wait, no, that's confusing, it's some other Rocky".
• Menu

  hasyimibhar
  Looks cool, I've been waiting for someone to build this since dbt and SQLMesh acquisition. It would be great to have model versioning and support for ClickHouse SQL.
• alphainfo
  [flagged]
• Menu

  mergisi
  * * *
• ahmad212o
  [dead]
• anon
  undefined
• Dorrell
  [dead]
• Dorrell
  [dead]
• anon
  undefined