<- Back
Comments (40)
- reticulates> Those consequences are the reason that most database systems are built for either transactions or analytics, and not both. [24]> [...]> [24] People have tried to build one system that does both in the past (HTAP, or hybrid transactional/analytical processing), but those attempts generally didn't work out. Some newer systems seem to be bridging the gap, however. CedarDB, the commercial version of Umbra (a research system from TU Munich), runs transactions and analytics on just one engine and one copy of the data. Databricks' Lakebase keeps Postgres (a row store) as the engine for transactional queries, but its storage layer rewrites the same data into columnar formats in the background, allowing an analytical engine to query it.The footnote says "those attempts generally didn't work out" but then mentions Lakebase and CedarDB as newer efforts. Can anyone speak to whether or not CedarDB, Lakebase (and any others?) are achieving the fabled transactions + analytics in one database? Should we expect, at some point in the near future, for transactions + analytics to be standard in all databases, or is combining them a niche / esoteric thing with many tradeoffs?
- netcraftThe simple example of computing an average over a single column makes it seem like each columns data is stored by itself, but as soon as you want to do something with the correlation of two columns (such as a simple single column group by) doesnt that mean that each column would have to store both the column's value and some kind of row identifier that lets you join with the other columns in the same record? I've never looked into columnar stores so this is an honest question, I just always assumed thats how it would have to be structured
- remywang> Most formats split the table into large groups of rows and store the columns separately inside of each group (this is known as a "row group"). This idea goes back to PAX, from Weaving Relations for Cache Performance (VLDB 2001), and it's how Apache Parquet files work today.PAX’s “row group” has the same size as a disk page, whereas Parquet row groups are much larger so a page stores only data from the same column. I’ve been trying to track down the origin of this change in design but haven’t been able to find the right paper
- 4ndrewl"a database that keeps each record's details together can't take advantage of that: to read two columns, it has to read all five."My understanding is that modern databases will take advantage of that if you create an index of those fields - ie it'll read from the index, not the records.
- j0shk0Am I missing something or is this nothing more but AOS vs SOA ?
- postexitusWhat is the competition to spiraldb, providing similar functionality?
- kinleydGreat read! I do enjoy exploring both row and column data stores every now and then. Love the way I get my head bent.
- krawcuI enjoyed reading this post