HN

Comments (14)

• Menu

  scott_s
  I worked in the streaming area for a decade, doing research and development (see: https://scholar.google.com/citations?user=Rdf5OIYAAAAJ&hl=en). After moving on from streaming specifically and moving into the general problems in large data warehouses, I also concluded: just default to SQL for all analytics and the database lens is the best way to think about streaming for analytics.I still do think that stream programming models are extremely interesting and powerful. But I used to think they would eventually become more mainstream as a way to elegantly program for high throughput, low latency massively parallel systems. That has not been the case, and I no longer think that it will be. People get by with the existing programming languages and models, that seems to be fine.
• jasonwatkinspdx
  I've always thought this paper was a bit of an overlooked gem: https://www.vldb.org/conf/2002/S07P01.pdfIt views stream processing as a join between queries and data, where internally everything is just materialized views probing each other.Interesting that the authors above ended up landing in a similar place conceptually.
• janpeuker
  I used to be extremely into Dataflow/Apache Beam and literally had the paper printed on my desk AND the book. I agree they got Event time versus processing time and Never rely on completeness right, and I love they go deeper into why that was just a hard thing to accept. My head still hurts thinking about unbounded stream triggers and I'm happy we got around to a table-centric model. I still think taking some ideas from Spanner, databases as message bus or consistency information per row, basically in-database CQRS would have been nice, though. Great paper.
• guglecwoam
  > SplittingThis is one of the things I wished for in the Beam APIs, and I'm glad they included it in the doc. Some (single) elements are bigger than a single function, and can't be processed one at a time. For example. a very large CSV file. They added support for it later (as Splittable DoFn's) but each implementation had varying support for it.
• 7e
  Streaming is expensive, both in terms of machine and human cost, and the benefit is quite marginal for most applications. However, Google engineers need to get promotion somehow, you know?
• kevinbaiv
  [flagged]
• rawgabbit
  TLDR. 15 years ago they avoided SQL and created a language for streaming. Today, the industry decided that SQL materialized views with a “freshness” or time lag promise is the correct mental model.
• lincpa
  [dead]