saved
The Dataflow Model Revisited
Hraness cites a source capture. The source author remains the source.
gist
Eleven years after the Dataflow Model, its authors grade their VLDB Test of Time work: event time, refusing to wait for completeness, and strong consistency aged well, but windowing, triggers, and a stream-only worldview did not. The winning path for analytics was the one they barely named—SQL and incremental materialized views with declared freshness—so streaming complexity disappears into the database. Watermarks remain for finality; snapshot-consistent refresh reached far more users by asking far less. Leave physics in, leave user-facing triggers out, and push streams-and-tables duality harder.
ideas
- Physics aged; the interface did not. Event time versus processing time, never waiting for completeness, and consistency-or-bust remain load-bearing; windowing-plus-triggers as the user-facing contract did not.
- Two roads from completeness. Watermarks still serve finality and absence reasoning; the bigger win was table-centric materialized views with snapshot-consistent refresh that never ask users to author pipelines.
- Declare freshness, not firings. Target lag / max_staleness / FRESHNESS replaced the trigger language: users state how stale is acceptable; engines schedule emission.
- Streams and tables are one object. Changelog versus snapshot is duality, not conversion; retractions become ordinary deltas; the product everyone wanted is the illusory table→table op with streams hidden inside.
- Analytics disappeared into the database. Beyond analytics, streaming complexity returns in apps and workflows until a general enough model absorbs it the way the relational model absorbed streaming analytics.
quotes
“We as a field must stop trying to groom unbounded datasets into finite pools of information that eventually become complete.”
“we got the physics right, but the interface wrong.”
“eventual consistency is entirely compatible with perpetual inconsistency.”
“Users should declare the result they want.”