The end of “good enough” predictions
Fifteen years ago, energy products were sold weeks or months in advance. Today, the energy system has changed drastically. “We moved from weekly products to daily, to 15-minute products, and now down to second-based settlements,” explains Philipp Cüppers, Head of Data Innovation at Vattenfall Hydro Germany.
While Vattenfall’s hydro assets are fully dispatchable, they operate in a market defined by uncertainty. The team needed to understand how hundreds of real-time sensor streams, weather patterns across Europe, and power prices from dozens of bidding regions all interacted. “Dashboards normally answer yesterday’s questions, because it takes time to build them,” says Cüppers. “We needed to provide answers for tomorrow. Asking questions in natural language is changing how data can be used for decision-making.”
A Lakehouse for the energy transition
The platform ingests data from real-time Kafka streams, delivering SCADA machine data and trading signals, REST APIs with renewable forecasts and price curves, operational databases with metering, historian systems for hydrology data, and pan-European transparency platforms.
All data flows through a medallion architecture — raw, qualified, and curated layers — governed by Unity Catalog and stored in Delta Lake, giving teams a trusted foundation for dashboards, natural language exploration, and custom applications. The qualified layer applies business logic, deduplication, and interpolation at sub-second granularity; the curated layer produces actionable insights for strategic decision-making. Data Quality is monitored through automated checks based on DQX, which check for freshness, domain-specific outliers, and gaps in the data. On any given day, the platform writes close to 3 billion rows as data cascades through all layers.
The platform runs on a mature DTAP infrastructure with multiple VNET-integrated Databricks workspaces on Azure, automated CI/CD, and a custom Core Library with over 40 modules encoding domain-specific logic — from Protobuf deserialization to reservoir scenario modeling.
