Market Data ETL Platform
Services
Custom Software
Business Consulting
Industries
Energy sector
Client
RWE Supply & Trading GmbH
Period
Januar 2016 - Dezember 2018
The Market Data ETL Platform is an enterprise-wide system for automated acquisition, processing, and provision of energy market data at RWE Supply & Trading. In a dynamic trading environment, analysts and traders require access to current and historical market data from a wide variety of sources — from exchanges and regulatory authorities to industry services.
The platform is built on a microservices architecture with C#/.NET on the backend and Angular on the frontend. Specialized extraction services connect to diverse data sources via SFTP, email parsing, web scraping, and file imports (Excel, CSV). The extracted raw data passes through configurable transformation pipelines that convert heterogeneous formats into unified time series structures.
Processed data is stored in Amazon DynamoDB as a time series database and MSSQL for relational metadata. A generic REST API with role-based access control enables secure, enterprise-wide provision of market data. An Angular-based dashboard provides visualization, analysis, and monitoring of data flows.
The infrastructure runs on Amazon AWS with S3 for file storage and Docker for microservice containerization. The system processes data from over ten different sources daily, providing real-time insights for trading decisions.
Credits
Technologies