Market Data ETL Platform

Contribution to an ETL platform for automated extraction, transformation, and storage of market data. The frontend is based on Angular, the backend on C#/.NET with MSSQL for relational data and Amazon DynamoDB for time series data. The infrastructure runs on Amazon AWS with S3 for file storage.

Technical features:

• Generic connectors for diverse data sources (websites, FTP servers, email, Excel)
• Automated data extraction with configurable transformation rules
• Standardized time series storage for efficient queries
• REST API with role-based access control
• Dashboard for visualization and analysis of market data

Development was managed agile with Atlassian JIRA and Git. The system processes data from over ten different sources daily and provides it in real-time for trading decisions.

Credits

Technologies

C#

CSS

HTML 5

TypeScript

MSSQL

.NET

Angular

NPM

Amazon AWS

Amazon DynamoDB

Amazon S3

Atlassian JIRA

Git