Real-Time Olist Lakehouse Data Platform | Databricks, PySpark, Delta Lake |
Built an end-to-end Lakehouse platform on Databricks + Delta Lake processing 500k+ records (orders, customers, order items), implementing scalable ETL pipelines for ingestion, profiling, cleansing, validation, and transformation. Designed Star Schema warehouse (fact + dimension tables) supporting customer, seller, and revenue analytics. Implemented incremental / near-real-time processing with Delta Lake MERGE, Structured Streaming, Auto Loader, and Databricks Workflows.
Autonomous Fleet Telemetry & Battery Degradation Anomaly Engine |
Architected real-time IoT Lakehouse pipeline processing high-frequency CAN-bus, GPS & BMS telemetry using Auto Loader, watermarked Structured Streaming, stateful sessionization & Delta CDF for battery thermal stress and cell voltage degradation detection. Built high-throughput Medallion architecture (Bronze → Silver → Gold) with micro-batch deduplication, 10-min watermarking, atomic MERGE INTO (SCD Type 1), Liquid Clustering, Z-Ordering & Unity Catalog governance. Implemented Gold-tier 3-sigma anomaly detection, CI/CD with PySpark unit tests & GitHub Actions, plus automated OPTIMIZE/VACUUM for scalable, production-grade reliability