Azure Databricks & Synapse Analytics End-to-End Pipeline

This project demonstrates an end-to-end data pipeline using Azure Databricks, Azure Data Lake, and Azure Synapse Analytics . It showcases how to ingest, transform, and analyze data efficiently using modern cloud-based tools.

🚀 Project Overview

The pipeline includes:

• Data ingestion from raw sources (api sample data included in github data folder) into Azure Data Lake using Azure Data Factory
• Data transformation using Azure Databricks (PySpark notebooks)
• Data loading into Azure Synapse Analytics for reporting and analysis
• Integration with Azure Data Factory for orchestration

🛠️ Technologies Used

• Azure Databricks
• Azure Synapse Analytics
• Azure Data Lake Storage Gen2
• Azure Data Factory
• PySpark
• SQL

Data ingestion

• Data ingestion into Azure Data Lake using Azure Data Factory
• 

🧩 Architecture Diagram