Fluentd output plugin for Azure Datalake Storage Gen2 (append support)

OctopuFS library helps managing cloud storage, ADLSgen2 specifically. It allows you to operate on files (moving, copying, setting ACLs) in very efficient manner. Designed to work on databricks, but should work on any other platform as well.

Repository for all blog scripts and code

Procedimento para criação de um Azure Data Lake Storage usando Terraform, através de uma assinatura MS Learn Sandbox

Data Engineering Project on Supply Chain ETL. Creating a dynamic ADF pipeline to ingest both Full Load and Incremental Load data from SQL Server and then transform these datasets based on medallion architecture using Databricks.

Explore the Tokyo Olympics data journey! We ingested a GitHub CSV into Azure via Data Factory, stored it in Data Lake Storage Gen2, performed transformations in Databricks, conducted advanced analytics in Azure Synapse, and visualized insights in Synapse or Power BI.

This sample demonstrates how to create a Linux Virtual Machine in a virtual network that privately accesses a blob storage account using an Azure Private Endpoint.

Using SAS to authenticate and access to ADLS Gen 2 from Azure Databricks

COVID19-ADF is a project that leverages Azure services to collect, analyze, and visualize COVID-19 data. With seamless data integration and advanced analytics, it provides valuable insights into the pandemic's impact, enabling informed decision-making in the fight against COVID-19.

Azure ADLS Gen2 CLI Tool

Deploy apache spark in client mode on Kubernetes cluster, integrate with Jupyter notebook through Jupyterhub server.

POC projects working on Cloud Platforms

A Python package for Azure Datalake Storage adlsgen2 [abfss://] and Microsoft Fabric Lakehouse [abfss://], Google Cloud Storage [gs://bucket], AWS S3 bucket [s3://bucket] enables format detection and schema retrieval for Iceberg, Delta, and Parquet formats. It helps identify partitioned columns for parquet datasets. It also supports querying Delta,

Implementation of most useful services of Azure Data Platform.

ETL project with Spark and Airflow

🚀 Production ETL pipeline with Apache Airflow, Spark & Azure Data Lake

This project demonstrates an end-to-end data engineering pipeline using Azure and Databricks, following a Medallion architecture to process and analyze Netflix data.

Implemented Azure Databricks for real-time data processing and governance using Unity Catalog, Spark Structured Streaming, Delta Lake features, Medallion Architecture, and end-to-end CI/CD pipelines. Focused on incremental loading, compute cluster management, maintaining data quality, and creating workflows.

Databricks medallion architecture pipeline for NFL Big Data Bowl 2026 prediction using PySpark, Delta Lake, SparkML, and Azure ADLS Gen2

The objective of this project is to design and implement a scalable, cloud-based data pipeline using Azure Databricks, Azure Data Lake, and Azure Synapse Analytics

End-to-end Retail Sales Analysis using Databricks, Unity Catalog, and Spark. Automates data ingestion from GitHub sources to Bronze/Silver layers for exploratory data analysis.

Azure Data Lake Gen2 with azcopy

"Explore Formula 1 data analytics with this project. Leveraging the Ergast API, it utilizes Databricks Spark for ingestion, transformation, and analysis. ADLS acts as the storage layer, while Power BI visualizes the ADLS presentation layer. Uncover insights in the world of Formula 1 through powerful data analytics."

Contract-first Azure batch data product using Synapse Spark with deterministic recompute guarantees and audit-style run evidence.

This is an End to End Azure Data Engineering project copying data from Rest API to Azure cloud.

This project demonstrates how to build a modern, scalable data pipeline in the cloud using Azure Data Factory, Azure DevOps, Delta Lake, and Databricks. The pipeline builds silver and gold layers with PySpark and Delta Live Tables, and implements continuous integration using DevOps.

Spring Boot application that consumes Kafka messages in batches and writes them to Azure Data Lake Storage Gen2 using SAS authentication, generating one ADLS file per poll.

AirBnB CDC Ingestion Pipeline: Near Real-Time Change Data Capture (CDC) Pipeline on Azure for Seamless Integration of Continuous Data Streams

Explore the Paris Olympics data journey! We ingested a GitHub CSV into Azure via Data Factory, stored it in Data Lake Storage Gen2, performed transformations in Databricks, conducted analytics in Azure Synapse, and visualized insights in Synapse.

This project ingests daily equity price data from the yfinance API and processes it through a medallion architecture using PySpark on Databricks. The pipeline is orchestrated with Databricks Jobs and stores all intermediate and final datasets in Azure Data Lake Storage (ADLS).