Viinay Kumaar

This repo contains details about Movies CDC data using Snowflake Dynamic Tables and Building Snowflake Streamlit app, Thanks

This repo contains details around Sample Airport data sourced in ADLS and leveraging ADF to transform the data and loading into ADLS and implementing ADF Pipeline in Dev and leveraging Azure DevOps CI/CD to deploy the pipeline to PROD using DevOps Release/Pipeline using ARM template method. Thanks

This repo contains details about how to extract API data from News API website and leveraging airflow to load the API data into GCS bucket in Parquet format and using Airflow to load the data from GCS to Snowflake Tables as needed, Thanks

This repo contains details about BookMyShow Mock data leveraging Azure Event Hub Bookings and Payments Streams and Azure Stream Analytics to Consume the Streams datasets and loading into Azure Synapse Table and Leveraging n8n to build Final Leadership Report Workflow to send email, Thanks

This repo contains details about building near real time pipeline using Azure ADLS, CosmosDB and building Facts and Dimensions in Synapse Dedicated SQL Warehouse/Database and leveraging n8n to build workflow and send booking confirmation email, Thanks

This repo contains details about dbt with Databricks project

This repo contains details about Azure Fintech data pipeline involving datasets from Azure SQL Database to ADLS Storage as Target with Multiple Facts and Dimension tables, Thanks

This repo contains details about travel booking project executed on Databricks, Thanks

This repo contains details about how food delivery data pipeline is implemented in real time using Multiple AWS services and Spark Streaming, Thanks

This Repo holds information about various Azure data engineering projects

This Repo contains details about Kafka Real Time Stock Market Data Engineering Project, Thanks

This repo contains details about real time streaming implementation using Confluent Cloud Kafka as Source and MongoDB as Sink. Thanks

This repo contains details about end to end implementation of the GCP GCS to BQ pipeline using CI/CD leveraging Airflow DEV and PROD Environments, Thanks

This repo contains details about how API data is extracted using Python and using Pyspark performing transformations and loading the data from Bronze to Silver to Gold tables and creating PowerBI report on the top of the Gold table present in the lakehouse

This Repo contains details about Extracting Yahoo Finance API using Apache Airflow, Thanks

This repo contains details about building Azure Analysis Model from On Prem SQL Server as Source

Newsletter to help busy software engineers become good at system design 👇

This repo contains details about using GCP Cloud Run function with CI/CD with sample datasets. Thanks

This repo contains details about how to use Airflow to create a Backfill DAG using Python and Pyspark and load the data accordingly from source to destination GCS buckets, Thanks

This repo contains details about leveraging the Airflow in the GCP Composer and running the Ephemeral Dataproc Cluster to run Pyspark Job and Orchestrate within the Airflow, Thanks

This repo contains details about implementation of the real time streaming using GCP Cloud Pub/Sub Service leveraging sample datasets and using Python code. Thanks

This repo contains details about leveraging the Confluent Cloud Kafka for performing real time streaming project leveraging retail sales data and to perform end to end solutions, Thanks

This repo contains details about how dbt is leveraged with Snowflake and Preset is used for Dashboarding, Thanks

This repo contains details about data engineering project leveraged using Microsoft Fabric

This Repo contains details about Running Airflow on GCP Cloud VM Instance and Building end to end Data Engineering Project using multiple GCP services, Thanks

Docker Apache Airflow

This is a repo with links to everything you'd ever want to learn about data engineering

No description provided.

Python Deep Dive Course - Accompanying Materials

This repo contains details about Microsoft Fabric Uber Data Analytics