Learn how to auto-ingest streaming data into Snowflake using Snowpipe.

SQL Scripts related to my learning on the Snowflake data cloud provider

Generating Streaming Data for Snowpipe Streaming

SQL, Databases, warehouses, Data lake, cloud storage, MYSQL, Data Pipeline

SQL, Databases, warehouses, Data lake, cloud storage, MYSQL, Data Pipeline

CLI script for interacting with Snowpipe REST APIs

Realtime data pipeline using Kafka + Spark + AWS S3 (Terraform) + Snowflake

No description provided.

The goal is to build an example of a simple data collection pipeline which collects data from multiple customers and uploads the data to Snowflake could look like. The OpenMeteo Api acts as the "customer system" in this case.

Automated Snowflake pipeline using Snowpipe, Streams, Tasks, MERGE logic, and AWS S3.

Advanced Healthcare Claims Pipeline using Snowflake, Snowpipe, Streams, Tasks, SCD Type 2, and AWS S3. Automates ingestion, CDC, dimensional modeling, and data quality checks for healthcare patient and claims data.

No description provided.

No description provided.

Snowflake DataScience Realtime HandsOn Projects

No description provided.

❄️ Stream real-time data into Snowflake with Amazon Kinesis Firehose

Automatically move data from AWS S3 to Snowflake datawarehouse continuously with snowpipe

Real-time ETL (Extract, Transform, Load) data pipeline to process insurance claims data with Snowflake, Apache Airflow, AWS S3, EC2, python pandas and creating a real time data visualization dashboard using Tableau.

End-to-end Change Data Capture (CDC) pipeline from PostgreSQL to Snowflake using AWS DMS, S3 (Parquet), Snowpipe, Streams, and Tasks. Demonstrates real-time data ingestion and incremental warehouse processing.

This project showcases a scalable ETL pipeline that automates the extraction, transformation, and storage of Redfin housing market data using AWS, Apache Airflow, and Snowflake, with Power BI for data visualization. The pipeline is configured to run monthly, ensuring your data remains up-to-date.

Get the streaming data from the S3 bucket with SQS queue. Load into Snowflake with Snowpipe and modify the data with Snowflake task

Implemented Snowflake project on AWS for efficient data storage and transformation. Utilized JSON-to-CSV conversion, Snowpipe for real-time ingestion, and reader accounts for secure data access. Employed streams, tasks, and materialized views for data synchronization and optimization. Implemented masking policies for enhanced data security.

End-to-end ETL pipeline for live cricket streaming data using AWS, Snowflake, and Power BI for visualization.

Analyze real-time global market data using AWS Kinesis and Snowflake. We utilize CSV datasets extracted via API calls, stream them through Kinesis Firehose, and transform them with Snowflake. Our agile workflow ensures efficiency, providing a one stop comprehensive solution for real-time data insights.

Building a Seamless Data Pipeline with AWS and Snowpipe

Vibecoded Snowpipe Streaming .NET SDK

Designed a cloud-to-cloud migration pipeline through migrating data with event-driven architecture and automated data loads. Implemented data quality checks at each stage, ensuring data consistency and reliability.

This is an end-to-end AWS Cloud ETL project. This data pipeline orchestration uses Apache Airflow on AWS EC2 as well as Snowpipe. It demonstrates how to build ETL data pipeline that would perform data transformation using Python on Apache Airflow as well as automatic ingestion into Snowflake data warehouse via Snowpipe. Also features Power BI.

Terraform-first implementation of Snowflake Basic Snowpipe, demonstrating infrastructure provisioning via external modules and JSON-driven configuration. Focuses on basic pipes (manual refresh) with COPY INTO, stage, file format, table, and ingestion validation workflows.

Retail data analysis pipeline utilizing AWS S3, Snowflake, Python, SQL, and Tableau. It demonstrates data transformation and setup in Jupyter Notebook, integrates real-time retail insights via an automated Tableau dashboard with Snowflake, and employs a CRON job in Jupyter Lab connected to Amazon SQS for consistent data updates.