Implementing best practices for PySpark ETL jobs and applications.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Mass processing data with a complete ETL for .net developers

A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.

Provides guidance for fast ETL jobs, an IDataReader implementation for SqlBulkCopy (or the MySql or Oracle equivalents) that wraps an IEnumerable, and libraries for mapping entites to table columns.

No description provided.

Terraform modules for provisioning and managing AWS Glue resources

A Python PySpark Projet with Poetry

This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.

This repo will guide you step-by-step method to create star schema dimensional model.

An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.

A declarative, SQL-like DSL for data integration tasks.

Built a Data Pipeline for a Retail store using AWS services that collects data from its transactional database (OLTP) in Snowflake and transforms the raw data (ETL process) using Apache spark to meet business requirements and also enables Data Analyst create Data Visualization using Superset. Airflow is used to orchestrate the pipeline

Extract transform load CLI tool for extracting small and middle data volume from sources (databases, csv files, xls files, gspreadsheets) to target (databases, csv files, xls files, gspreadsheets) in free combination.

Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE

PHP ETL library: pipeline of extractors, transformers, and loaders (CSV/JSON/DB, etc.) run via a fluent API.

A simple in-memory, configuration driven, data processing pipeline for Apache Spark.

A data pipeline from source to data warehouse using Taipei Metro Hourly Traffic data

Telecom ETL is a SSIS package that ingest it's data from CSVs to DB

Comms processing (ETL) with Apache Flink.

Sentiment Analysis of Tweets Using ETL process and Elastic Search

An ETL pipeline where data is captured from REST API (Remotive, Adzuna & GitHub) and RSS feeds (StackOverflow). The data collected from the API is stored on local disk. The files are preprocessed and ETL jobs are written in spark and scheduled in Prefect to run every week. Transformed data is moved to PostgreSQL.

python 3.5 package for ETL jobs

Building a modern data warehouse with SQL Server, including ETL processes, data modeling and analytics.

Event-Driven Python on AWS #CloudGuruChallenge

This project involves using the Reddit API to extract data, processing it using EC2 instances, and storing the output in CSV format within an AWS S3 bucket, with Airflow managing the overall workflow orchestration.

Ferramenta para Cópia de Dados SQL Server, que foi desenvolvida para auxiliar na geração de arquivos e cópia eficiente de dados entre bases de dados SQL Server.

Code for unofficial API for the brazillian stocks data website called Fundamentus. Uses requests and bs4 for scraping

This is a simple ETL Batch processing to extract data, here are messages, stored in a table. transform them into a new object, then insert them in another table

A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.