Airflow Unit Tests and Integration Tests

Gerador de DAGs no Apache Airflow para fazer clipping do Diário Oficial da União.

Queue-Based State Machine - A lightweight workflow execution engine with DAG-based stage orchestration. Unlike simple task queues (like Celery) or advanced orchestrators (like Highway), Stabilize strikes a balance specifically optimized for high-throughput, stateful DAG execution, making it highly suitable for coordinating autonomous AI agents

Classwork projects and home works done through Udacity data engineering nano degree

Sync DAG changes from Git to Airflow

Data Engineering examples for Airflow, Prefect; dbt for BigQuery, Redshift, ClickHouse, Postgres, DuckDB; PySpark for Batch processing; Kafka for Stream processing

No description provided.

Zero configuration Airflow plugin that let you manage your DAG files.

🎵 LyricWave – AI Music Composer (Proof of Concept) 🎶 A personal project exploring automatic generation of unique MP4 songs. LyricWave blends lyrics with AI-generated melodies and synthetic vocals to experiment with new forms of musical expression. A creative testbed to push your ideas into sound. 🚀🎧

Orchestration of data science and earth observation models in Apache Airflow, scale-up with Celery Executor, experiment with jupyter notebook using a docker containers composition

This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.

Apache Airflow Guide

My self-learning about Apache Airflow

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows

Loan Default Prediction using PySpark, with jobs scheduled by Apache Airflow and Integration with Spark using Apache Livy

WebApp to bring together Text Summarization and Sentiment Analysis of the stock related news to better understand the stock price trends.

An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.

A comprehensive ELT pipeline for analyzing passenger satisfaction data. Features a modern data architecture with Apache Airflow for extraction, dbt/Snowflake for transformation, Python/Pandas for cleaning, and interactive dashboards for visualization with NextJS.

Here I added 9 projects which have been made by me during my apprenticeship in Yandex.Practicum as data engineer.

This project creates a basic web service for solving image-based CAPTCHAs. Using the Flask framework, it allows users to upload CAPTCHA images and employs an Optical Character Recognition (OCR) pipeline to extract the embedded text.

Apache Airflow DAG examples

A starting point for a data stack using Python, Apache Airflow and Metabase.

An ETL Data Pipelines Project that uses AirFlow DAGs to extract employees' data from PostgreSQL Schemas, load it in AWS Data Lake, Transform it with Python script, and Finally load it into SnowFlake Data warehouse using SCD type 2.

DAGs, Python Scripts, and SQL scripts for creating and updating the Saint Louis Regional Entity Database

dag sketch tool (dst) helps you create and design Apache Airflow DAGs visually and programmatically.

End-to-end data engineering processes for the NIGERIA Health Facility Registry (HFR). The project leveraged Selenium, Pandas, PySpark, PostgreSQL and Airflow

Automate your data pipelines using Apache Airflow with this ready-to-use DAG for data integration, ETL and workflow automation.

A simple AirFlow mwaa cli command utility. It can be used to pause all the DAGS for a MWAA environment

Automated pipeline for generating and processing Dock balance reports using Apache Airflow, SFTP, AWS S3, and Lambda.

This is a simple action that helps you to fetch your Apache Airflow DAGs to Google Cloud Composer