大数据采集,抽取平台,zdh_web是zdh系列服务的可视化管理平台，包含数据采集,调度,权限,审批流,私域营销等模块

🏭 Mega Scale Multimodal DataPipeline for SOTA Foundation Models

Roadmap for Data Engineering

Official Implementation of "CLI-Gym: Scalable CLI Task Generation via Agentic Environment Inversion"

Simple stream processing pipeline

Build super simple end-to-end data & ETL pipelines for your vector databases and Generative AI applications

High Performance Tensorflow Data Pipeline with State of Art Augmentations and low level optimizations.

Step by step instructions to create a production-ready data pipeline

Tensorflow 2 Tutorials (use tensorflow and keras in a better way!)

Terraform module designed to easily backup EFS filesystems to S3 using DataPipeline

Awesome list for datapipeline

Ethereum client written in Go, modified for full-hierarchy data exports and block specimen production

Spark package to "plug" holes in data using SQL based rules ⚡️ 🔌

kedro cli plugin for generating a static kedro viz site (html, css, js) that can be deployed on many serverless tools.

Building Json data pipeline within Snowflake using Streams and Tasks

Domain-specific language to help build and maintain AWS Data Pipelines

This course is designed to provide learners with the fundamental skills needed for data engineering using Python. The objective is to introduce anyone interested in the topic to Python's data engineering-related features.

A GitHub Action to lint, test, build-docs, package, and run your kedro pipelines. Supports any Python version you'll give it (that is also supported by pyenv).

Go library that provides easy-to-use interfaces and tools for TensorFlow users, in particular allowing to train existing TF models on .tar and .tgz datasets

A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.

DBT and clickhouse test project with dagster

High speed message passing between various queues and services

Modeling tool like DBT to use SQL Alchemy core with a DataFrame interface like

This project predicts wind turbine failure using numerous sensor data by applying classification based ML models that improves prediction by tuning model hyperparameters and addressing class imbalance through over and under sampling data. Final model is productionized using a data pipeline

Extract transform load CLI tool for extracting small and middle data volume from sources (databases, csv files, xls files, gspreadsheets) to target (databases, csv files, xls files, gspreadsheets) in free combination.

A comprehensive project focusing on setting up and configuring the Elastic Stack (Elasticsearch, Logstash, and Kibana) for efficient log management and analytics. This project includes Elasticsearch configurations, Logstash pipelines, and Kibana visualizations, with detailed step-by-step documentation.

Global Tree Cover Loss Analysis using Geotrellis and SPARK

Материалы для курса Введение в Data Engineering: дата пайплайны

No description provided.

Simple Airflow on Kubernetes (GKE)