A curated list of awesome big data frameworks, ressources and other awesomeness.

Apache Kafka® running on Kubernetes

A lightweight stream processing library for Go

NIST Certified SCAP 1.2 toolkit

Probabilistic data structures for processing continuous, unbounded streams.

An open-source framework for real-time anomaly detection using Python, ElasticSearch and Kibana

A stream processing API for Go (now with generic type support)

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

Series and Panels for Real-time and Exploratory Analysis of Data Streams

Public tracker for Scramjet Cloud Platform, a platform that bring data from many environments together.

Data stream analytics: Implement online learning methods to address concept drift and model drift in data streams using the River library. Code for the paper entitled "PWPAE: An Ensemble Framework for Concept Drift Adaptation in IoT Data Streams" published in IEEE GlobeCom 2021.

Kafka-ML: connecting the data stream with ML/AI frameworks (now TensorFlow and PyTorch!)

The Open Source Time-Series Data Historian

The Tornado :tornado: framework, designed and implemented for adaptive online learning and data stream mining in Python.

Probabilistic deep learning for data streams.

Full-stack Highly Scalable Cloud-native Machine Learning system for demand forecasting with realtime data streaming, inference, retraining loop, and more

Event Based Applications [DEPRECATED]

An online learning method used to address concept drift and model drift. Code for the paper entitled "A Lightweight Concept Drift Detection and Adaptation Framework for IoT Data Streams" published in IEEE Internet of Things Magazine.

Explore Apache Kafka data pipelines in Kubernetes.

Learn how to use Kinesis Firehose, AWS Glue, S3, and Amazon Athena by streaming and analyzing reddit comments in realtime. 100-200 level tutorial.

🗂 List all repositories on Github (separated by language)

:gear: Ralph, the ultimate Learning Record Store (and more!) for your learning analytics

Simple yet powerful live data computation framework.

unsupervised concept drift detection

Data stream analytics: Implement online learning methods to address concept drift and model drift in dynamic data streams. Code for the paper entitled "A Multi-Stage Automated Online Network Data Stream Analytics Framework for IIoT Systems" published in IEEE Transactions on Industrial Informatics.

A suite of benchmark applications for distributed data stream processing systems

A simple C# network library.

Verifies data streams from synchrophasor measurement devices

In this project, we implemented a topic detection system on Twitter. This system reads tweets from a data stream and assigns them to one of the existing clusters or a new one. Each cluster acts as an agent, which makes the proposed approach a multi-agent system. There is also a coordinator, who monitors the whole system and coordinates the agent.

A library for performing Content-Defined Chunking (CDC) on data streams.