NCRF++, a Neural Sequence Labeling Toolkit. Easy use to any sequence labeling tasks (e.g. NER, POS, Segmentation). It includes character LSTM/CNN, word LSTM/CNN and softmax/CRF components.

Content-Addressable Data Synchronization Tool

A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

Fully neural approach for text chunking

Alternative casync implementation

The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cognitive Search and RAG pattern.

A package for parsing PDFs and analyzing their content using LLMs.

A new chunking strategy developed by ZeroEntropy for general semantic chunking using Llama-70B.

A TensorFlow implementation of Neural Sequence Labeling model, which is able to tackle sequence labeling tasks such as POS Tagging, Chunking, NER, Punctuation Restoration and etc.

🍱 semantic-chunking ⇢ semantically create chunks from large document for passing to LLM workflows

PDFStract - The Extraction and Chunking Layer in Your RAG Pipeline - Available as CLI - WEBUI - API

A Python CLI to test, benchmark, and find the best RAG chunking strategy for your Markdown documents.

a monorepo featuring modular microkernel frameworks and single purpose extensions

Live TS segmenter and HLS manifest creation in Go

An LLM GUI application; enables you to interact with your files, offering dynamic parameters that can modify response behavior during runtime.

Postgres extensions to support end-to-end Retrieval-Augmented Generation (RAG) pipelines

An asynchronous event-driven HTTP client based on netty.

webpack 2, react hotloader 3, react router v4, code splitting and more

An Overview of the Latest Document Chunking Research

📑 Split Laravel jobs into multiple separate job chunks

Грамматический Словарь Русского Языка (+ английский, японский, etc)

Fast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.

smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and more time building what matters. From RAG systems to chatbots to document Q&A, SmartLLMLoader handles the heavy lifting so you can focus on creating exceptional AI applications.

One library to split them all: Sentence, Code, Docs. Chunk smarter, not harder — built for LLMs, RAG pipelines, and beyond.

Incremental asset delivery library

FastCDC implementation in Python https://pypi.org/project/fastcdc/

Labelling Sequential Data in Natural Language Processing with R - using CRFsuite

Extract and align grammar patterns from English sentences.

Build document-native LLM applications