R7L208/delta-rs
A native Rust library for Delta Lake, with bindings into Python and Ruby.
:toc: macro
= delta-rs
image:https://github.com/delta-io/delta-rs/workflows/build/badge.svg[Build Status,link=https://github.com/delta-io/delta-rs/actions]
image:https://img.shields.io/crates/v/deltalake.svg?style=flat-square[Crate,link=https://crates.io/crates/deltalake]
image:https://img.shields.io/badge/docs-rust-blue.svg?style=flat-square[Docs,link=https://docs.rs/deltalake]
image:https://img.shields.io/pypi/v/deltalake.svg?style=flat-square[Python binding,link=https://pypi.org/project/deltalake]
image:https://img.shields.io/badge/docs-python-blue.svg?style=flat-square[Docs,link=https://delta-io.github.io/delta-rs/python]
![https://img.shields.io/crates/v/deltalake.svg?style=flat-square[Crate,link=https://crates.io/crates/deltalake]](https://img.shields.io/crates/v/deltalake.svg?style=flat-square%5BCrate,link=https://crates.io/crates/deltalake%5D){kind=link}
![https://img.shields.io/badge/docs-rust-blue.svg?style=flat-square[Docs,link=https://docs.rs/deltalake]](https://img.shields.io/badge/docs-rust-blue.svg?style=flat-square%5BDocs,link=https://docs.rs/deltalake%5D){kind=link}
![https://img.shields.io/badge/docs-python-blue.svg?style=flat-square[Docs,link=https://delta-io.github.io/delta-rs/python]](https://img.shields.io/badge/docs-python-blue.svg?style=flat-square%5BDocs,link=https://delta-io.github.io/delta-rs/python%5D){kind=link}
image::logo.png[Delta-rs logo]
A native interface to
link:https://delta.io[Delta Lake].
toc::[]
== About
This library provides low level access to Delta tables in Rust, which can be
used with data processing frameworks like
link:https://github.com/apache/arrow-datafusion[datafusion],
link:https://github.com/apache/arrow-datafusion/tree/master/ballista[ballista],
link:https://github.com/pola-rs/polars[polars],
link:https://github.com/rajasekarv/vega[vega], etc. It also provides bindings to other higher level languages such as link:https://delta-io.github.io/delta-rs/python/[Python] or Ruby.
=== Features
Supported backends:
- Local file system
- AWS S3
- Azure Blob Storage / Azure Datalake Storage Gen2
- Google Cloud Storage
.Support features
|===
| Operation/Feature | Rust | Python | Ruby
| Read table
| ✔️
| ✔️
| ✔️
| Stream table update
| ✔️
| ✔️
|
| Filter files with partitions
| ✔️
| ✔️
|
| Vacuum (delete stale files)
| link:https://github.com/delta-io/delta-rs/issues/97[#97]
| link:https://github.com/delta-io/delta-rs/issues/97[#97]
|
| History
| ✔️
| ✔️
|
| Write transactions
| ✔️
|
|
| Checkpoint creation
| ✔️
|
|
| High-level file writer
|
| link:https://github.com/delta-io/delta-rs/issues/542[#542]
|
| Optimize
| link:https://github.com/delta-io/delta-rs/issues/98[#98]
|
|
|===
== Get Involved
Join link:https://dbricks.co/delta-users-slack[#delta-rs in the Delta Lake Slack workspace]
=== Development Meeting
We have a standing development sync meeting for those that are interested. The meeting is held every two weeks at 9am PST on Tuesday mornings. The direct meeting URL is shared in the Slack channel above ☝️ before the meeting.
These meetings are also link:https://www.youtube.com/channel/UCSKhDO79MNcX4pIIRFD0UVg[streamed live via YouTube] if you just want to listen in.
=== Development
delta-rs requires the Rust compiler, which can be installed with the
link:https://rustup.rs/[rustup]
command.
Running tests can be done with cargo test in the root directory, or one of the directories below:
=== Rust
The rust/ directory contains core Rust APIs for accessing Delta Lake from Rust, or for higher-level language bindings.
=== Python
The python/ directory contains the deltalake Python package built on top of delta-rs
=== Ruby
The ruby/ directory contains an early prototype of a Ruby library built on top of delta-rs