TypeStream

Why TypeStream? · Getting started · How to contribute · Code of conduct · License

TypeStream is an abstraction layer on top of Kafka that allows you to write
and run typed data pipelines with a minimal, familiar syntax.

Why TypeStream?

Building streaming data pipelines on top of Kafka comes with some fixed costs.
You have to write an app, test it, then deploy and manage it in production. Even
for the simplest pipelines, this can be a lot of work.

With TypeStream you can write powerful, typed data pipelines the way you'd write
a simple UNIX pipeline in your terminal. For example, imagine you'd like to
filter a "books" topic. With TypeStream, it's a one liner:

$ typestream
> grep /dev/kafka/local/topics/books the > /dev/kafka/local/topics/books_with_the

TypeStream will take care of type-checking your pipeline and then run it for
you. Here's how it looks like in action:

Another common use case that requires a lot of boilerplate is to enrich a topic
with data from an external source. For example, you might have a topic with an
ip address field and you may want to enrich it with country information. With
TypeStream, you can do it (again!) in a single line:

$ typestream
> cat /dev/kafka/local/topics/page_views | enrich { view -> http "https://api.country.is/#{$view.ip_address}" | cut .country } > /dev/kafka/local/topics/page_views_with_country

Here's how enriching looks like in action:

As you can see from the previous command, in the spirit of UNIX, we used cut to
extract the country field from the response. Here's the kick, you can use cut
(and many other Unix commands) on streams as well:

$ typestream
> cat /dev/kafka/local/topics/books | cut .title > /dev/kafka/local/topics/book_titles

Here's how cutting looks like in action:

Another problem that TypeStream solves is preventing you from writing faulty
pipelines by type-checking them before executing them. Here's type checking in
action:

Deploying a pipeline to production is as simple as running:

typestream run 'cat /dev/kafka/local/topics/books | cut .title > /dev/kafka/local/topics/book_titles'

See it in action here:

If you'd like to learn more about TypeStream, check out the official
documentation.

Getting started

If you use Homebrew:

brew install typestreamio/tap/typestream
$ typestream --version

if you see something like this:

typestream version 2023.08.31+3 42f7762daac1872416bebab7a34d0b79a838d40a (2023-09-02 09:20:52)

then you're good to go! You can now run:

typestream local start
typestream local seed

to start a local TypeStream server and seed it with some sample data. Now you're
ready to start writing your own pipelines!

Check out our documentation to learn more about
TypeStream.

How to contribute

We love every form of contribution! Good entry points to the project are:

• Our contributing guidelines document.
• Issues with the tag
  gardening.
• Issues with the tag good first
  patch.

If you're not sure where to start, open a new
issue or hop on to our
discord server and we'll gladly help you get
started.

Code of Conduct

You are expected to follow our code of conduct when
interacting with the project via issues, pull requests, or in any other form.
Many thanks to the awesome contributor
covenant initiative!

License

Apache 2.0