jastang/spring-cloud-dataflow

Spring Cloud Data Flow provides orchestration for data microservices, including both stream and task processing

image:https://spring.io/badges/spring-cloud-dataflow/ga.svg[Latest Release Version, link=http://cloud.spring.io/spring-cloud-dataflow/#quick-start]
image:https://spring.io/badges/spring-cloud-dataflow/snapshot.svg[Latest Snapshot Version, link=http://cloud.spring.io/spring-cloud-dataflow/#quick-start]

== Spring Cloud Data Flow image:https://build.spring.io/plugins/servlet/buildStatusImage/SCD-BMASTER[Build Status, link=https://build.spring.io/browse/SCD-BMASTER] image:https://badge.waffle.io/spring-cloud/spring-cloud-dataflow.svg?label=ready&title=Ready[Stories Ready, link=http://waffle.io/spring-cloud/spring-cloud-dataflow] image:https://badge.waffle.io/spring-cloud/spring-cloud-dataflow.svg?label=In%20Progress&title=In%20Progress[Stories In Progress, link=http://waffle.io/spring-cloud/spring-cloud-dataflow]

The Spring Cloud Data Flow project provides orchestration for data microservices, including long-lived
https://github.com/spring-cloud/spring-cloud-stream[Spring Cloud Stream] applications and
short-lived https://github.com/spring-cloud/spring-cloud-task[Spring Cloud Task] applications.

== Components

The https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-core[Core]
domain module includes the concept of a stream that is a composition of spring-cloud-stream
modules in a linear pipeline from a source to a sink, optionally including processor module(s)
in between. The domain also includes the concept of a task, which may be any process that does
not run indefinitely, including https://github.com/spring-projects/spring-batch[Spring Batch] jobs.

The https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-registry[App Registry]
maintains the set of available apps, and their mappings to URIs.
For example, if relying on Maven coordinates, an app's URI would be of the format:
maven://<groupId>:<artifactId>:<version>

The Data Flow Server is a Spring Boot application that provides a common REST API and UI. For each
runtime environment there is a different version of the Data Flow Server that depends upon a
deployer SPI implementation for that environment. The github locations for these Data Flow Servers are:

There are also community maintained Spring Cloud Data Flow implementations for Hashicorp's https://github.com/donovanmuller/spring-cloud-dataflow-server-nomad[Nomad] and RedHat's https://github.com/donovanmuller/spring-cloud-dataflow-server-openshift[Openshift].

The deployer SPI mentioned above is defined within the https://github.com/spring-cloud/spring-cloud-deployer[Spring Cloud Deployer]
project. That provides an abstraction layer for deploying the apps of a given stream or task and managing their lifecycle.
The github locations for the corresponding Spring Cloud Deployer SPI implementations are:

The https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-shell[Shell]
connects to the Data Flow Server's REST API and supports a DSL that simplifies the process of
defining a stream or task and managing its lifecycle.

Instructions for running the Data Flow Server for each runtime environment can be found in their respective github repositories.

=== Building

Clone the repo and type