ilayaperumalg/spring-cloud-dataflow-acceptance-tests

= Spring Cloud Data Flow Acceptance Tests =

This project bootstraps a dataflow server on a target platform, executes a series of tests by creating a series of streams and tasks and then cleans up after its done.

== How to run it

The main script is called run.sh and supports a few flags:

USAGE: run.sh -p <PLATFORM> -b <BINDER> [-s -t -c]
  The default mode will setup, run tests and clean up, you can control which stage you want to
  have executed by toggling the flags (-s, -t, -c)

Flags:

[*] -p  | --platform - define the target platform to run
    -b  | --binder - define the binder (i.e. rabbit, kafka) defaults to rabbit
    -tests - comma separated list of tests to run. Wildcards such as *http* are allowed. (e.g. -tests TickTockTests#tickTockTests)
    -s  | --skipSetup - skip setup phase
    -t  | --skipTests - skip test phase
    -c  | --skipCleanup - skip the clean up phase
    -d  | --doNotDownload - skip the downloading of the server
    -m  | --skipperMode - specify if skipper mode should be enabled
    -cc | --skipCloudConfig - skip Cloud Config server tests for CF
    -sv | --skipperVersion - set the skipper version to test (e.g. 1.0.4.BUILD-SNAPSHOT)
    -dv | --dataflowVersion - set the dataflow version to test (e.g. 1.5.0.BUILD-SNAPSHOT)
    -av | --appsVersion - set the stream app version to test (e.g. Celsius.SR2). Apps should be accessible via maven repo or docker hub.
    -tv | --tasksVersion - set the task app version to test (e.g. Elston.RELEASE). Tasks should be accessible via maven repo or docker hub.
    -se | --schedulesEnabled - installs scheduling infrastructure and configures SCDF to use the service.
    -na | --noAutoreconfiguration - tell the buildpack to disable spring autoreconfiguration

[*] = Required arguments

The first option is to choose a PLATFORM, available options cloudfoundry, local, gke (kubernetes) and pks (kubernetes). The scripts for each platform are located in folders of the same name in the main directory.

By default the script will execute three main phases:

Setup: The setup phase will traverse each folder and call create.sh scripts. At the end of this phase you should expect to have an environment available with the Spring Cloud Data Flow server along with the services required for it to run.
Test: The test phase will invoke the mvn test and deploy apps into the environment and run tests.
Clean: The clean up phase will undeploy the server and remove any services.

Each phase can be toggled by setting the appropriate flag (-s, -t, -c)

The services created in the setup phase are mysql and redis. Depending on the binder selected, it will create a Rabbit or Kafka service.

== Examples

To run the tests locally cleaning up services using Rabbit (the default binder)

./run.sh -p local

To run the tests locally cleaning up services using Kafka

./run.sh -p local -b kafka

To run the tests locally and keep the Data Flow Server, Kafka, and other services running afterwards

./run.sh -p local -b kafka -c

To run the tests on cloudfoundry, cleaning up services and using Rabbit (the default binder)

./run.sh -p cloudfoundry

To setup a Data Flow Server and services on CloudFoundry, but not run tests

./run.sh -p cloudfoundry -c -t

= General configuration

Make sure you have JAVA_HOME configured correctly in your environment.

Each platform will have a file named env.properties located on init/env.properties, change those to reflect your
environment. Each platform has different flags, but the global ones should be:

RETRIES : Number of times to test for a port when checking a service status (6 by default)
WAIT_TIME: How long to wait for another port test (5s by default)
SPRING_CLOUD_DATAFLOW_SERVER_DOWNLOAD_URL: Location of the dataflow jar file to be downloaded.

= Required environment variables when skipping setup

If you want to point to an already running Data Flow server, set the environment variables

SERVER_URI - default is http://localhost:9393
STREAM_REGISTRATION_RESOURCE - default is maven + rabbit based apps
TASK_REGISTRATION_RESOURCE - default is maven + rabbit based tasks

To point to where your server is located and also specify which artifacts you want to register with the server.

= Running Scheduling Acceptance Tests

By default scheduling acceptance tests are disabled, because not all data flow implementations support it yet.
To enable scheduling acceptance tests use the -se flag.
Also you will need to set the scheduler URL if the platform is cloudfoundry. For example export SCHEDULES_URL=<your scheduler url>.

NOTE: Currently Spring Cloud Data Flow Cloud Foundry and Kubernetes are the only implementations that support scheduling.

= Platform specific notes

== Local

=== Pre-requisites

docker and docker-compose installed. Make sure you can connect to the docker daemon without using 'sudo', e.g. docker info works.
$DOCKER_SERVER environment variable properly set. Defaults to localhost, which works on unix. For MacOS 192.168.99.100 should work.

The local deployment will always try to connect to a service running on your local machine. So if you have
a local redis we will use it.

If a local service is not found, the script will try to deploy using docker-compose so it's important that
you have that installed and configured properly.

When cleaning up, the script will only remove docker images, if you are using a local service like redis or mysql
the script will not do anything to it

== CloudFoundry

=== Pre-requisites
On Cloudfoundry, make sure you have the following environment variables exported. We will not include them on any files
to prevent it to be leaked into github repos with credentials.

SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_URL
SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_DOMAIN
SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_USERNAME
SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_PASSWORD
SPRING_APPLICATION_JSON with the following properties:
** maven.remote-repositories.repo1
** spring.cloud.dataflow.task.platform
** Cloud Foundry connection.url
** Cloud Foundry org
** Cloud Foundry space
** Cloud Foundry domain
** Cloud Foundry username
** Cloud Foundry password
** Cloud Foundry skipSSLValidation
** Cloud Foundry deployment.memory
** Cloud Foundry deployment.disk
** Cloud Foundry services
** Cloud Foundry deleteRoutes
** For example:

SPRING_APPLICATION_JSON="'{ \"maven\": { \"remote-repositories\": { \"repo1\": { \"url\": \"https://repo.spring.io/libs-snapshot\" } } }, \"spring\": { \"cloud\": { \"dataflow\": { \"task\": { \"platform\": { \"cloudfoundry\": { \"accounts\": { \"default\": { \"connection\": { \"url\": \"https://mycfurl\", \"org\": \"myorg\", \"space\": \"myspace\", \"domain\": \"apps.mydomain.springapps.io\", \"username\": \"admin\", \"password\": \"mypass\", \"skipSslValidation\": true }, \"deployment\": { \"memory\": \"2048m\", \"disk\": \"2048m\", \"services\": \"mysql,atscheduler\", \"deleteRoutes\": false } } } } } } } } } }'"

=== Configuration

You can override service names and plans by either exporting or changing the following properties:

MYSQL_SERVICE_NAME
MYSQL_PLAN_NAME
RABBIT_SERVICE_NAME
RABBIT_PLAN_NAME
REDIS_SERVICE_NAME
REDIS_PLAN_NAME

The creation and deletion of services are implemented as blocking functions, i.e. a test job will wait, for instance,
during setup until a service is created before continuing. After requesting CloudFoundry to create or delete a service, these functions
periodically poll until the request has been fully met. The defaults for the number of polls and the delay between
polling can be overridden using the following properties:

SCDFAT_RETRY_MAX (default 100, set to <0 for no max)
SCDFAT_RETRY_SLEEP (in seconds, default 5)

== Kubernetes (vSphere)

=== Pre-requisites

The kubectl command line tool needs to be installed. Installation information can be found at: https://kubernetes.io/docs/tasks/tools/install-kubectl/

=== Configuration

The following environment variables must be set:

KUBECONFIG - the path to the kube config file to use
CLUSTER_NAME - the name of the cluster to target (must be present in KUBECONFIG)

NOTE: You can also set a KUBERNETES_NAMESPACE environment variable that specifies an existing namespace to use for the testing. If this is not specified, the 'default' namespace will be used.

=== Code formatting guidelines

The directory /etc/eclipse has two files for use with code formatting, eclipse-code-formatter.xml for the majority of the code formatting rules and eclipse.importorder to order the import statements.
In eclipse you import these files by navigating Windows -> Preferences and then the menu items Preferences > Java > Code Style > Formatter and Preferences > Java > Code Style > Organize Imports respectfully.
In IntelliJ, install the plugin Eclipse Code Formatter.
You can find it by searching the "Browse Repositories" under the plugin option within IntelliJ (Once installed you will need to reboot Intellij for it to take effect).
Then navigate to Intellij IDEA > Preferences and select the Eclipse Code Formatter.
Select the eclipse-code-formatter.xml file for the field Eclipse Java Formatter config file and the file eclipse.importorder for the field Import order.
Enable the Eclipse code formatter by clicking Use the Eclipse code formatter then click the OK button.

** NOTE: If you configure the Eclipse Code Formatter from File > Other Settings > Default Settings it will set this policy across all of your Intellij projects.

ilayaperumalg/spring-cloud-dataflow-acceptance-tests

Languages

Contributors