Cloud Security Watch

(csw-backend)
This repository contains the code for creating an
instance of Cloud Security Watch running in an
infrastructure created by alphagov/csw-infra

Note - Cloud Security Watch audits can now be run offline

from the command line. If you want to run CSW audits without
a web infrastructure please read the CLI ReadMe

Create your virtual env

TODO - switch to venv

If you're running Python 2.7 or 3.7 you will need to install 3.6.5

brew install pyenv
pyenv install 3.6.5
virtualenv -p <path to your 3.6.5/bin/python> <virtual env name>

Then you'll need to install the dependencies and dev dependencies
locally

cd chalice
pip install -r requirements-dev.txt

The requirements file is not stored in the repository root
since the requirements need to be packaged by chalice as part
of the chalice deploy

Unit Tests

Now you can run the unit tests.

python -m unittest discover -v

Run the command above before adding/commiting.
If tests fail, don't worry, review the reason for the failures.
It might even be that the test is no longer valid,
in which case you need to update it or delete it.

End to end testing

If you are running the e2e tests for the first time you probably
need to do some installs first.

brew install geckodriver
cd /path/to/csw-backend/chalice 
pip install -r requirements-dev.txt
cd ../build
reinstall
gulp environment.terraform --env=[env]
gulp environment.chalice_s3_deploy --env=[env]

For the e2e tests in addition to your env you need to tell it
which user to assume. This is the name part of your email address.

gulp environment.e2e --env=[env] --user=[firstname.surname]

Create your AWS environment

Prerequisites

Create a Google Cloud Console credentials file.

You will only need to do this in an AWS
account where this has not already been set up.

####Create an SSH key.
The build script uses ssh to tunnel to the RDS instance and create
the database. It also creates a box which can
be used for deploying chalice when working
remotely with less bandwidth.

The simplest approach is probably to match the env name.
The key will be uploaded to AWS so matching the env name
avoids conflict and makes it easy to find.

Like id_rsa, the difference between the
private and public key file names should be:
* private=/path/to/[keyname]
* public=/path/to/[keyname].pub

Install gulp-cli globally

sudo npm install -g gulp-cli

Gulp is a task runner used to run various build tasks.

Install npm-reinstall globally

sudo npm install -g npm-reinstall

Does a complete reinstall of npm dependencies not via
the cache. Important for picking up github code changes
if you're installing a branch rather than a tagged
release and want the current commit on that branch.

AWS vault

If you're running in a non-default AWS account where
access is assumed via a profile you will need to set
up aws-vault so you can assume the profile to run
these commands.

If you're running against the AWS account attached
to your default credentials and don't use aws-vault
you can just run the commands from gulp onwards.

gulp [task]

.. becomes ..

aws-vault exec [profile] -- gulp [task]

From here onwards aws-vault exec [profile] has been left out
but it's assumed that any gulp task should be preceded
with an appropriate aws-vault profile.

Building your environment

All the build tasks are run from the build directory

cd /path/to/csw-backend/build

NPM install installs;

alphagov/govuk-frontend and its dependencies
gulp and some modules for running buid tasks
alphagov/csw-infra to terraform the infrastructure

npm install

Upload shared parameters

Create shared credentials in parameter store.
These are the Google API OAuth credentials and the
name of the S3 bucket used to store the terraform
states.

These are created once in each AWS account and
reused for multiple environments. The assumption is
that the live environment will be the only env
deployed to the live account. So credentials are
shared by test environments but different
credentials are used in production.

If you're creating a new environment in an account
that is already running an existing environment you
can skip this step.

gulp parameters.shared

Environments can be test stages or named for
individual developers. Names should be short,
lower case with no spaces.

environment.build

aws-vault exec [profile] -- gulp environment.build --env=[env name]

The environment.build task is made up of several
sub-tasks.

Especially when building test or developer accounts you
may run into AWS Service Limits. In most cases these
will be triggered in the terraform apply task.

If terraform fails the output from the script should
show the limit which needs increasing and this can be
done through an AWS support request.

environment.settings

You will first be prompted for some settings:

A 16bit mask for your internal IP ranges.
We've used:
- 10.x for developers
- 10.10x for test environments
The name of your ssh key and the path to
the public key. This will create an ssh key
on AWS and upload your public key.

If you've followed the process above the ssh key name
should match your env name. It asks for the path to the
public part of the key /path/to/[keyname].pub

If you put your private key path in this setting you
will need to generate a new ssh key and start again

It will create a settings.json file for your
environment in /path/to/csw-backend/environments/[env]

Make sure you have a .ssh/config file in your home
directory. If it's not there just use touch to create
an empty file.

environment.params

It then generates some random passwords for
RDS and uploads them to AWS parameter store.

environment.tfvars

Then it creates your terraform tfvars files.

environment.terraform

Then it initialises terraform with S3 (For
an existing environment it will retrieve
the existing environment state file)

It runs terraform apply to build your
infrastructure and saves the terraform
outputs to your settings.json file.

The environment.terraform task has sub-tasks
which can all be run independently as necessary

sass.csw

Compiles SASS

copy.assets

Copies assets from govuk-frontend

environment.chalice

Then it creates a chalice config.json and
runs chalice deploy --stage=[env]

The environment.chalice task has sub-tasks which
can all be run independently as necessary

environment.database_build

Finally it creates and bootstraps the
database, creating the tables and populating
the lookup content and checks.

Loading an existing environment

Creates your settings file and tfvars files
Reinitialises terraform from S3
Reads terraform output into the settings file
Creates the chalice config

gulp environment.load --env=[env]

Deploy an existing environment

Compiles SASS and copies assets from govuk-frontend
Runs terraform apply
Runs chalice deploy

gulp environment.deploy --env=[env]

Alternatively you can run ..

gulp environment.terraform_apply --env=[env]

.. and/or ..

gulp environment.chalice_deploy --env=[env]

.. independently.

Delete and existing environment

Runs chalice delete
Runs terraform destroy

gulp environment.cleanup --env=[env]

alphagov/csw-backend