README.md

Plutus/Marlowe Deployment

staging environment deployment

alpha environment deployment

production environment deployment

Overview

The deployment uses a combination of Terraform and Morph where Terraform creates the instances and network configuration on AWS and Morph deploys and activates the NixOS configurations.

• terraform configuration files
• morph deployment files
• NixOS modules

Continuous Delivery

The server configurations that will be deployed via morph are built by hydra and are deployed by buildkite pipelines (see pipeline.yml). Two pipelines are set up for this purpose:

• master: Successful master branch builds are automatically deployed to the staging environment via the master pipeline
• production: Production deployments are triggered by the production deployment pipeline on buildkite whenever the production branch changes.

Environments

There are several environments to deploy to. All available environments are listed in deployment/envs.nix:

# deployment/envs.nix (excerpt)
{
  alpha = { region = "eu-west-2"; };
  production = { region = "eu-west-1"; };
}

Environments are deployed to <env>.iohkdev.io.

Deployment Versions

alpha is the staging environment to which the master branch is deployed automatically. The production environment is reserved for the live environment. Additional environments are available for testing purposes.

Ideally the alpha deployment should always reflect the current state of master, and production should always reflect the current state of the production branch. This can be verified using the /version endpoint:

$ curl https://alpha.plutus.iohkdev.io/version
{"rev": "13342f6981faabdc2bb7e88a9cb5a3990f7a4930"}

$ curl https://production.plutus.iohkdev.io/version
{"rev": "acc7a4486d50844690fb485b74abab44908bd39b"}

Usage

NOTE: Deployments to either the staging or the production environment should never be done manually unless there is a good reason to do so. Instead, the buildkite pipelines should be used for this.

Prerequisites

• AWS account
• Active plutus nix-shell environment

In order to to deploy anything you need to log in to your AWS account:

$ eval $(aws-mfa-login <username> <mfa-code>)

The deployment scripts will validate the login status and abort if no valid session can be found.

Commands

The following deployment commands are made available through a nix-shell:

• provision-infra provisions the infrastructure on AWS using terraform.
• destroy-infra deletes previously provisioned infrastructure on AWS using terraform.
• deploy-nix deploys nixos configurations to Terraform provisioned servers using morph.
• deploy runs both, provision-infra followed by deploy-nix.

provision-infra

$ nix-shell -A <env> --run "provision-infra"

• The provision-infra command executes terraform apply updating AWS to be in sync with the current state of configuration.
• Running provision-infra may destroy and/or create several, all, or no resources at all. Execution times will differ respectively.

destroy-infra

$ nix-shell -A <env> --run "destroy-infra"

• The destroy-infra command executes terraform destroy and destroys all resources previously created by terraform.

deploy-nix

$ nix-shell -A <env> [--argstr rev <git commit>] --run "deploy-nix"

• The deploy-nix command executes morph deploy to copy and activate the most recent nix packages
• If the environment infrastructure is not up to date (meaning terraform apply would not be a no-op) the deployment will abort.

deploy

$ nix-shell -A <env> [--argstr rev <git commit>] --run "deploy"

• The deploy command combines provision-infra and deploy-nix: It performs terraform apply followed by morph deploy.
• The rev argument is optional and defaults to dev when not specified. The value of rev is returned by the /version endpoint as explained above.

Secrets

The deployment depends on several credentials which are maintained using the AWS Secrets Manager. The secrets are organized per environment with the following structure:

{
  "env": "<name of the environment>",
  "marlowe": {
    "githubClientId": "<id>",
    "githubClientSecret": "<secret>",
    "jwtSignature": "<jwt-token>"
  },
  "plutus": {
    "githubClientId": "<id>",
    "githubClientSecret": "<secret>",
    "jwtSignature": "<jwt-token>"
  }
}

The deployment scripts will obtain this json document for the respective environment and expose them to terraform through several environment variables.

Maintenance

The sections below describe actions relevant for advanced usage or maintenance of the deployment process.

Adding Users

Adding new users that are able to perform deployments requires 2 individual steps:

1. Creating an AWS account for the new user
2. Adding the user's ssh key

The AWS login is required in order to provision infrastructure using Terraform. The ssh key has to be added in order to enable users to perform deployments with morph through the ssh jump host.

AWS

The new user has to be added to the appropriate AWS organization. Please talk to a plutus/marlowe team member and request access. New users with appropriate permissions have to be added manually through the aws console.

Terraform

In order to perform nix deployments a ssh-key has to be configured in deployment/terraform/locals.tf:

1. Create a new ssh keypair:

$ ssh-keygen -t ed25519

2. Add the user/key to the ssh_keys map in deployment/terraform/locals.tf

ssh_keys = {
    username = "ssh-ed25519 AAAAC...f3JfmL3A2 usernamer@host
}

3. Add the new user to environments that they should be able to deploy to

In order to allow the user (ssh key) to deploy to the testing environment deployment/terraform/locals.tf needs to be edited as shown below:

  bastion_ssh_keys_ks = {
    testing = ["username"]
    ...
  }

  root_ssh_keys_ks = {
    testing = ["username"]
    ..
  }

Adding Environments

Deployments can be performed to different environments. Each environment is a full aws setup with multiple ec2 instances and networking, deployed to different iohkdev.io subdomains:

• The alpha environment is deployed to alpha.iohkdev.io
• testing is deployed to testing.iohkdev.io

Terraform uses different workspaces for each environment which are also separated in the shared state which is stored in a S3 bucket. When entering a nix-shell the respective terraform workspace is chosen automatically.

In order to add a new environment the following steps need to be followed:

1. Add the environment to deployment/envs.nix

In order to add an environment environment it needs to be added to the attribute set in deployment/envs.nix as follows:

{
    environment = { region = "eu-west-3"; };
}

2. Add users that can deploy to the environment:

Make sure that users that should be able to deploy to the new environment are added to it in deployment/terraform/locals.tf as described above in section about adding users.

3. Configure credentials for the environment:

In order for the deployment to work it requires access to the secrets described above in the Secrets section. The secrets are obtained from the AWS Secrets Manager but they need to be imported first for every environment:

First create a json file containing the necessary credentials in the deployment directory:

{
  "env": "<name of the environment>",
  "marlowe": {
    "githubClientId": "<id>",
    "githubClientSecret": "<secret>",
    "jwtSignature": "<jwt-token>"
  },
  "plutus": {
    "githubClientId": "<id>",
    "githubClientSecret": "<secret>",
    "jwtSignature": "<jwt-token>"
  }
}

Then use aws-upload-secrets to submit it:

$ nix-shell aws-upload-secrets.nix --argstr env <name> --run "aws-upload-secrets ./file.json"

You should now be able to acess the nix-shell for the newly created environment in which the credentials you just uploaded should be set in several environment variables:

$ nix-shell -A <name>
$ echo $TF_VAR_plutus_github_client_id # should print the value you just configured

Extending The Deployment

The deployment process is split between provisioning the infrastructure on AWS using Terraform and deploying NixOS configurations with updated packages or service descriptions using morph. Depending on the respective changes, either one or both of these layers have to be updated.

Adding Servers

The currently configured ec2 instances are easy to discover:

• morph: deployment/morph/machines.nix
• Terraform: deployment/terraform/machines.tf

The terraform file represents a local resource which is consumed by the morph configuration to obtain information that only terraform can provide. Both files represent the respective entry point to configuring a new server. On the Terraform side the ec2 instance hardware, network and SSL certificates have to be configured. On the morph side there has to be a NixOS configuration describing the software setup.

Adding Services

Assuming you only want to add a service to an existing server, you can follow these steps:

• Expose relevant packages in default.nix
• Create a NixOS module describing your service in nix/modules

Configuring A New Domain

Configuring a newly purchased domain for use with a deployment environment requires several changes, most of them to the Terraform code:

1. Hosted Zone Configuration: Add a new hosted zone in the route53 configuration on the aws console
2. Update NS Entries: Configure the new domain (externally) to use the name servers of the hosted zone
3. ALB Configuration: Configure the routing for the new domain in loadbalancing.tf
4. Configure Certificates: Configure certificates for the new domain in certificates.tf

The changes in this PR can be used as reference.