Deployment on AWS
==========================

The pipeline is deployed to ECS, the elastic container service. We use the
`copilot cli <https://aws.github.io/copilot-cli/>`_ to manage this. 

Copilot glues together several services:
- ECS and ECR
- ParameterStore secret credentials
- CodeBuild automatic deployment
- CloudWatch logging

And others, like CloudFormation, that are more behind-the-scenes.

You can find a step-by-step guide to setting up a new copilot application
:doc:`here <copilot-app-creation>`.

ECS Essentials
----------------------

- *environment:* separate network space in which code is deployed
- *service:* persistent deployment in an environment which can run tasks
- *job:* scheduled/rated run in an environment that runs one task
- *task:* Docker image, compute power and command to run in service or job
- *pipeline*: CodeBuild process to automatically provision jobs and tasks

Pipeline on ECS
----------------------

This is how the pipeline is deployed onto ECS.

Environments
#####################

We have two environments, which are identical -- test and prod. This is currently
only for sanity, as they are identical but nonetheless separate: both were created
with the default options through

.. code-block:: bash
   
   copilot env init


Services
#####################

We use services as persistent spaces to test and run code, interactively.

.. code-block:: bash
  
   copilot svc init

Following the defaults here, while specifying a Dockerfile path, gives us a sandbox.

.. code-block:: bash
   
   copilot svc exec --name <name>

This empty ``exec`` command gives you a bash prompt in the service. After exiting the session,
it is recorded in a separate CloudWatch entry.

Jobs
#####################

Jobs are how the pipeline runs on schedule. We have one per normal runtime, each of which amounts to 
some call to ``run.sh``.

Both jobs and services have entries in their ``manifest.yml`` that exposes secrets from Parameter Store
to their runtime. These are credentials for connecting to S3 and our database.

.. code-block:: bash
  
   copilot job init

Pipelines
#####################

We have a single pipeline that exists solely to manage private credentials between different platforms
the IJF needs, especially the problem of cloning a private GitHub repository every time a job runs.

Dev and Prod
----------------------

We have two Dockerfiles. 

``Dockerfile`` is the image we use for development. It builds this documentation
on container build and it runs ``COPY . .``, which merely copies the file in the project directory to the
``/app`` folder in the container.

We then have a group of dockerfiles, currently in project root, named by ``<db>-<s>``. We use these in 
production. It does not build the docs (or install their dependencies), and although it still runs ``COPY``, 
this Dockerfile is only ever called from our CodeBuild object called ``pipeline``. 

This automatic build process was created to support cloning our private GitHub repository 
every run. It is connected to the repository such that any change to the ``main`` branch will reploy
the pipeline automatically. It pulls the ``GITHUB_TOKEN`` secret from Parameter Store, which it uses
to clone; this secret is not available during container runtime, only during the build.

We achieve this process by adding these lines to the ``buildspec.yml`` generated by the CLI:

.. code-block:: yaml
   
  pre_build:
    commands:
        - echo "Cloning private repo..."
        - git clone https://$GITHUB_TOKEN@github.com/InvestigativeJournalismFoundation/pipeline.git
        - cd pipeline
  build:
    commands:
      - echo "Building Docker image..."
      - docker build -t temporary-tag-name -f Dockerfile-prod .
      # returning to cb dir, where copilot is installed
      - cd $CODEBUILD_SRC_DIR

Adding a new job
----------------------

When adding a new job, follow these steps:

1. ``copilot job init``: this will give the ``manifest.yml`` file and some small changes somewhere
   in AWS.
2. ``copilot job deploy``: this will actually deploy the job such that it can be invoked manually and 
   will otherwise run on schedule
3. ``copilot pipeline update``: this will add the job to our CodeBuild pipeline's registry of jobs
4. ``git push``: push to main, triggering a build of the CodeBuild pipeline, which will include the
   new job

Bug report: CloudFormation, StackSets and Drift
------------------------------

.. note::
   Since working on this, we've created a new copilot app. This is good reference for
   how CloudFormation works though.

The copilot CLI glues together several cloud services. One is CloudFormation.

When creating a copilot pipeline, you silently create a CloudFormation StackSet. This is
a grouping of Stacks. However, counter-intuitively, the created StackSet is holds a single 
stack, at least in our current use; perhaps with other features we haven't tried, the set 
would contain multiple Stacks.

This single stack has resources. There are some periphery resources like credentials but also
one for each job or service created in the pipeline, presumably in one environment.

I have encountered issues before with the creation and deletion of jobs and services. In 
particular, trying to delete jobs that refuse to delete from the command line. Repeated
changes like this to the StackSet eventually created a bug preventing the creation of more
jobs. The message was:

``Unable to retrieve Arn attribute for AWS::ECR::Repository, with error message null``

In the CloudFormation console, I "detected drift" to see that the stack set had drifed. The
AWS docs describe some way to see the ways that the set has drifted but I couldn't find it.
The issue was, I already knew, many job definitions in the template YAML file which I tried
to delete from the CLI but which were lingering here. I simply deleted the corresponding lines
from this YAML file.

I also found lingering images in the ECR. I deleted those manually. I'm sure there are more
surviving remants of these jobs, e.g. in CloudWatch or IAM, but I won't hunt around for them
now.