01/01: cdn: Update README.org.

[Chris Marusich]

marusich pushed a commit to branch master
in repository maintenance.

commit 03121abf87d75b5a17737e41a137dfd437492f37
Author: Chris Marusich <address@hidden>
Date:   Fri Jan 4 01:13:35 2019 -0800

    cdn: Update README.org.
    
    * cdn/README.org: Add an overview and clean up some sections.
---
 cdn/README.org | 235 +++++++++++++++++++++++++++++++++------------------------
 1 file changed, 137 insertions(+), 98 deletions(-)

diff --git a/cdn/README.org b/cdn/README.org
index eee1371..8d35bd9 100644
--- a/cdn/README.org
+++ b/cdn/README.org
@@ -1,3 +1,131 @@
+* Overview
+The Guix project maintains an [[https://aws.amazon.com/][AWS]] account to 
provide a [[https://en.wikipedia.org/wiki/Content_delivery_network][CDN]] for 
the
+berlin build farm.  Our CDN is built using 
[[https://aws.amazon.com/cloudfront/][Amazon CloudFront]].
+
+We practice 
[[https://en.wikipedia.org/wiki/Infrastructure_as_code]["Infrastructure as 
Code"]].  Nearly all aspects of our AWS
+account are managed via [[https://www.terraform.io/][Terraform]].  The 
"terraform" directory contains
+the Terraform configuration files that define our CDN's
+infrastructure.  The few aspects of our AWS account which were not
+configured via Terraform are called out clearly either here in this
+README.org file or in the Terraform configuration files.
+
+If you're new to Terraform, consider reading Terraform's official
+[[https://learn.hashicorp.com/terraform/#getting-started]["Getting Started"]] 
guide first to get the basic idea.  After that, read
+the "How to Use Terraform" section below to see how we use Terraform
+in practice.  Once you're familiar with Terraform, hopefully you'll be
+able to understand the terraform/main.tf file.  This file is the
+starting point of our infrastructure configuration.  It's well
+documented.  Try reading it and following the links if you need more
+information.
+
+If you're new to AWS, that's not a problem.  There's a ton of
+information available about AWS and its web services, so the best way
+to learn about them is to look up information as you need it.  If you
+try to understand all of AWS up front - even just the services that
+Guix uses to build its CDN - it might be frustrating and unproductive.
+
+By practicing Infrastructure as Code using Terraform and AWS, anyone
+can easily create a similar CDN in their own AWS account by following
+the instructions here.  If you are doing that, you will probably need
+to customize a few things (e.g., you will probably have a different
+list of administrators), but it should be far easier to set things up
+by re-using this configuration than by starting from nothing.
+
+* How to Use Terraform
+** Prerequisites
+
+To get started with managing Guix's AWS infrastructure using
+Terraform, you first need to do the following:
+
+- Have a Guix project administrator create an IAM user for you and
+  provide you with (1) the user's access keys for programmatic access
+  to AWS and (2) the user's login password if you need to log into the
+  AWS Management Console.
+- Store the access keys in ~/.aws/credentials (or somewhere else if
+  you prefer a non-default location).  See the comments in
+  terraform/main.tf for details about this, including an example.
+- Install Terraform.  Currently, it isn't packaged by Guix, but you
+  can download a pre-built, stand-alone release binary from the
+  Terraform website, or you can build it yourself manually.
+
+Once you've done all those things, you can try running Terraform.
+
+** Running Terraform
+
+Generally speaking, you always run the terraform command from the
+directory containing the terraform configuration (e.g., main.tf).  You
+also need to set the AWS_PROFILE environment variable (see the
+comments in main.tf for details).  In the commands below, we've
+omitted the AWS_PROFILE environment variable for brevity.
+
+First, you must perform this initial one-time setup:
+
+- "terraform init": to set things up and install the AWS provider if
+  you don't have it already.
+
+After that, the most common commands you will run will be:
+
+- "terraform plan": to show the actions Terraform would take, without
+  storing a new state in the backend.
+- "terraform apply": to show the actions Terraform will take, and then
+  take them if you say "yes" at the prompt.  Stores a new state in the
+  backend when you say "yes".
+- "terraform show": to display information about the state.  In
+  particular, this prints out information such as the output from the
+  last run, which can be useful.
+
+A "backend" is the place where Terraform stores its state, which
+describes the current state of the infrastructure under management.
+We use the S3 backend with locking enabled, so that multiple project
+administrators can simultaneously manage the infrastructure from
+multiple computers without stepping on each others' toes.  See the
+next section, titled "Notes about Terraform's S3 Backend," for more
+information.
+
+You may notice that when creating or updating a CloudFront
+distribution, "terraform apply" will finish quickly.  However, it
+seems this is an asynchronous operation, so the distribution may not
+return to the "Deployed" state for many minutes.  To check on its
+progress, you can simply run "terraform apply" repeatedly (maybe
+saying "no" at the prompt if it doesn't exit immediately with a
+message saying there are no proposed changes), and eventually the
+distribution should arrive at the desired end state.
+
+** Notes about Terraform's S3 Backend
+
+Terraform stores state.  Since this AWS Account is administered by
+more than one person using more than one computer, it makes sense to
+share state.  The most sensible way to do this in Terraform is to use
+a so-called "backend" that enables everyone to coordinate even when
+using different computers.  For details, see:
+
+https://www.terraform.io/docs/state/index.html
+https://www.terraform.io/docs/state/remote.html
+https://www.terraform.io/docs/backends/config.html
+
+Depending on the backend, it can also provide locking.  The docs say
+this:
+
+https://www.terraform.io/docs/state/locking.html
+"If supported by your backend, Terraform will lock your state for all
+operations that could write state. This prevents others from acquiring
+the lock and potentially corrupting your state."
+
+Presumably, this also means that the locking will prevent two people
+from mutating resources (e.g., IAM group membership) at the same time.
+
+We use the S3 backend:
+
+https://www.terraform.io/docs/backends/types/s3.html
+
+For details on how it's configured, see main.tf.
+
+S3 is an appropriate storage location since using it in this way is
+not SaaSS (it's only storing and publishing data), and we don't
+already have an installation of something else that can do the job
+(e.g., Consul).  To support locking, we also use DynamoDB, which again
+is not SaaSS in this case (it's only storing and publishing data).
+
 * Software Environment
 
 The following software versions were used, on a GuixSD system:
@@ -27,6 +155,9 @@ Some possibly relevant IceCat about:config details:
 network.http.referer.spoofSource;false
 privacy.resistFingerprinting;false
 
+Terraform version (not managed by Guix): v0.11.11
+Terraform AWS Provider version (not managed by Guix): 1.54.0
+
 * Create the AWS Account
 
 Website: https://aws.amazon.com/
@@ -635,11 +766,6 @@ Customers can access the information above through AWS 
Offerings, such as the AW
 
 #+END_QUOTE
 
-* Get Terraform
-No Guix package yet.  But it's free software, so that's good.
-
-https://www.terraform.io
-
 * Initial manual bootstrap
 Create a user named safe-to-delete-admin and attach an IAM policy to
 it that lets it do anything.  We'll delete this in a little bit.
@@ -797,80 +923,6 @@ account.  It isn't possible (yet?) to configure this in 
Terraform
 configuration.  For details, see:
 
 
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html
-* Process
-
-** General
-
-Initial, one-time setup:
-
-- terraform init: to set things up and install the AWS provider if you
-  don't have it already.
-
-After that, do this:
-
-- terraform apply: to show the actions Terraform will take, and then
-  take them if you say "yes" at the prompt.
-- terraform show: to display information about the state.  In
-  particular, this prints out information such as the output from the
-  last run, which can be useful.
-
-Example command:
-
-  AWS_PROFILE=guix AWS_DEFAULT_REGION=us-east-1 terraform apply
-
-The AWS_PROFILE can be customized to whatever you like, but the
-AWS_DEFAULT_REGION must be us-east-1, since we store Terraform's state
-in a bucket in that region specifically.
-
-When creating or updating a CloudFront distribution, "terraform apply"
-will finish quickly.  However, it seems this is an asynchronous
-operation, so the distribution may not return to the "Deployed" state
-for many minutes.  To check on its progress, you can simply run
-"terraform apply" repeatedly (maybe saying "no" at the prompt if it
-doesn't exit immediately with a message saying there are no proposed
-changes), and eventually the distribution should arrive at the desired
-end state.
-
-** Backend-specific
-
-Terraform stores state.  Since this AWS Account is administered by
-more than one person using more than one computer, it makes sense to
-share state.  The most sensible way to do this in Terraform is to use
-a so-called "backend" that enables everyone to coordinate even when
-using different computers.  For details, see:
-
-https://www.terraform.io/docs/state/index.html
-https://www.terraform.io/docs/state/remote.html
-https://www.terraform.io/docs/backends/config.html
-
-Depending on the backend, it can also provide locking.  The docs say this:
-
-https://www.terraform.io/docs/state/locking.html
-"If supported by your backend, Terraform will lock your state for all
-operations that could write state. This prevents others from acquiring
-the lock and potentially corrupting your state."
-
-Presumably, this also means that the locking will prevent two people
-from mutating resources (e.g., IAM group membership) at the same time.
-
-This section here will document how I set up the remote backend for
-S3.  S3 is an appropriate storage location since using it in this way
-is not SaaSS (it's only storing and publishing data), and we don't
-already have an installation of something else that can do the job
-(e.g., Consul).  To support locking, we would also need to use
-DynamoDB, which again is not SaaSS in this case (it's only storing and
-publishing data).
-
-https://www.terraform.io/docs/backends/types/s3.html
-
-So, let's get started!  Let's set up S3 without DynamoDB first.
-
-- Back up the state.
-- modify the main.tf file
-- Run "terraform init" and say "yes" when it prompts you if you want
-  to copy existing state into the new backend.
-- Ignore .terraform directory from version control
-
 * Configuration strucure
 There can be multiple files (*.tf, *.tfvars), or just one file.  Name
 doesn't matter, as long as it ends in .tf or .tfvars.  We could
@@ -919,22 +971,9 @@ https://www.terraform.io/docs/state/remote.html
 a collection of 'modules':
 https://registry.terraform.io/
 
-* Terraform getting started guide
-A good, brief intro to all main concepts.
-https://learn.hashicorp.com/terraform/getting-started/install
-
-This how-to guide is much better for newcomers than trying to read the
-reference documentation (e.g., for the configuration file syntax)
-first.
-
-* Amazon ACM specific resources
-https://www.terraform.io/docs/providers/aws/d/acm_certificate.html
-https://www.terraform.io/docs/providers/aws/r/acm_certificate_validation.html
-* Amazon CloudFront specific resources
-
-https://www.terraform.io/docs/providers/aws/r/cloudfront_distribution.html
 * IAM Login URL
 Log into this URL with your IAM user name and password:
+
 https://354378008360.signin.aws.amazon.com/console
 
 Alternatively, you can use this friendlier URL:
@@ -1025,11 +1064,11 @@ Management Console for ad-hoc investigation.
 
 ** Terraform
 - What does Terraform store in its "state"?  Does it store anything
-  sensitive, like secrets?
-** General
-- How to send notifications for budget alarms?  Email?  To where?  For
-  now, during the test, I suppose I should just send it to my own
-  personal email address.
+  sensitive, like secrets?  Depending on the configuration, it might
+  store secrets, and those secrets might be encrypted (e.g., by
+  someone's PGP key) in the state file.  This is independent of any
+  server side encryption that might also be happening within the S3
+  bucket, in the case of the S3 backend.
 * Avoiding "Service as a Software Substitute"
 Dave made an awesome Guile module for using CloudFormation: