[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
01/01: cdn: Update README.org.
From: |
Chris Marusich |
Subject: |
01/01: cdn: Update README.org. |
Date: |
Fri, 4 Jan 2019 04:14:36 -0500 (EST) |
marusich pushed a commit to branch master
in repository maintenance.
commit 03121abf87d75b5a17737e41a137dfd437492f37
Author: Chris Marusich <address@hidden>
Date: Fri Jan 4 01:13:35 2019 -0800
cdn: Update README.org.
* cdn/README.org: Add an overview and clean up some sections.
---
cdn/README.org | 235 +++++++++++++++++++++++++++++++++------------------------
1 file changed, 137 insertions(+), 98 deletions(-)
diff --git a/cdn/README.org b/cdn/README.org
index eee1371..8d35bd9 100644
--- a/cdn/README.org
+++ b/cdn/README.org
@@ -1,3 +1,131 @@
+* Overview
+The Guix project maintains an [[https://aws.amazon.com/][AWS]] account to
provide a [[https://en.wikipedia.org/wiki/Content_delivery_network][CDN]] for
the
+berlin build farm. Our CDN is built using
[[https://aws.amazon.com/cloudfront/][Amazon CloudFront]].
+
+We practice
[[https://en.wikipedia.org/wiki/Infrastructure_as_code]["Infrastructure as
Code"]]. Nearly all aspects of our AWS
+account are managed via [[https://www.terraform.io/][Terraform]]. The
"terraform" directory contains
+the Terraform configuration files that define our CDN's
+infrastructure. The few aspects of our AWS account which were not
+configured via Terraform are called out clearly either here in this
+README.org file or in the Terraform configuration files.
+
+If you're new to Terraform, consider reading Terraform's official
+[[https://learn.hashicorp.com/terraform/#getting-started]["Getting Started"]]
guide first to get the basic idea. After that, read
+the "How to Use Terraform" section below to see how we use Terraform
+in practice. Once you're familiar with Terraform, hopefully you'll be
+able to understand the terraform/main.tf file. This file is the
+starting point of our infrastructure configuration. It's well
+documented. Try reading it and following the links if you need more
+information.
+
+If you're new to AWS, that's not a problem. There's a ton of
+information available about AWS and its web services, so the best way
+to learn about them is to look up information as you need it. If you
+try to understand all of AWS up front - even just the services that
+Guix uses to build its CDN - it might be frustrating and unproductive.
+
+By practicing Infrastructure as Code using Terraform and AWS, anyone
+can easily create a similar CDN in their own AWS account by following
+the instructions here. If you are doing that, you will probably need
+to customize a few things (e.g., you will probably have a different
+list of administrators), but it should be far easier to set things up
+by re-using this configuration than by starting from nothing.
+
+* How to Use Terraform
+** Prerequisites
+
+To get started with managing Guix's AWS infrastructure using
+Terraform, you first need to do the following:
+
+- Have a Guix project administrator create an IAM user for you and
+ provide you with (1) the user's access keys for programmatic access
+ to AWS and (2) the user's login password if you need to log into the
+ AWS Management Console.
+- Store the access keys in ~/.aws/credentials (or somewhere else if
+ you prefer a non-default location). See the comments in
+ terraform/main.tf for details about this, including an example.
+- Install Terraform. Currently, it isn't packaged by Guix, but you
+ can download a pre-built, stand-alone release binary from the
+ Terraform website, or you can build it yourself manually.
+
+Once you've done all those things, you can try running Terraform.
+
+** Running Terraform
+
+Generally speaking, you always run the terraform command from the
+directory containing the terraform configuration (e.g., main.tf). You
+also need to set the AWS_PROFILE environment variable (see the
+comments in main.tf for details). In the commands below, we've
+omitted the AWS_PROFILE environment variable for brevity.
+
+First, you must perform this initial one-time setup:
+
+- "terraform init": to set things up and install the AWS provider if
+ you don't have it already.
+
+After that, the most common commands you will run will be:
+
+- "terraform plan": to show the actions Terraform would take, without
+ storing a new state in the backend.
+- "terraform apply": to show the actions Terraform will take, and then
+ take them if you say "yes" at the prompt. Stores a new state in the
+ backend when you say "yes".
+- "terraform show": to display information about the state. In
+ particular, this prints out information such as the output from the
+ last run, which can be useful.
+
+A "backend" is the place where Terraform stores its state, which
+describes the current state of the infrastructure under management.
+We use the S3 backend with locking enabled, so that multiple project
+administrators can simultaneously manage the infrastructure from
+multiple computers without stepping on each others' toes. See the
+next section, titled "Notes about Terraform's S3 Backend," for more
+information.
+
+You may notice that when creating or updating a CloudFront
+distribution, "terraform apply" will finish quickly. However, it
+seems this is an asynchronous operation, so the distribution may not
+return to the "Deployed" state for many minutes. To check on its
+progress, you can simply run "terraform apply" repeatedly (maybe
+saying "no" at the prompt if it doesn't exit immediately with a
+message saying there are no proposed changes), and eventually the
+distribution should arrive at the desired end state.
+
+** Notes about Terraform's S3 Backend
+
+Terraform stores state. Since this AWS Account is administered by
+more than one person using more than one computer, it makes sense to
+share state. The most sensible way to do this in Terraform is to use
+a so-called "backend" that enables everyone to coordinate even when
+using different computers. For details, see:
+
+https://www.terraform.io/docs/state/index.html
+https://www.terraform.io/docs/state/remote.html
+https://www.terraform.io/docs/backends/config.html
+
+Depending on the backend, it can also provide locking. The docs say
+this:
+
+https://www.terraform.io/docs/state/locking.html
+"If supported by your backend, Terraform will lock your state for all
+operations that could write state. This prevents others from acquiring
+the lock and potentially corrupting your state."
+
+Presumably, this also means that the locking will prevent two people
+from mutating resources (e.g., IAM group membership) at the same time.
+
+We use the S3 backend:
+
+https://www.terraform.io/docs/backends/types/s3.html
+
+For details on how it's configured, see main.tf.
+
+S3 is an appropriate storage location since using it in this way is
+not SaaSS (it's only storing and publishing data), and we don't
+already have an installation of something else that can do the job
+(e.g., Consul). To support locking, we also use DynamoDB, which again
+is not SaaSS in this case (it's only storing and publishing data).
+
* Software Environment
The following software versions were used, on a GuixSD system:
@@ -27,6 +155,9 @@ Some possibly relevant IceCat about:config details:
network.http.referer.spoofSource;false
privacy.resistFingerprinting;false
+Terraform version (not managed by Guix): v0.11.11
+Terraform AWS Provider version (not managed by Guix): 1.54.0
+
* Create the AWS Account
Website: https://aws.amazon.com/
@@ -635,11 +766,6 @@ Customers can access the information above through AWS
Offerings, such as the AW
#+END_QUOTE
-* Get Terraform
-No Guix package yet. But it's free software, so that's good.
-
-https://www.terraform.io
-
* Initial manual bootstrap
Create a user named safe-to-delete-admin and attach an IAM policy to
it that lets it do anything. We'll delete this in a little bit.
@@ -797,80 +923,6 @@ account. It isn't possible (yet?) to configure this in
Terraform
configuration. For details, see:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html
-* Process
-
-** General
-
-Initial, one-time setup:
-
-- terraform init: to set things up and install the AWS provider if you
- don't have it already.
-
-After that, do this:
-
-- terraform apply: to show the actions Terraform will take, and then
- take them if you say "yes" at the prompt.
-- terraform show: to display information about the state. In
- particular, this prints out information such as the output from the
- last run, which can be useful.
-
-Example command:
-
- AWS_PROFILE=guix AWS_DEFAULT_REGION=us-east-1 terraform apply
-
-The AWS_PROFILE can be customized to whatever you like, but the
-AWS_DEFAULT_REGION must be us-east-1, since we store Terraform's state
-in a bucket in that region specifically.
-
-When creating or updating a CloudFront distribution, "terraform apply"
-will finish quickly. However, it seems this is an asynchronous
-operation, so the distribution may not return to the "Deployed" state
-for many minutes. To check on its progress, you can simply run
-"terraform apply" repeatedly (maybe saying "no" at the prompt if it
-doesn't exit immediately with a message saying there are no proposed
-changes), and eventually the distribution should arrive at the desired
-end state.
-
-** Backend-specific
-
-Terraform stores state. Since this AWS Account is administered by
-more than one person using more than one computer, it makes sense to
-share state. The most sensible way to do this in Terraform is to use
-a so-called "backend" that enables everyone to coordinate even when
-using different computers. For details, see:
-
-https://www.terraform.io/docs/state/index.html
-https://www.terraform.io/docs/state/remote.html
-https://www.terraform.io/docs/backends/config.html
-
-Depending on the backend, it can also provide locking. The docs say this:
-
-https://www.terraform.io/docs/state/locking.html
-"If supported by your backend, Terraform will lock your state for all
-operations that could write state. This prevents others from acquiring
-the lock and potentially corrupting your state."
-
-Presumably, this also means that the locking will prevent two people
-from mutating resources (e.g., IAM group membership) at the same time.
-
-This section here will document how I set up the remote backend for
-S3. S3 is an appropriate storage location since using it in this way
-is not SaaSS (it's only storing and publishing data), and we don't
-already have an installation of something else that can do the job
-(e.g., Consul). To support locking, we would also need to use
-DynamoDB, which again is not SaaSS in this case (it's only storing and
-publishing data).
-
-https://www.terraform.io/docs/backends/types/s3.html
-
-So, let's get started! Let's set up S3 without DynamoDB first.
-
-- Back up the state.
-- modify the main.tf file
-- Run "terraform init" and say "yes" when it prompts you if you want
- to copy existing state into the new backend.
-- Ignore .terraform directory from version control
-
* Configuration strucure
There can be multiple files (*.tf, *.tfvars), or just one file. Name
doesn't matter, as long as it ends in .tf or .tfvars. We could
@@ -919,22 +971,9 @@ https://www.terraform.io/docs/state/remote.html
a collection of 'modules':
https://registry.terraform.io/
-* Terraform getting started guide
-A good, brief intro to all main concepts.
-https://learn.hashicorp.com/terraform/getting-started/install
-
-This how-to guide is much better for newcomers than trying to read the
-reference documentation (e.g., for the configuration file syntax)
-first.
-
-* Amazon ACM specific resources
-https://www.terraform.io/docs/providers/aws/d/acm_certificate.html
-https://www.terraform.io/docs/providers/aws/r/acm_certificate_validation.html
-* Amazon CloudFront specific resources
-
-https://www.terraform.io/docs/providers/aws/r/cloudfront_distribution.html
* IAM Login URL
Log into this URL with your IAM user name and password:
+
https://354378008360.signin.aws.amazon.com/console
Alternatively, you can use this friendlier URL:
@@ -1025,11 +1064,11 @@ Management Console for ad-hoc investigation.
** Terraform
- What does Terraform store in its "state"? Does it store anything
- sensitive, like secrets?
-** General
-- How to send notifications for budget alarms? Email? To where? For
- now, during the test, I suppose I should just send it to my own
- personal email address.
+ sensitive, like secrets? Depending on the configuration, it might
+ store secrets, and those secrets might be encrypted (e.g., by
+ someone's PGP key) in the state file. This is independent of any
+ server side encryption that might also be happening within the S3
+ bucket, in the case of the S3 backend.
* Avoiding "Service as a Software Substitute"
Dave made an awesome Guile module for using CloudFormation: