06/14: cdn: Add an S3 bucket to hold Terraform state.

[Chris Marusich]

marusich pushed a commit to branch master
in repository maintenance.

commit 03cff9ec02dd91d21320dea466139cbe18e829e2
Author: Chris Marusich <address@hidden>
Date:   Fri Dec 28 02:20:28 2018 -0800

    cdn: Add an S3 bucket to hold Terraform state.
    
    * cdn/terraform/main.tf (guix-terraform-state): New bucket.
    * cdn/README.org: Add more documentation.
---
 cdn/README.org        | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 cdn/terraform/main.tf | 42 ++++++++++++++++++++++
 2 files changed, 139 insertions(+), 1 deletion(-)

diff --git a/cdn/README.org b/cdn/README.org
index a51b396..bed9562 100644
--- a/cdn/README.org
+++ b/cdn/README.org
@@ -770,8 +770,37 @@ information (but not anything else).  For example, we 
could create a
 group called "accountants" that contains users who need access to view
 billing information (but nothing else).
 
+** Get invoice as PDF via email
+To further reduce the need to run JavaScript just to view your
+invoice, follow these instructions to have the invoice mailed to you
+as a PDF file:
+
+https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/emailed-invoice.html
+
+I have done this for our account.  It seems this cannot be configured
+using Terraform or any existing AWS API.
+** Enable cost explorer
+This is Cost Explorer:
+https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/ce-what-is.html
+
+To ensure I have the ability to drill into the costs we might incur,
+I've enabled this for now.  It isn't possible to enable it (yet) with
+Terraform configuration.
+
+Enabling it adds no additional cost to our bill, even if we choose to
+use the Cost Explorer tool in the AWS Management Console.  However, if
+we choose to use the Cost Explorer APIs, those do cost extra.
+** Enable "Receive billing alerts"
+It is necessary to enable "receive billing alerts" in order to create
+alarms on billing-related metrics.  I've manually enabled this for our
+account.  It isn't possible (yet?) to configure this in Terraform
+configuration.  For details, see:
+
+https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html
 * Process
 
+** General
+
 Initial, one-time setup:
 
 - terraform init: to set things up and install the AWS provider if you
@@ -794,6 +823,47 @@ doesn't exit immediately with a message saying there are 
no proposed
 changes), and eventually the distribution should arrive at the desired
 end state.
 
+** Backend-specific
+
+Terraform stores state.  Since this AWS Account is administered by
+more than one person using more than one computer, it makes sense to
+share state.  The most sensible way to do this in Terraform is to use
+a so-called "backend" that enables everyone to coordinate even when
+using different computers.  For details, see:
+
+https://www.terraform.io/docs/state/index.html
+https://www.terraform.io/docs/state/remote.html
+https://www.terraform.io/docs/backends/config.html
+
+Depending on the backend, it can also provide locking.  The docs say this:
+
+https://www.terraform.io/docs/state/locking.html
+"If supported by your backend, Terraform will lock your state for all
+operations that could write state. This prevents others from acquiring
+the lock and potentially corrupting your state."
+
+Presumably, this also means that the locking will prevent two people
+from mutating resources (e.g., IAM group membership) at the same time.
+
+This section here will document how I set up the remote backend for
+S3.  S3 is an appropriate storage location since using it in this way
+is not SaaSS (it's only storing and publishing data), and we don't
+already have an installation of something else that can do the job
+(e.g., Consul).  To support locking, we would also need to use
+DynamoDB, which again is not SaaSS in this case (it's only storing and
+publishing data).
+
+https://www.terraform.io/docs/backends/types/s3.html
+
+So, let's get started!  Let's set up S3 without DynamoDB first.
+
+- Back up the state.
+- modify the main.tf file
+- Run "terraform init"
+- Ignore .terraform directory from version control (already done!)
+
+
+
 * Configuration strucure
 There can be multiple files (*.tf, *.tfvars), or just one file.  Name
 doesn't matter, as long as it ends in .tf or .tfvars.  We could
@@ -896,7 +966,6 @@ signatures.
 * Next steps
 Currently, we have all the IAM configuration in Terraform config.  That's 
great!
 
-- Define the CloudFront distribution (don't worry about ACM at first).
 - Integrate the CloudFront distribution with ACM.
 - Figure out how to share the state.  Maybe use the S3 backend?
 - Package Terraform
@@ -905,6 +974,26 @@ Currently, we have all the IAM configuration in Terraform 
config.  That's great!
 - Use origin failover to server requests via the CDN from berlin
   first, and hydra second?
   
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/high_availability_origin_failover.html
+- Set a billing alarm (or perhaps a "budget"): 
https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-what-is.html
+
+** Setting up a budget
+I haven't actually set up a "budget".  But this guide explains how to
+do so with Terraform:
+
+https://blog.kylegalbraith.com/2018/10/08/how-to-better-watch-your-aws-costs-before-you-forget/
+
+Summary: set up some CloudWatch alarms in Terraform, set up a "budget"
+in Terraform, and manually configure an alarm on the budget.  Manually
+configuring the alarm on the budget is unfortunately necessary, since
+Terraform doesn't support it yet:
+
+https://github.com/terraform-providers/terraform-provider-aws/issues/4548
+
+That said, it isn't clear to me why we would need a "budget" right
+now.  Having one or two alarms on estimated and actual total cost
+seems just as good, and we CAN do that in Terraform today.  If we need
+to dive into the cost, there is always the invoice PDF or the AWS
+Management Console for ad-hoc investigation.
 
 * Questions
 ** Guix build farm (berlin)
@@ -936,6 +1025,13 @@ Currently, we have all the IAM configuration in Terraform 
config.  That's great!
 - Do we need a "default root object"?  Probably not, but try making a
   request to the distribution, and see what happens:
   
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DefaultRootObject.html
+** Terraform
+- What does Terraform store in its "state"?  Does it store anything
+  sensitive, like secrets?
+** General
+- How to send notifications for budget alarms?  Email?  To where?  For
+  now, during the test, I suppose I should just send it to my own
+  personal email address.
 * Avoiding "Service as a Software Substitute"
 Dave made an awesome Guile module for using CloudFormation:
 
diff --git a/cdn/terraform/main.tf b/cdn/terraform/main.tf
index dede45e..2ea6516 100644
--- a/cdn/terraform/main.tf
+++ b/cdn/terraform/main.tf
@@ -243,3 +243,45 @@ output "berlin-mirror-status" {
 output "berlin-mirror-domain-name" {
   value = "${aws_cloudfront_distribution.berlin-mirror.domain_name}"
 }
+
+# S3
+
+# DO NOT DELETE THIS BUCKET!  If you think you need to delete this
+# bucket, think twice.  This bucket contains the Terraform state,
+# shared by all Terraform users in the Guix project.  Only delete the
+# bucket if you're sure it's OK to delete that state!
+resource "aws_s3_bucket" "guix-terraform-state" {
+  bucket = "guix-terraform-state"
+  # Access should be granted via IAM policies.
+  acl = "private"
+  # This allows us to recover state if something ever goes wrong.  The
+  # cost of storing all versions of the state is negligible for the
+  # foreseeable future, but if it ever becomes a concern, we can
+  # (1) stop using the S3 backend, (2) suspend versioning, (3) delete
+  # old versions manually, or (4) add a "lifecycle policy" to expire
+  # non-current versions.  For details, see:
+  # https://docs.aws.amazon.com/AmazonS3/latest/dev/DeletingObjectVersions.html
+  versioning {
+    enabled = true
+  }
+  # When we destroy the bucket, destroy all objects first so that the
+  # bucket deletion succeeds.  Of course, you should think twice
+  # before deleting this bucket!
+  force_destroy = true
+  # The best region!  :-) This could be any region, really, but since
+  # I'm the primary user right now, I'm choosing the one that's
+  # closest to me.
+  region = "us-west-2"
+  # Encrypt data at rest using S3's server side encryption.  See:
+  # https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
+  server_side_encryption_configuration {
+    rule {
+      apply_server_side_encryption_by_default {
+        sse_algorithm = "AES256"
+      }
+    }
+  }
+  lifecycle {
+    prevent_destroy = true
+  }
+}