This post is the first part of a long running series about running a blog in the cloud using production ready DevOps practices on the cheap. The code in this blog post is the same code I use to run this blog, and it is all available in a public github repository. The automatic CI and CD processes will be ran directly out of this public repository, and you will be able to see all the moving parts of what makes this blog work. Any credentials and secrets will of course not be included ;)

I chose to go with the Google Cloud Platform for this, as it’s the platform that has the least publicity around their services, and they have some services that is difficult to beat. Their Kubernetes offerings is probably the best in the market. And with their release of Anthos, which allows you to manage Kubernetes on different cloud providers, they are making a move to reduce lock-in to a specific platform.

Google is not as mature as the other two big platforms: Azure and AWS. And this is apparent when you look at some of the limitations that around hosting serverless and static sites. Some we will be having to work around as we build up this blog. That said, google probably has the best developer experience of the three major platforms.

Before we begin

Before we get started, there are a few things we need to get set up first.

We will be using Google Cloud so you will need to set up an account there, you can set up an new account and get US$300 of free credit to play with, and configure the CLI to interact with the account.

We will be using Hashicorp’s Terraform to configure any infrastructure, I assume you have rudimentary knowledge of this, but you should be able to follow the guide without it.

Set up the git repository

In this solution, we will be deploying both infrastructure and the application, so we will need to structure the repository to support this. I generally try to do this by placing different components in different sub-folders, as this makes it a lot easier later when we automate the build and deploy process. We have to main components so far, infrastructure and blog, so we will create a folder for each of them. Also create a README.md file for documenting the solution.

This is the structure I’m using:

1
2
3
4
5
6

.
├── blog
│   └── .keep
├── infra
│   └── .keep
└── README.md

Initiate the git repository and commit the structure

1
2
3

git init
git add .
git commit -m "Initial Commit"

.keep is a convention make sure folders are committed to git, we will remove those later when we add files to the sub-folders.

Set up blog

Install Hugo

On macOS installing Hugo is easy using homebrew

In terminal:

1

brew install hugo

For other platforms, the official documentation has all the information needed to get Hugo set up: https://gohugo.io/getting-started/installing/

Generating the web-site scaffold

Hugo has a built in generator that we will ise to generate a scaffold for us that we will use to drive the the generation of the static web-site. It contains the folder structure that Hugo relies on, as well as a few defaults that we will have to update to work with the structure we require for our blog.

To generate the scaffold, go to the root of your git repository and type in this command:

1

hugo new site blog

Once that command completes you should have a structure like this in the blog folder:

1
2
3
4
5
6
7
8

.
├── archetypes
├── content
├── data
├── layouts
├── static
├── themes
└── config.toml

For more information on what the different folders are, and what they are used for, please visit the official Hugo documentation. This post is about how to set up and deploy on google, how to use Hugo can be a series in its own right.

Layout

Hugo comes with a theme engine, and you can either create your own, or download a theme from their theme library. For simplicity I went with downloading a theme that I liked, which I will use as a starting point and modify to fit my taste.

Installing a theme with Hugo is really easy, it’s a matter of downloading the theme into a sub-folder in the themes folder and update the blog/config.toml file with the details of the theme.

I’ve used the hello-friend theme by panr. It’s licensed under MIT, which means we can modify it at will, as long as we retain the license file intact.

The description of the theme has a good guide on how to install it, but for the lazy, I’ve included it here as well.

1

git clone https://github.com/panr/hugo-theme-hello-friend.git themes/hello-friend

Paste the following into blog/config.toml to get the site configured with a theme and defaults for the hello-friend theme:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

baseurl = "/"
languageCode = "en-us"
theme = "hello-friend"
paginate = 5

[params]
  # dir name of your blog content (default is `content/posts`)
  contentTypeName = "posts"
  # "light" or "dark"
  defaultTheme = "dark"
  # if you set this to 0, only submenu trigger will be visible
  showMenuItems = 2
  # Show reading time in minutes for posts
  showReadingTime = false

[languages]
  [languages.en]
    title = "Hello Friend"
    subtitle = "A simple theme for Hugo"
    keywords = ""
    copyright = ""
    menuMore = "Show more"
    writtenBy = "Written by"
    readMore = "Read more"
    readOtherPosts = "Read other posts"
    newerPosts = "Newer posts"
    olderPosts = "Older posts"
    minuteReadingTime = "min read"
    dateFormatSingle = "2006-01-02"
    dateFormatList = "2006-01-02"

    [languages.en.params.logo]
      logoText = "hello friend"
      logoHomeLink = "/"
    # or
    #
    # path = "/img/your-example-logo.svg"
    # alt = "Your example logo alt text"

    [languages.en.menu]
      [[languages.en.menu.main]]
        identifier = "about"
        name = "About"
        url = "/about"
      [[languages.en.menu.main]]
        identifier = "showcase"
        name = "Showcase"
        url = "/showcase"

Make and changes to this as you see fit.

Testing the blog

To test what we have done so far, we open the terminal and go to the blog directory. Once there, start the hugo dev server with the following command:

1

hugo server -D

Once that has started, your new blog should be available at http://localhost:1313, if you keep this running, you will be able to see live changes to the blog as you modify settings and add posts.

Tweaks before we commit changes

First we will remove the .keep file in the blog folder, as we now have files there, and will add .keep files to all the empty directories as well. To help keep the directory structure for the future.

We’ll also be adding a .gitignore file to filter out files and folders we do not want to preserve in git. This can be either be files with sensitive data, or generated files that are considered output of running the hugo static-site generator.

When Hugo runs, it will create two folders with static files that we will later publish to our hosting solution, we will add these folders to the .gitignore file in the blog folder to prevent us from accidentally committing these in the future.

blog/.gitignore should now contain this:

1
2

public/
resources/

Commit and continue

Commit all your changes to git and continue with setting up the Google Cloud project and configure service accounts to deploy the solution.

1
2

git add .
git commit -m "Added hugo blog with hello-friend theme"

The state of the repository should be similar to this: GitHub repository

Configure Google Cloud Project

To complete this part you will need to have the google command line utilities installed: gcloud and gutil. If you haven’t yet, please go install them from the link given in the requirements section.

The first thing we will have to do is to set up a new project in GCP. Identity and access is managed using projects in the Google’s cloud, by creating a new project we can get a fully isolated environment that other users and services don’t have access to by default.

To create a new project you need to be logged into your account

1

gcloud auth login

This will open a browser window and allow you to log into your account. If it’s the fist time you log in, it will also ask you to select a project.

Create the project

Creating the project is quite easy. In fact only one command is required

1

gcloud projects create --name <project-name>

This will create a new project called project-name and generate a unique id for that project, usually the name with a set of numbers as a suffix. Remember this id, as we will use it later.

Alternatively you can set the project directly, this gives you more control, but it has to be globally unique, so it will likely take you a couple of goes before you get it right. I wouldn’t recommend it.

1

gcloud projects create <project-id>

Once the project has been created, we need to set it to the active project in the command line. We do this by calling the config command.

1

gcloud config set project <project-id>

Configure service account

Next up is configuring a service account for deploying the infrastructure. We are using Terraform, and the google cloud provider only supports using a service account to deploy the infrastructure. This is also considered best practice, and makes it a lot simpler when working with automated tools. Calling gcloud auth login opens a browser window, doing that as part of an automated script seems a bit silly. Also very difficult when running on a server that we are not logged into like we would when running the deployment through a CI/CD process.

To create a new service account we are going to use the gcloud command like we did when setting up the project.

1

gcloud iam service-accounts create <account-name> --display-name "<Service account name>"

Now that the service account has been created, we need to set up a key for the service account to authenticate with the Google API. This key will be stored in a file on disk to be referenced as part of the Terraform deployment scripts. Remember to add this to the .gitignore file, so you don’t accidentally upload it to your git repository. Having access to this file means you can make changes to all the things the service account has access to.

1
2

mkdir -p infra/.creds
gcloud iam service-accounts keys create infra/.creds/gcp-sa-key.json --iam-account <account-name>@<project-id>.iam.gserviceaccount.com

The key should be stored in the infra/.creds/gcp-sa-key.json file. This will be referenced by Terraform when we deploy the infrastructure.

The service account is created, but it does not have access to anything. This is part of the security model that Google follows, accounts needs to be given explicit access to the resources the require. We will give it the owner role for the project. This is probably more access than it needs, but it’s a good starting point, and we can reduce it once we got a better idea of what the service account needs to do. To give a role to a service account you crate a three way binding between the SA, a project, and a role.

1

gcloud projects add-iam-policy-binding <project-id> --member serviceAccount:<account-name>@<project-id>.iam.gserviceaccount.com --role roles/owner

The project has been set up and you should have a service account with access to deploy resources.

Infrastructure using Terraform

I’m a big proponent of using code to deploy infrastructure, Infrastructure as Code (IaC). If you have been working with the cloud for a while, it becomes second nature and quite surprising when you have to explain to someone why you have to do it. In short, it gives you many benefits over using a portal to deploy. Having you infrastructure as code allows you deploy the same environment multiple times, quickly, securely, and more importantly, always the same. No room for human error. In addition to this, it has a built in Disaster Recovery component. If something happens that forces you to redeploy the solution somewhere else, it is fast, easy and you can trust that it will deploy the same way. I’ll stop there, there is enough information in the benefits and challenges to IaC that it would warrant it’s own blog post, just keep in mind that in a majority of cases, IaC is probably the right choice.

There are as many different types of IaC as there are programming languages. Most cloud providers have their own flavour, as well as SDKs to allow you to integrate into their platforms using your programming language of choice. One of the key requirements I look for when choosing which approach and language I use is ‘state management’, which allows meta data to be added maintained for all the resources deployed. This gives a really big performance benefit when working in environments with a lot of different resources, as pulling information for each and every one of them is going to be very slow and prone to get rate limited by the cloud provider. Another key thing I look for is support for multiple different cloud providers. Using the same language across multiple providers, cloud and others. This eliminates the provider specific languages, and leaves 3rd party providers like Terraform and Pulumi.

I went with Terraform for this as that is the one I know the best.

Static file hosting and custom domains

Google Cloud Platform allows you to host static files in their storage buckets and make them public, this allows websites to keep their static files like images separate from their dynamic content in a cheaper and more scalable way. We are going to take advantage of this, but take it one step further. Because we do not have any dynamic content, why not host the whole website in the storage bucket? To do that, we need a way to default to a specific page if no specific path has been requested, like when you type http://www.example.com. Luckily Google Cloud allows us to specify this as part of the configuration of the storage bucket. It also allows us to configure our own domain.

To configure your own domain is quite easy, but has a few caveats that you need to know about.

  1. You need to prove that you own the domain, and the service account we use to deploy the bucket needs to have access to this.
  2. You use a CName to redirect to the bucket, this means you can not use http://example.com, but have to use http://www.example.com. The reason for this is the way the RFC for DNS is defined. More details on that can be found in this excellent blog post by Dominic Fraser.

To prove you own the domain you intend to use with your blow, please follow the documentation on the google docs page on this subject.

Once you have verified the domain, make sure you add your service account defined earlier as an owner of the domain. If you add it as a user or administrator, it will not work.

Terraform main file

I won’t go into too much details on how to set up and configure terraform as this is something that has been covered many times before. I will rather go into the things that are different for the Google Cloud Platform.

Authentication

There are two main ways to authenticate with Google Cloud, you can either get an OAuth token using the google cli utilities, or you can download a key file and use that. The first of those two options requires user authentication, which isn’t ideal for automation, so we have gone with the second option in this guide.

When we configure the terraform provider we pass in the path to the configuration file that we created earlier for the service account. In the future we will upload this file to our CI tool to allow it to authenticate as our service account and execute any changes we need.

1
2
3
4
5

provider "google" {
  credentials = "${file(".creds/gcp-sa-key.json")}"
  project     = "${var.google_project}"
  region      = "${local.region}"
}

Resources

There are two resources we need to deploy: the bucket and an acl object to give public users access to the files.

The storage bucket is configured to be located in the us and the name is set to the domain we intend to use for this website. It’s important that this is the full domain, as it is what google uses to direct requests to the correct bucket.

We have also set up two pages that Google will redirect to by default, one for if there no path provided, and one if there is a path provided, but it doesn’t lead to anything, a 404 error.

The ACL contains the default roles, with one additional one READER:allUsers, which makes the bucket public and allows users that have not authenticated to our google account read the files.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

resource "google_storage_bucket" "static-store" {
  name     = "www.winsnes.io"
  location = "US"

  website {
    main_page_suffix = "index.html"
    not_found_page   = "404.html"
  }
}

data "google_project" "project" {}

resource "google_storage_bucket_acl" "static-store-default" {
  bucket = "${google_storage_bucket.static-store.name}"

  role_entity = [
    "OWNER:project-owners-${data.google_project.project.number}",
    "OWNER:user-${var.service_account}",
    "READER:allUsers",
  ]
}

DNS Configuration

Google cloud has made it very easy to configure a custom domain for a storage bucket, all you have to do, is configure a CName to redirect to a specific google domain (c.storage.googleapis.com.), and google takes care of the rest.

Add the DNS entry to your DNS zone and you should be able to access your blog once you upload the content.

Generate and Publish

Now that everything is set up and ready, all that is left is to generate the static content and publish it to the bucket.

To generate the content go to the blog folder and use the generate command for hugo

1

hugo

This commands will create two folders in the blog directory: public and resources. Next we will upload the content of the public folder to the storage bucket using gsutil.

1

gsutil -m cp -r public/* gs://<name of bucket>

In my case I use www.winsnes.io as the bucket name.

Once the command is completed, you should see the content of the blog by going to the registered DNS entry.

Every time you create a new blog post, you need to step through this last process and upload the generated content. This is something that I will automate using CI/CD. But that is something we will look at in the future.