Adding a new website (Digital Ocean + Docker)

In this post, I documents the steps I take to add a new website to aaronlenoir.com using Digital Ocean and docker.

I think I'd like to upgrade my blogs to blog engine Ghost 2 (I'm using version 1 now).

It's quite the update, so first I want to run a new instance on the side to see how it works.

I will be adding, as a test, blog-test.aaronlenoir.com.

Update: I though this would be a simple ten minute task, but Ghost 2 is giving me problems

Current Setup

At the moment, I'm hosting four websites.

Two Ghost 1 blogs:

  • blog.aaronlenoir.com
  • flightschool.aaronlenoir.com

Two home-made .NET Core applications:

  • tracks.aaronlenoir.com
  • kicker.aaronlenoir.com

All of them run on a single Digital Ocean "droplet" (a Virtual Private Server or VPS).

In front of them is an nginx instance configured as a reverse proxy. Nginx is responsible for redirecting requests to the appropriate application, which are not exposed to the internet directly. Additionally nginx takes care of the SSL certificates.

All 4 applications and nginx run as a docker container. Additionally, I run a sixth docker contain responsible for renewing my SSL certificates for each of the hostnames.

To manage the configuration of each container, I use "docker-compose". The docker-compose configuration can be found on GitHub: https://github.com/AaronLenoir/aaronlenoir.com/blob/a059df908ddbccc6147ea064a2fdf55ae2296669/docker-compose.yml

In summary I run 6 docker containers that do all the work. To add a new website I must add another container and associate it with a new hostname so that nginx can forward the requests correctly.

Step 1: Run Ghost 2 via docker-compose

I log into my VPS using SSH. It's a UNIX system (I know this).

Ghost says they don't offer an "official" docker image to run the blog. But they do point to an "unofficial" one: https://hub.docker.com/_/ghost/.

Interestingly, the docker page says it's the Official image. So maybe the ghost FAQ is a little outdated and it IS official now.

According to the docs, to run a ghost image on the default port I must run the following:

$ docker run -d --name some-ghost ghost

I COULD do that, but I wouldn't be able to test it. So I'm going to immediatly put it in my docker-compose.yml file:

  ghost2:
    image: ghost:2-alpine
    restart: always
    depends_on:
      - "nginx-proxy"
    ports:
      - 127.0.0.1:8120:2368
    volumes:
      - ./blog2/data/ghost:/var/lib/ghost/content
    environment:
      - url=https://blog-test.aaronlenoir.com
      - VIRTUAL_HOST=blog-test.aaronlenoir.com
      - LETSENCRYPT_HOST=blog-test.aaronlenoir.com
      - LETSENCRYPT_EMAIL=info@aaronlenoir.com

ghost2 is the name of the container.

image indicates I wish to use the latest 2.x version. I use alpine because of the lower footprint and it worked well with version 1.

restart is set to always that way I'm sure if it's stopped correctly via docker that it gets restarted. I can still stop it using docker-compose.

depends-on specifies the nginx-proxy must be running before this container can start up.

ports means internally ghost runs on port 2368 (the default) but the container should listen on port 8120 externally.

volumes points to the location on my server where all the files should be (theme, images hosted on the blog, logs, database, ...). The first part is the location on my server, the second part is the location inside the container to which it should map. I'm assuming ghost 2 still expects the data in /var/lib/ghost/content

environment sets a number of environment variables in the container:

  • url the URL to use for the blog - may not be necessary for ghost 2
  • VIRTUAL_HOST is used for the nginx-proxy to know that requests for that hostname must go to this container
  • LETSENCRYPT_HOST tells the let's encrypt container the hostname for which it must fetch a certificate
  • LETSENCRYPT_EMAIL is passed to Let's Encrypt when I'm asking a new certificate, they use that to send me mails about the certificate expiration (but I renew automatically - normally)

After adding this to my docker-compose.yml file I restart everything. Which results in the download of the image:

Creating network "aaronlenoircom_default" with the default driver
Pulling ghost2 (ghost:2-alpine)...
2-alpine: Pulling from library/ghost
e7c96db7181b: Pull complete
50958466d97a: Pull complete
56174ae7ed1d: Pull complete
284842a36c0d: Pull complete
237455e2fb15: Pull complete
e9505cbbbd44: Pull complete
711e6ff570b7: Extracting [====================================>              ]  4.522MB/6.145MBownload complete
 65.56MB/68.44MBwnload complete
    548B/548B

After that the following message indicates at least something started:

Creating aaronlenoircom_ghost2_1               ... done

Step 2: Testing

To test, I usually tell my local machine to resolve the test hostname to the IP address of my VPS.

In windows, that's done by editing the file C:\Windows\Systems32\drivers\etc\hosts

I add the following entry:

82.196.2.207	blog-test.aaronlenoir.com

With that I entered the url in my browser. This immediatly gives me an error because I don't yet have an SSL certificate. This is normal because Let's Encrypt won't yet know of the hostname.

Of course I can tell Firefox to "accept the risk and continue", because I know what this site is.

Sadly, I'm greeted by a server error:

500 Internal Server Error
nginx/1.17.3

Step 2.1: Troubleshooting

TL;DR This troubleshooting session goes through a lot of useless stuff that eventually turned out not to be important. You could skip to Step 3 ...

Since the error is coming from nginx, I'm assuming the nginx proxy server wasn't able to forward my request. I presume this is because the ghost container couldn't start correctly.

Docker-Compose logs

To check this, I usually run docker-compose in interactive mode, so that I can see what happens in each of the containers:

sudo docker-compose up

Strangely it looks like ghost 2 is running. This was the only logging I could find at start-up:

ghost2_1                | [2019-09-28 22:12:11] INFO Ghost is running in production...
ghost2_1                | [2019-09-28 22:12:11] INFO Your site is now available on https://blog-test.aaronlenoir.com/
ghost2_1                | [2019-09-28 22:12:11] INFO Ctrl+C to shut down
ghost2_1                | [2019-09-28 22:12:11] INFO Ghost boot 25.093s

When I visit the site, the logging tells me, not as much as I was hoping for:

nginx-proxy_1           | nginx.1    | blog-test.aaronlenoir.com 84.192.158.84 - - [28/Sep/2019:22:15:11 +0000] "GET / HTTP/2.0" 500 177 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"

Ghost Logs

I did remember pointing to a data directory that doesn't yet exist: blog2/data/ghost

It was created and it did generate some logs in the folder: /blog2/data/ghost/logs

In the logs, there are no errors and I can see it created the database correctly:

{"name":"Log","hostname":"ec4432e19f48","pid":1,"level":30,"msg":"Ghost is running in production...","time":"2019-09-28T22:04:14.110Z","v":0}
{"name":"Log","hostname":"ec4432e19f48","pid":1,"level":30,"msg":"Your site is now available on https://blog-test.aaronlenoir.com/","time":"2019-09-28T22:04:14.112Z","v":0}
{"name":"Log","hostname":"ec4432e19f48","pid":1,"level":30,"msg":"Ctrl+C to shut down","time":"2019-09-28T22:04:14.114Z","v":0}
{"name":"Log","hostname":"ec4432e19f48","pid":1,"level":30,"msg":"Ghost boot 33.975s","time":"2019-09-28T22:04:14.117Z","v":0}

So everything looks hunky-dory. Then why is nginx giving me a 500?!

Shell into nginx

I can attach a console to the running nginx proxy server:

Use docker ps to find the container name:

user@docker:~/aaronlenoir.com$ sudo docker ps
CONTAINER ID        IMAGE                                    COMMAND                                                                                      CREATED             STATUS              PORTS                                                                                                          NAMES
8c2683b52ce0        jrcs/letsencrypt-nginx-proxy-companion   "/bin/bash /app/ent                                                                    r…"   16 minutes ago      Up 2 minutes                                                                                                                       aaronlenoircom_nginx-proxy-companon_1
5c899f091de8        ghost:1-alpine                           "docker-entrypoint.                                                                    s…"   16 minutes ago      Up 2 minutes        127.0.0.1:8090->2368/tcp                                                                                       aaronlenoircom_flightschool_1
023f4d1bed3f        ghost:2-alpine                           "docker-entrypoint.                                                                    s…"   16 minutes ago      Up 2 minutes        127.0.0.1:8120->2368/tcp                                                                                       aaronlenoircom_ghost2_1
0448fff371fe        kicker                                   "dotnet Kicker.Stat                                                                    s…"   16 minutes ago      Up 2 minutes        127.0.0.1:8100->80/tcp                                                                                         aaronlenoircom_kicker_1
9200cd720030        tracks                                   "dotnet Mapper.dll"                                                                          16 minutes ago      Up 2 minutes        127.0.0.1:8110->80/tcp                                                                                         aaronlenoircom_tracks_1
eb8844ca3d79        ghost:1-alpine                           "docker-entrypoint.                                                                    s…"   16 minutes ago      Up 2 minutes        127.0.0.1:8080->2368/tcp                                                                                       aaronlenoircom_ghost_1
411b175ed115        jwilder/nginx-proxy                      "/app/docker-entryp                                                                    o…"   16 minutes ago      Up 2 minutes        0.0.0.0:80->80/tcp, 0.0.0.0:443->4                                                                    43/tcp   aaronlenoircom_nginx-proxy_1

It seems to be aaronlenoircom_nginx-proxy_1. Then I must execute bash to get a shell in the running container:

user@docker:~/aaronlenoir.com$ sudo docker exec -it aaronlenoircom_nginx-proxy_1 /bin/bash
root@411b175ed115:/app#

Being in the shell I can see if nginx decided to log anything. But where?

In /etc/nginx/nginx.conf the location of the logs is mentioned:

error_log  /var/log/nginx/error.log warn;

However, the file itself points to /dev/stderr so I think I would've seen something in the docker-compose test run. So no luck there!

Shell into ghost

I can do the same with ghost though:

user@docker:~/aaronlenoir.com$ sudo docker exec -it aaronlenoircom_ghost2_1 /bin/bash
bash-4.4#

Looking at netstat I can see it is in fact running and listening on port 2368.

bash-4.4# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:2368            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.11:43769        0.0.0.0:*               LISTEN
udp        0      0 127.0.0.11:44987        0.0.0.0:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path

Log files? No we already checked that!?

Wget from the host

Normally, nginx sends a request to the port 8120. So I can do that too, from the VPS. And that yeilds some more information!

user@docker:~/aaronlenoir.com$ wget localhost:8120
--2019-09-28 22:44:35--  http://localhost:8120/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8120... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://localhost:8120/ [following]
--2019-09-28 22:44:35--  https://localhost:8120/
Connecting to localhost (localhost)|127.0.0.1|:8120... connected.
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Unable to establish SSL connection.

What we see is that ghost sees my request, but tells me to come back via "https". Since I did not configure https this - of course - won't work.

Since nginx takes care of this, the internal applications shouldn't be doing this. I'm a little surprised ghost has https enabled by default - BUT THAT'S GOOD, usually.

Step 2.2: Let Ghost 2 allow HTTP

I'll shell back into the ghost container:

user@docker:~/aaronlenoir.com$ sudo docker exec -it aaronlenoircom_ghost2_1 /bin/bash
bash-4.4#

I wanted to use ghost cli to see the config but:

bash-4.4# ghost config
You can't run commands as the 'root' user.
Switch to your regular user, or create a new user with regular account privileges and use this user to run 'ghost config'.
For more information, see https://docs.ghost.org/install/ubuntu/#create-a-new-user-.

Since I'm in docker, I don't really have a normal user.

Where is the config file? Oh it's here: /var/lib/ghost/config.production.json

bash-4.4# cat config.production.json
{
  "url": "http://localhost:2368",
  "server": {
    "port": 2368,
    "host": "0.0.0.0"
  },
  "database": {
    "client": "sqlite3",
    "connection": {
      "filename": "/var/lib/ghost/content/data/ghost.db"
    }
  },
  "mail": {
    "transport": "Direct"
  },
  "logging": {
    "transports": [
      "file",
      "stdout"
    ]
  },
  "process": "systemd",
  "paths": {
    "contentPath": "/var/lib/ghost/content"
  }
}

I don't immediatly see a setting forcing or not forcing https?

I did set the URL environment variable to https://test-blog.aaronlenoir.com so maybe that's why it does https. I'll change it to http in my docker-compose.yml.

That seemed to yield another result:

user@docker:~/aaronlenoir.com$ wget localhost:8120
--2019-09-28 23:04:35--  http://localhost:8120/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8120... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21826 (21K) [text/html]
Saving to: ‘index.html’

index.html                           100%[======================================================================>]  21.31K  --.-KB/s    in 0s

2019-09-28 23:04:36 (345 MB/s) - ‘index.html’ saved [21826/21826]

But still I receive an error 500 from nginx! But we now know the blog is running and NOT redirecting to https ...

Step 2.3: Working without HTTPS?!

I took everything down and then ran

sudo docker-compose up

And now everything is working! But over HTTP. Why am I redirected to HTTP?

I already have multiple apps running internally on http and externally on https and I've never seen this.

I will first try to register the hostname so that maybe I get my SSL certificate sorted in nginx!

Step 3: Register hostname

I own the domain aaronlenoir.com. I can add a subdomain in my administration panel with Namecheap:

Good news! Doing that solved all the problems.

Step 3.1: More troubleshooting

I now want to set the url in docker-compose.yml back to https to see if it still works ...

...

Success, everything still works!

Conclusion

I thought this was going to be a short post.

And - apart from the two hour debug session - that's how eAsY it was to add a new Ghost 2 instance to my blog.

😭