In this post, I documents the steps I take to add a new website to aaronlenoir.com using Digital Ocean and docker.
I think I'd like to upgrade my blogs to blog engine Ghost 2 (I'm using version 1 now).
It's quite the update, so first I want to run a new instance on the side to see how it works.
I will be adding, as a test, blog-test.aaronlenoir.com.
Update: I though this would be a simple ten minute task, but Ghost 2 is giving me problems
Current Setup
At the moment, I'm hosting four websites.
Two Ghost 1 blogs:
- blog.aaronlenoir.com
- flightschool.aaronlenoir.com
Two home-made .NET Core applications:
- tracks.aaronlenoir.com
- kicker.aaronlenoir.com
All of them run on a single Digital Ocean "droplet" (a Virtual Private Server or VPS).
In front of them is an nginx instance configured as a reverse proxy. Nginx is responsible for redirecting requests to the appropriate application, which are not exposed to the internet directly. Additionally nginx takes care of the SSL certificates.
All 4 applications and nginx run as a docker container. Additionally, I run a sixth docker contain responsible for renewing my SSL certificates for each of the hostnames.
To manage the configuration of each container, I use "docker-compose". The docker-compose configuration can be found on GitHub: https://github.com/AaronLenoir/aaronlenoir.com/blob/a059df908ddbccc6147ea064a2fdf55ae2296669/docker-compose.yml
In summary I run 6 docker containers that do all the work. To add a new website I must add another container and associate it with a new hostname so that nginx can forward the requests correctly.
Step 1: Run Ghost 2 via docker-compose
I log into my VPS using SSH. It's a UNIX system (I know this).
Ghost says they don't offer an "official" docker image to run the blog. But they do point to an "unofficial" one: https://hub.docker.com/_/ghost/.
Interestingly, the docker page says it's the Official image. So maybe the ghost FAQ is a little outdated and it IS official now.
According to the docs, to run a ghost image on the default port I must run the following:
$ docker run -d --name some-ghost ghost
I COULD do that, but I wouldn't be able to test it. So I'm going to immediatly put it in my docker-compose.yml file:
ghost2:
image: ghost:2-alpine
restart: always
depends_on:
- "nginx-proxy"
ports:
- 127.0.0.1:8120:2368
volumes:
- ./blog2/data/ghost:/var/lib/ghost/content
environment:
- url=https://blog-test.aaronlenoir.com
- VIRTUAL_HOST=blog-test.aaronlenoir.com
- LETSENCRYPT_HOST=blog-test.aaronlenoir.com
- LETSENCRYPT_EMAIL=info@aaronlenoir.com
ghost2 is the name of the container.
image indicates I wish to use the latest 2.x version. I use alpine because of the lower footprint and it worked well with version 1.
restart is set to always that way I'm sure if it's stopped correctly via docker that it gets restarted. I can still stop it using docker-compose.
depends-on specifies the nginx-proxy must be running before this container can start up.
ports means internally ghost runs on port 2368 (the default) but the container should listen on port 8120 externally.
volumes points to the location on my server where all the files should be (theme, images hosted on the blog, logs, database, ...). The first part is the location on my server, the second part is the location inside the container to which it should map. I'm assuming ghost 2 still expects the data in /var/lib/ghost/content
environment sets a number of environment variables in the container:
urlthe URL to use for the blog - may not be necessary for ghost 2VIRTUAL_HOSTis used for the nginx-proxy to know that requests for that hostname must go to this containerLETSENCRYPT_HOSTtells the let's encrypt container the hostname for which it must fetch a certificateLETSENCRYPT_EMAILis passed to Let's Encrypt when I'm asking a new certificate, they use that to send me mails about the certificate expiration (but I renew automatically - normally)
After adding this to my docker-compose.yml file I restart everything. Which results in the download of the image:
Creating network "aaronlenoircom_default" with the default driver
Pulling ghost2 (ghost:2-alpine)...
2-alpine: Pulling from library/ghost
e7c96db7181b: Pull complete
50958466d97a: Pull complete
56174ae7ed1d: Pull complete
284842a36c0d: Pull complete
237455e2fb15: Pull complete
e9505cbbbd44: Pull complete
711e6ff570b7: Extracting [====================================> ] 4.522MB/6.145MBownload complete
65.56MB/68.44MBwnload complete
548B/548B
After that the following message indicates at least something started:
Creating aaronlenoircom_ghost2_1 ... done
Step 2: Testing
To test, I usually tell my local machine to resolve the test hostname to the IP address of my VPS.
In windows, that's done by editing the file C:\Windows\Systems32\drivers\etc\hosts
I add the following entry:
82.196.2.207 blog-test.aaronlenoir.com
With that I entered the url in my browser. This immediatly gives me an error because I don't yet have an SSL certificate. This is normal because Let's Encrypt won't yet know of the hostname.
Of course I can tell Firefox to "accept the risk and continue", because I know what this site is.
Sadly, I'm greeted by a server error:
500 Internal Server Error
nginx/1.17.3
Step 2.1: Troubleshooting
TL;DR This troubleshooting session goes through a lot of useless stuff that eventually turned out not to be important. You could skip to Step 3 ...
Since the error is coming from nginx, I'm assuming the nginx proxy server wasn't able to forward my request. I presume this is because the ghost container couldn't start correctly.
Docker-Compose logs
To check this, I usually run docker-compose in interactive mode, so that I can see what happens in each of the containers:
sudo docker-compose up
Strangely it looks like ghost 2 is running. This was the only logging I could find at start-up:
ghost2_1 | [2019-09-28 22:12:11] INFO Ghost is running in production...
ghost2_1 | [2019-09-28 22:12:11] INFO Your site is now available on https://blog-test.aaronlenoir.com/
ghost2_1 | [2019-09-28 22:12:11] INFO Ctrl+C to shut down
ghost2_1 | [2019-09-28 22:12:11] INFO Ghost boot 25.093s
When I visit the site, the logging tells me, not as much as I was hoping for:
nginx-proxy_1 | nginx.1 | blog-test.aaronlenoir.com 84.192.158.84 - - [28/Sep/2019:22:15:11 +0000] "GET / HTTP/2.0" 500 177 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"
Ghost Logs
I did remember pointing to a data directory that doesn't yet exist: blog2/data/ghost
It was created and it did generate some logs in the folder: /blog2/data/ghost/logs
In the logs, there are no errors and I can see it created the database correctly:
{"name":"Log","hostname":"ec4432e19f48","pid":1,"level":30,"msg":"Ghost is running in production...","time":"2019-09-28T22:04:14.110Z","v":0}
{"name":"Log","hostname":"ec4432e19f48","pid":1,"level":30,"msg":"Your site is now available on https://blog-test.aaronlenoir.com/","time":"2019-09-28T22:04:14.112Z","v":0}
{"name":"Log","hostname":"ec4432e19f48","pid":1,"level":30,"msg":"Ctrl+C to shut down","time":"2019-09-28T22:04:14.114Z","v":0}
{"name":"Log","hostname":"ec4432e19f48","pid":1,"level":30,"msg":"Ghost boot 33.975s","time":"2019-09-28T22:04:14.117Z","v":0}
So everything looks hunky-dory. Then why is nginx giving me a 500?!
Shell into nginx
I can attach a console to the running nginx proxy server:
Use docker ps to find the container name:
user@docker:~/aaronlenoir.com$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8c2683b52ce0 jrcs/letsencrypt-nginx-proxy-companion "/bin/bash /app/ent r…" 16 minutes ago Up 2 minutes aaronlenoircom_nginx-proxy-companon_1
5c899f091de8 ghost:1-alpine "docker-entrypoint. s…" 16 minutes ago Up 2 minutes 127.0.0.1:8090->2368/tcp aaronlenoircom_flightschool_1
023f4d1bed3f ghost:2-alpine "docker-entrypoint. s…" 16 minutes ago Up 2 minutes 127.0.0.1:8120->2368/tcp aaronlenoircom_ghost2_1
0448fff371fe kicker "dotnet Kicker.Stat s…" 16 minutes ago Up 2 minutes 127.0.0.1:8100->80/tcp aaronlenoircom_kicker_1
9200cd720030 tracks "dotnet Mapper.dll" 16 minutes ago Up 2 minutes 127.0.0.1:8110->80/tcp aaronlenoircom_tracks_1
eb8844ca3d79 ghost:1-alpine "docker-entrypoint. s…" 16 minutes ago Up 2 minutes 127.0.0.1:8080->2368/tcp aaronlenoircom_ghost_1
411b175ed115 jwilder/nginx-proxy "/app/docker-entryp o…" 16 minutes ago Up 2 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->4 43/tcp aaronlenoircom_nginx-proxy_1
It seems to be aaronlenoircom_nginx-proxy_1. Then I must execute bash to get a shell in the running container:
user@docker:~/aaronlenoir.com$ sudo docker exec -it aaronlenoircom_nginx-proxy_1 /bin/bash
root@411b175ed115:/app#
Being in the shell I can see if nginx decided to log anything. But where?
In /etc/nginx/nginx.conf the location of the logs is mentioned:
error_log /var/log/nginx/error.log warn;
However, the file itself points to /dev/stderr so I think I would've seen something in the docker-compose test run. So no luck there!
Shell into ghost
I can do the same with ghost though:
user@docker:~/aaronlenoir.com$ sudo docker exec -it aaronlenoircom_ghost2_1 /bin/bash
bash-4.4#
Looking at netstat I can see it is in fact running and listening on port 2368.
bash-4.4# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:2368 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.11:43769 0.0.0.0:* LISTEN
udp 0 0 127.0.0.11:44987 0.0.0.0:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
Log files? No we already checked that!?
Wget from the host
Normally, nginx sends a request to the port 8120. So I can do that too, from the VPS. And that yeilds some more information!
user@docker:~/aaronlenoir.com$ wget localhost:8120
--2019-09-28 22:44:35-- http://localhost:8120/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8120... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://localhost:8120/ [following]
--2019-09-28 22:44:35-- https://localhost:8120/
Connecting to localhost (localhost)|127.0.0.1|:8120... connected.
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Unable to establish SSL connection.
What we see is that ghost sees my request, but tells me to come back via "https". Since I did not configure https this - of course - won't work.
Since nginx takes care of this, the internal applications shouldn't be doing this. I'm a little surprised ghost has https enabled by default - BUT THAT'S GOOD, usually.
Step 2.2: Let Ghost 2 allow HTTP
I'll shell back into the ghost container:
user@docker:~/aaronlenoir.com$ sudo docker exec -it aaronlenoircom_ghost2_1 /bin/bash
bash-4.4#
I wanted to use ghost cli to see the config but:
bash-4.4# ghost config
You can't run commands as the 'root' user.
Switch to your regular user, or create a new user with regular account privileges and use this user to run 'ghost config'.
For more information, see https://docs.ghost.org/install/ubuntu/#create-a-new-user-.
Since I'm in docker, I don't really have a normal user.
Where is the config file? Oh it's here: /var/lib/ghost/config.production.json
bash-4.4# cat config.production.json
{
"url": "http://localhost:2368",
"server": {
"port": 2368,
"host": "0.0.0.0"
},
"database": {
"client": "sqlite3",
"connection": {
"filename": "/var/lib/ghost/content/data/ghost.db"
}
},
"mail": {
"transport": "Direct"
},
"logging": {
"transports": [
"file",
"stdout"
]
},
"process": "systemd",
"paths": {
"contentPath": "/var/lib/ghost/content"
}
}
I don't immediatly see a setting forcing or not forcing https?
I did set the URL environment variable to https://test-blog.aaronlenoir.com so maybe that's why it does https. I'll change it to http in my docker-compose.yml.
That seemed to yield another result:
user@docker:~/aaronlenoir.com$ wget localhost:8120
--2019-09-28 23:04:35-- http://localhost:8120/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8120... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21826 (21K) [text/html]
Saving to: ‘index.html’
index.html 100%[======================================================================>] 21.31K --.-KB/s in 0s
2019-09-28 23:04:36 (345 MB/s) - ‘index.html’ saved [21826/21826]
But still I receive an error 500 from nginx! But we now know the blog is running and NOT redirecting to https ...
Step 2.3: Working without HTTPS?!
I took everything down and then ran
sudo docker-compose up
And now everything is working! But over HTTP. Why am I redirected to HTTP?
I already have multiple apps running internally on http and externally on https and I've never seen this.
I will first try to register the hostname so that maybe I get my SSL certificate sorted in nginx!
Step 3: Register hostname
I own the domain aaronlenoir.com. I can add a subdomain in my administration panel with Namecheap:
Good news! Doing that solved all the problems.
Step 3.1: More troubleshooting
I now want to set the url in docker-compose.yml back to https to see if it still works ...
...
Success, everything still works!
Conclusion
I thought this was going to be a short post.
And - apart from the two hour debug session - that's how eAsY it was to add a new Ghost 2 instance to my blog.
😭
