I’ve been playing around with Kamal from Basecamp (previously called MRSK) for deploying simple apps on a single server.
There’s a lot to like about Kamal’s ergonomics and principles. But there were a few things that I struggled with or that confused me.
It mostly boils down to Kamal offering some kind of a layer of abstraction over docker, SSH and some linux commands. But perhaps more importantly, DHH, the creator of Kamal quite explicitly says that:
“[It] is designed for multi-server operation”.
DHH
Why is this distinction important?
Because it implicitly avoids some of the nice (and more secure) features of docker on a single host, primarily: internal network connections and name resolution.
[It] is designed for multi-server operation, so the internal network idea breaks down pretty quick with that. You’d have to unstrip all of that when you go to scale it. So I think we’re better off keeping the network host transparent.
DHH
This is a completely fair design choice, and simplifies a lot of complexity for Kamal. However, when you’re running your new startup or a hobby project, you want to keep things simple and run it all on one host.
But you don’t want to compromise on security and unintentionally expose your Database or Redis to the outside world, right?
A naive single-server deploy might look something like this:
service: app
image: app-image
servers:
web:
hosts:
# external IP of your server / VPS
- 64.65.67.68
env:
clear:
# this ENV is used by your Rails database.yml
# if you're running Rails to connect to the database
- DATABASE_HOST: 64.65.67.68
secret:
- POSTGRES_PASSWORD
accessories:
db:
image: postgres:16
host: 64.65.67.68
port: 5432This works, but there’s a problem here. You’re exposing your database publicly to the world. Port 5432 would be open on your public IP address 64.65.67.68. You probably don’t want to do that. Even if you use a strong password, you don’t really want script kids hammering your database with bruteforce attacks or whatnot.
I found examples of this on the official documentation as well as several posts about getting started with Kemal and particularly for using it on a single server.
The problem is that it’s easy to miss the full picture. On Josef Strzibny’s post for example, deploy.yml exposed the Postgres database, even though his own blog post was highlighting this as a potential problem.
Josef explained why he’s doing it, which is 100% valid. So it’s not a mistake. However it’s easy to skip the explanation and copy&paste the example — without being fully aware of the consequences when running a hobby project.
In my deployment book I suggest using the peer authentication for better security (no passwords) and direct UNIX sockets for better performance on a single host, but this setup will become handy later when scaling up to multiple hosts.
Josef Strzibny
Hidden treasures
Another option I’ve seen on the official docs and online (even from Greg Molnar, who’s a Rails security expert) was to map the service to a different port. i.e. instead of port: 5432, use port: "7890:5432" and then map your database to this port.
This will definitely reduce the volume of script kiddie attacks, but still leaves your database exposed. On Greg’s post the deploy.yml configuration exposed a redis instance to the world — and redis does not use any authentication by default!
Keeping our ships at bay
What you want to do in this case is to keep the connection from your app to your database(s) internal to the host. And docker supports it well using the network option.
The only problem is that Kamal doesn’t support docker networks out-of-the-box. Again, that’s not by omission, but as a valid design choice. But we can make it work. How?
The recent master version of Kamal introduced a new docker-setup hook. As of the time of writing, this isn’t available on the released version 1.3.1. So you’d need to run kamal from master until it’s available. Alternatively, you can SSH to your host and manually create the network.
Let’s call this network private. The command to create it (once docker is installed) is docker network create -d bridge private.
If you are running the latest version of kamal, you can place this script in your .kamal/hooks/docker-setup and make sure it’s executable (chmod +x .kamal/hooks/docker-setup)
#!/usr/bin/env sh
# Based on https://github.com/basecamp/kamal/issues/41#issuecomment-1789223148
NETWORK_NAME=private
USER=root
# Check if KAMAL_HOSTS environment variable is set
if [ -z "$KAMAL_HOSTS" ]; then
echo "Error: KAMAL_HOSTS environment variable is not set."
exit 1
fi
# Split comma-separated hosts into an array
IFS=','
# Use a for loop to iterate over hosts
for host in $KAMAL_HOSTS; do
echo "Executing on $host"
ssh "$USER@$host" << EOF
# Check if the Docker network already exists
if ! docker network inspect "$NETWORK_NAME" &>/dev/null; then
# If it doesn't exist, create it
docker network create -d bridge "$NETWORK_NAME"
echo "Created Docker network: $NETWORK_NAME"
else
echo "Docker network $NETWORK_NAME already exists, skipping creation."
fi
EOF
doneOnce you have the private network created, your docker containers can communicate between each other without exposing ports to the outside world. In addition, you can connect to your database from your app using its internal name, which is nice.
Your deploy.yml would look like this now:
service: app
image: app-image
servers:
web:
hosts:
# external IP of your server / VPS
- 64.65.67.68
# tells docker to use the new network called "private"
options:
network: private
env:
clear:
# The name is comprised of your service name (app) and accessory name (db)
- DATABASE_HOST: app-db
secret:
- POSTGRES_PASSWORD
accessories:
db:
image: postgres:16
host: 64.65.67.68
# port: 5432 (remove this! Important!)
# use the same (private) network as your app
options:
network: private
# note you also need to use the same network for traefik
# for it to connect to your app now
traefik:
options:
network: privateWith the private network in place, your docker containers can connect to each other, only exposing ports that are intended to be accessible to the outside world.
