Updating LCOJ

Move a Docker install to a newer version: pull code, rebuild when needed, run migrations and restart the right services.
⏱ ~15–30 min (longer if images must be rebuilt) · 👤 Operators · 🔑 SSH + permission to run docker and git on the server

Always back up first

Before updating, back up the database. Git can bring old code back, but migrations that have already run don't undo themselves.

Before you start

You have backed up the database (ideally problems/ and media/ too).
You picked a quiet time with no contest running.
You have SSH access and can run docker compose and git in lcoj-docker/dmoj/.
You know which lcoj-site branch your install follows (the examples on this page use prod/luyencode).
You know what a migration is (a database schema change generated by Django). See the Glossary.

The process at a glance:

Two repos to update

An install is made of two nested Git repos:

Directory	Repo	Contains
`lcoj-docker/`	lcoj-docker	`docker-compose.yml`, Dockerfiles, scripts, config templates, nginx
`lcoj-docker/dmoj/repo/`	lcoj-site (submodule)	All Django code, templates, CSS/JS, WebSocket server

Submodules and "detached HEAD"

The outer repo doesn't record a branch for the submodule. It pins dmoj/repo to one specific commit. As a result:

git submodule update checks out exactly that pinned commit, which leaves dmoj/repo in a detached HEAD state (not on any branch). git pull inside it fails until you check out a branch.
.gitmodules has no branch entry, so git submodule update --remote follows lcoj-site's default branch (master), not prod/luyencode.
The commands on this page follow lcoj-site's prod/luyencode branch. If you follow a different branch, substitute its name.

Check where you are:

git -C repo status | head -1   # "On branch prod/luyencode" or "HEAD detached at ..."

If you're on a detached HEAD and want to follow prod/luyencode:

git -C repo fetch origin
git -C repo checkout prod/luyencode

Update steps

Run every command from lcoj-docker/dmoj/.

Record the current commits of both repos so you can see what changed and roll back if needed:

OLD=$(git -C repo rev-parse HEAD); echo $OLD
OLD_DOCKER=$(git rev-parse HEAD); echo $OLD_DOCKER

Update the outer repo (Dockerfiles, scripts, config templates):
sh
```
git pull --ff-only
```

Update the site code. Pick one approach:

Follow prod/luyencodeUse the commit pinned by the outer repo

git -C repo fetch origin
git -C repo checkout prod/luyencode
git -C repo pull --ff-only origin prod/luyencode

git submodule update --init --recursive
# repo/ is now on a detached HEAD, which is expected

See what changed:

git -C repo diff --stat $OLD HEAD
git diff --stat $OLD_DOCKER HEAD -- .   # changes under lcoj-docker/dmoj

Follow the table below for whatever changed.

What to do for each kind of change

Change	Action
`requirements.txt`, `additional_requirements.txt`, `package.json`, `package-lock.json`	Rebuild the `base` image, then the images built on it (see below)
`dmoj/base/Dockerfile`, `dmoj/site/Dockerfile`, `dmoj/celery/Dockerfile`, `dmoj/bridged/Dockerfile`, `dmoj/wsevent/Dockerfile`	`docker compose build <service>`, then `docker compose up -d`
New models / files in `*/migrations/`	`./scripts/migrate`
SCSS, JS or images in `resources/`, translations in `locale/`	`./scripts/copy_static`
Python code, templates	`docker compose restart site celery bridged`
`websocket/*.js`	`docker compose restart wsevent`
`docker-compose.yml`, `environment/*.env.example`	Compare with your `.env` files, add any new variables, then `docker compose up -d`
`config/local_settings.py`, `config/uwsgi.ini`, `config/config.js`	Port the changes into the copies in `repo/` by hand (see the warning below), then restart the matching service
`nginx/conf.d/nginx.conf`	`docker compose restart nginx`

Code changes do not need an image rebuild because ./repo is bind-mounted into the containers. A restart is enough.

Config files in repo/ don't update themselves

repo/dmoj/local_settings.py, repo/uwsgi.ini and repo/websocket/config.js are ignored by lcoj-site's .gitignore, so git pull never touches them. If a template in config/ changes, compare them (diff config/local_settings.py repo/dmoj/local_settings.py) and edit by hand. Re-running ./scripts/initialize would overwrite your own changes.

Rebuilding when dependencies change

Python, Node.js and every library live in the lcoj/lcoj-base image. The site, celery and bridged images are built from it, while wsevent installs package.json on its own. Rebuilding only site does not install new libraries.

docker compose build base
docker compose build site celery bridged wsevent
docker compose up -d

If you suspect Docker is reusing a stale cache, add --no-cache to the base build.

Finishing up

If you're not sure what changed, run all of these. Each one is safe to repeat:

./scripts/migrate
./scripts/copy_static
docker compose restart site celery bridged wsevent
docker compose ps

Sample update script

This script follows the steps above for the prod/luyencode branch. Save it as dmoj/update.sh (*.sh files in dmoj/ are already ignored by .gitignore) and chmod +x it.

bash

#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")"                 # the dmoj/ directory
export COMPOSE_EXEC_FLAGS=-T         # lets the scripts in scripts/ run without a terminal
BRANCH=prod/luyencode
STAMP=$(date +%F_%H%M%S)

echo "1. Back up the database"
mkdir -p backups
docker compose exec -T db sh -c \
  'MYSQL_PWD="$MYSQL_ROOT_PASSWORD" exec mariadb-dump -u root --single-transaction "$MYSQL_DATABASE"' \
  | gzip > "backups/db_before_update_$STAMP.sql.gz"

echo "2. Pull new code"
OLD=$(git -C repo rev-parse HEAD)
git pull --ff-only
git -C repo fetch origin
git -C repo checkout "$BRANCH"
git -C repo pull --ff-only origin "$BRANCH"
NEW=$(git -C repo rev-parse HEAD)

if [ "$OLD" = "$NEW" ]; then
  echo "Site code unchanged."
else
  echo "Updating $OLD -> $NEW"
  CHANGED=$(git -C repo diff --name-only "$OLD" "$NEW")

  if echo "$CHANGED" | grep -qE '^(requirements\.txt|additional_requirements\.txt|package(-lock)?\.json)$'; then
    echo "3. Dependencies changed: rebuilding images"
    docker compose build base
    docker compose build site celery bridged wsevent
  fi
fi

echo "4. Start (recreates containers whose image or config changed)"
docker compose up -d

echo "5. Migrations and static files"
./scripts/migrate
./scripts/copy_static

echo "6. Restart to load the new code"
docker compose restart site celery bridged wsevent
docker compose ps

echo "Done. Follow the logs with: docker compose logs -f site"

The script doesn't compare config/ against the config files in repo/. Check the diff from step 4 above after every update.

Verify

docker compose ps: every service (except base) is Up.
docker compose logs --tail=50 site celery bridged: no tracebacks.
Open the site, log in, view a problem and make a test submission.
In docker compose logs bridged, confirm the judges have reconnected.

Troubleshooting

Symptom	Fix
`git pull` inside `repo/` fails because you're not on a branch	`repo/` is in detached HEAD. Run `git -C repo checkout prod/luyencode`, then pull again (see Submodules and "detached HEAD")
`ModuleNotFoundError` after updating	New dependencies weren't installed: rebuild `base` first, then the other images (Rebuilding)
CSS missing or old UI	`./scripts/copy_static`, then `docker compose restart site`
Errors about missing tables/columns	Migrations weren't run: `./scripts/migrate`
A new variable in `.env` has no effect	Use `docker compose up -d`, not `restart`
A new feature needs config but doesn't work	Compare `config/` with the copies in `repo/` and edit by hand (see the warning above)
The site is badly broken and can't be fixed quickly	Follow Rolling back below

Rolling back

DANGER

Going back to old code does not reverse migrations that already ran. If the new version included migrations, the safest route is restoring the database backup you took before updating.

Stop the services that write data:
sh
```
docker compose stop site celery bridged
```
Put the code back on the old commit ($OLD from step 1):
sh
```
git -C repo checkout <OLD>
```
If there were new migrations, restore the database from the pre-update backup.
If dependencies changed, rebuild as described above.

Start again:

docker compose up -d
./scripts/copy_static
docker compose restart site celery bridged wsevent

Once the problem is fixed, get back on the branch with git -C repo checkout prod/luyencode.

Tips

Update during quiet hours, and never in the middle of a contest.
Warn users ahead of time. While site is stopped, nginx serves the 502.html page (see Maintenance page).
If you can, try the new version on a test machine first.

Next steps

Operating LCOJ: logs, clearing the cache, regular backups.
Environment Variables: when a new version adds variables to environment/*.env.example.
Management Commands: ./scripts/manage.py commands you may need after an update.

Need help?

Open an issue on lcoj-docker, or reach us via behitek.com or luyencode.net/about/#lien-he.

Updating LCOJ ​

Before you start ​

Two repos to update ​

Submodules and "detached HEAD" ​

Update steps ​

What to do for each kind of change ​

Rebuilding when dependencies change ​

Finishing up ​

Sample update script ​

Verify ​

Troubleshooting ​

Rolling back ​

Tips ​

Next steps ​