1. Install Docker
2. (optional) Edit your hosts file and make local.host point at 127.0.0.1. This is maybe unnecessary, but browsers and various tools often have special treatment for “localhost” or 127.0.0.1, which has caused problems for me in the past.
3. Locally, create a root dev directory and subdirectories db, wordpress, and whatever themes you want to work on (I used autonomie-dev). You can skip the theme directory if your theme doesn’t require any special build steps. I git-cloned Autonomie.
4. Create a docker-compose.yaml in the root directory. See below.
5. Run docker compose up -d. This should get an empty WordPress installation started.
6. docker compose down to stop it, so we could import the remote installation.
7. Copy over the remote wordpress directory into the local wordpress/ overwriting any existing files. I used rsync -av --delete remote:/path/to/wordpress/ wordpress/ from root.
8. If you don’t want to import your remote content, skip this step. Otherwise:
   • docker compose up db to start only MySQL
   • On the remote side run mysqldump and import locally via mysql -ppassword < dump.sql
   • docker compose up phpmyadmin . Log in to http://localhost:8180 with the MySQL username + password. In the “options” table update two records to point to http://localhost:8080/ (or http://local.host:8080/, if you updated the hosts file). The records are siteurl and home under the option_name column. This ensures that your local setup doesn’t redirect to your real remote domain.
9. docker compose up -d . You should be able to see your site on http://localhost:8080 (or http://local.host:8080 if you changed the hosts file). Now you can change the theme files as needed.

I also wanted to be able to version control the theme with Git and have it publish the local changes on the remote side when I push. For that I followed this tutorial with some tweaks:

1. Set up a bare git repo outside the web root on your remote host with git init --bare .
   • e.g. if your web root is /var/www/html/wordpress run git init --bare /var/www/html/your_theme
2. Create an empty theme directory. I used autonomie-dv, so for me the location was /var/www/html/wordpress/wp-content/themes/autonomie-dv. Set this up in the local wordpress install too.
3. In the bare git repo, under hooks create a file called post-update. Make it executable with chmod ug+x post-update . Contents are below. Now whenever you push into this repository it’ll copy over the theme files into your actual remote installation.
4. Inside your local theme directory at the root level, git init . and then add the bare repo as a git remote: git remote add origin your-ssh-host:/var/www/html/your_theme .
5. (optional) Specifically for Autonomie, I removed _build from .gitignore because I didn’t want to run the build step remotely. I wanted to build locally and then have the git hook just copy the files over. I may change my mind about this later. I also tweaked the grunt clean step to not remove the _build directory, because that would break the Docker volume bind and require restarting the containers for every change to remount the recreated directory.
   • I tweaked the watch target in Gruntfile.js to trigger on more file extensions, so I could run grunt build watch and have it automatically rebuild on any local changes.

Now when you git push you should see rsync output that copies the files over.

docker-compose.yaml

services:
  db:
    restart: unless-stopped
    image: mysql:8
    command: '--mysql-native-password=ON --disable-log-bin --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci'
    environment:
      MYSQL_DATABASE: "wordpress"
      MYSQL_ROOT_PASSWORD: "password"
      MYSQL_USER: "wordpress"
      MYSQL_PASSWORD: "password"
    volumes:
      - "./db:/var/lib/mysql"

  phpmyadmin:
    depends_on:
      - db
    image: phpmyadmin/phpmyadmin:latest
    container_name: phpmyadmin
    ports:
      - 8180:80
    environment:
      PMA_HOST: db
      MYSQL_ROOT_PASSWORD: "password"

  wordpress:
    depends_on:
      - db
    restart: unless-stopped
    image: wordpress
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_NAME: wordpress
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DB_PASSWORD: "password"

      # If you imported your DB and it uses a table prefix uncomment
      # this and update as needed.
      # WORDPRESS_TABLE_PREFIX: "wp_abc"

    volumes:
      - "./wordpress:/var/www/html"

      # Uncomment the next line and update the paths for your theme.
      # My local theme dir is autonomie-dev, but the theme name in
      # WordPress is dv-autonomie. You could totally name them the
      # same. Also note that I'm using _build/ because Autonomie
      # requires a build step (using grunt) and that's where it
      # outputs the results.
      # - "./autonomie-dev/_build:/var/www/html/wp-content/themes/dv-autonomie:ro"
    ports:
      - "127.0.0.1:8080:80"

post-update

#!/bin/sh

# Check out the theme into a temporary directory and then
# copy it over via rsync into the actual WordPress theme location

export target="../wordpress/wp-content/themes/autonomie-dv"
export GIT_WORK_TREE="/tmp/autonomie-dv"
mkdir -p "$GIT_WORK_TREE"
git checkout -f

# This will overwrite the target location. If you're not sure that
# you got it right, add '-n' to do a dry-run. i.e.:
# rsync -avn --delete "${GIT_WORK_TREE}/_build/" "${target}/"
rsync -av --delete "${GIT_WORK_TREE}/_build/" "${target}/"

A colleague asked this (paraphrased) question:

I’m adding an endpoint that will check an authentication token for a user in our token store and refresh it if necessary (/user-auth-tokens/{userId}). Which HTTP method should I use? POST seems to be a bad fit. JSON:API doesn’t support PUT. Should I use PATCH? JSON:API requires a body in PATCH requests, but I have no data to update. I’m passing the user id in the URL. Should I move it to the body, so I could use PATCH? Using GET seems worse, because I’m potentially making an update to refresh the token if it’s expired, and I don’t actually want the client to be able to fetch the user’s token either way. What do I do?

Note that while the question references JSON:API, which we use at my current job, the problem is more general. Assuming you don’t want to drop below level 2 of the Richardson Maturity Model, when designing REST APIs how do you represent operations that are conditional or don’t neatly align with the common HTTP verbs?

The following is my response. My colleague found it useful, so I’m reposting it here (after a bit of cleanup to remove internal/sensitive info).

I love this question because it lets me share a general approach with the group that’s worked well for me in the past:

Rule of thumb for REST: if you’re acting on something without updating anything, change the model so you have something to update! 😀

In this case, you could have GET /user-auth-tokens/{userId} return a refreshedOn datetime field, such that the client could then call PATCH /user-auth-tokens/{userId} and set refreshedOn to the current time. The server would respond with the updated refreshedOn.

Let’s pretend the current time is 2024-03-09T10:00:00.000Z. The client asks the server to refresh the user’s token if it’s invalid:

PATCH /user-auth-tokens/someUserId123

{"refreshedOn": "2024-03-09T10:00:00.000Z"}

The server could respond with the “new now” from the server’s perspective, if the token was refreshed:

200 OK

{"refreshedOn": "2024-03-09T10:00:00.150Z"}

Note that it took the server 150ms to refresh the token, so its date is different in the response than what the client provided. If the server supports the ability to set the refresh date explicitly, the date in the response may match the client’s.

If the token is still valid, the server could respond with the old token refresh date:

200 OK

{"refreshedOn": "2024-02-01T09:59:14.697Z"}

But the point is, there’s now a field that the client changes (or tries to change) to indicate their intent, and the server responds with the result. This means you don’t need a separate endpoint for different operations on the same entity. An operation here is actually a modification (PATCH) of one or more data model fields. This leaves POST, PUT, and DELETE for creating, replacing, or deleting the entity as a whole. This works for a lot of situations where you’d usually be tempted to switch to the RPC-over-HTTP style, i.e. POST /user-auth-tokens/{userId}/refresh-if-needed.

Let’s take another example. This time, we want to migrate legacy tokens to a new format. Instead of /user-auth-tokens/{userId}/migrate-legacy we could add a version field to the existing model. So, it would now contain two fields: refreshedOn and version. To migrate a user’s token, the client would (again) call PATCH /user-auth-tokens/{userId} and set the new version in the request body:

PATCH /user-auth-tokens/someUserId123

{"version": 2}

The server may reply with a bunch of different error codes if, for example, the token is already at version 2 or if it can’t be migrated for some reason. A 200 OK would mean “migration is done” and version in the response would be 2.

This extends to batch operations too! For these cases, it’s best to add new endpoints to represent the batch operation as its own entity. The client would call POST /user-auth-token-migrations with some user ids and versions in the body to kick off a migration. The response is 201 with an added header Location: /user-auth-token-migrations/migration-id-123. Calling GET /user-auth-token-migrations/migration-id-123 returns the model representing the status of the batch operation, e.g. {"status":"running"}. Client could PATCH with {"status":"paused"} to pause the batch, if the server supports that. If that’s not supported, the server could reply with a 400-series error.

The beauty of this approach is the server may execute many operations behind the scenes and update multiple fields in the response without requiring a new endpoint. This stops the proliferation of endpoints that’s common with RPC APIs and allows each one to evolve along with the product. For example, let’s go back to the token refresh endpoint. Let’s say the model is now two fields: refreshedOn and expiresOn. When clients call PATCH with {"refreshedOn": "2024-03-09T10:00:00.000Z"} the server would respond with that date in refreshedOn and a new expiresOn. Maybe that’s set to one day later (2024-03-10T10:00:00.000Z), if that’s the standard token validity duration:

PATCH /user-auth-tokens/someUserId123

{"refreshedOn": "2024-03-09T10:00:00.000Z"}

200 OK

{
  "refreshedOn": "2024-03-09T10:00:00.000Z",
  "expiresOn":   "2024-03-10T10:00:00.000Z"
}

Now the client knows when it should request the next refresh.

What if we want to support the ability for some clients to get a long-lived token? They could just send both fields in the PATCH, setting expiresOn to something like 2025-01-01. If that’s allowed, the server responds with 200 OK and new values in both fields, including the 2025 date in expiresOn. If not, the response is a 400-series error, or a 200 OK with new refreshedOn and the standard one-day expiresOn (basically ignoring the client’s update to the field). Perhaps we now support extending an existing valid token without refreshing it? Clients can PATCH with just the new expiresOn date! With the RPC style, you have to either create separate endpoints /refresh and /extend  or move this logic into query params (/user-auth-tokens/someUserId123?extendTo=date). The latter doesn’t seem too bad at first, but it opens the door to infinite scope creep of the endpoint, because the query param becomes a catch-all “do something” instruction, even if it doesn’t make sense for the entity (e.g. /user-auth-tokens/{userId}?generateReport=true). PATCHing fields in the model constrains the endpoint to the entity it represents.

This “one weird trick” really unlocked REST for me when I was struggling with it earlier in my career. It feels strange at first to represent operations as updates to some fields of the entity model. But REST is all about entities! The fundamental contract is: clients perform operations by attempting to update entities to look a desired way, and the server responds with the model that represents the result of that attempt. The client then needs to take that response as the new true state of the entity, even if nothing changed or if there were more changes to the model than the client expected.

Why? It seems that dragging and dropping or selecting several images directly from the Gallery block sends them all to the server concurrently. The uploads may be completing sequentially, but WordPress does a lot of post-processing like resizing for each uploaded image. This post-processing is very memory and cpu intensive, which can cause the server to respond very slowly or outright crash¹ if too much of it is happening at once. At least as of WordPress 6.3.2, this background post-processing runs in parallel for all the images selected from or drag-and-dropped directly onto the Gallery block. With the Media Library approach, the images are processed one by one.

If you’re struggling to upload even a single image you may need to tweak a few PHP settings². This article here describes a few ways of doing it. The htaccess mechanism worked well for me:

# I use Cloudflare as a proxy for my site. Their free tier has some additional limitations.
# Cloudflare's max is 100M, so there's no point in allowing file uploads larger than that.
php_value upload_max_filesize 100M
php_value post_max_size 100M

# Cloudflare's timeout is around 100s. PHP default is 30s. The value below could probably be 100, but I'm leaving it higher so that image post-processing completes in the background even if the request is aborted by the proxy.
php_value max_execution_time 300

# Default is 128M and not enough for images >~6mb. Photos from recent phones are around this size, but often larger.
php_value memory_limit 256M

I initially tried tuning the max_file_uploads setting. It controls how many concurrent file uploads PHP will accept. It didn’t make a difference in my testing, though! WP uploads images one at a time anyway, but background processing is concurrent and not restricted by this. Default is 20.

When the server is overloaded it either responds with a generic HTTP 500-series error or closes the connection. WordPress can’t discern the cause of the issue and, unfortunately, its error messages aren’t clear. They may say something like “The response is not a valid JSON response.” or “HTTP Error“. It’s not a valid JSON response because the web server is returning an HTML page containing some human-readable error text. If you open developer tools in the browser, reload the page, and click on the failing media requests in the Networks tab, you’ll see what I mean.

This ticket may be about this bug, or at least a very similar issue.

1. That’s what happened in my case. I run WordPress inside Docker on a small VPS and give the container a memory limit. When it goes over that limit the container gets killed (out-of-memory / OOM). ↩︎
2. If you’re using a shared host like wordpress.com this may not be allowed. Your best bet is to reach out to their support folks and plead your case for a higher memory limit. ↩︎

Here are a couple of ways to mitigate it.

#1: Caching plugin

I use W3 Total Cache. It acts as a pseudo static site generator by crawling your own WordPress instance, creating, and then serving static HTML files. Other WP caching plugins work similarly.

This solves half the problem, as hits against the post URL quickly start returning static data.

#2: Caching oembed URLs

The problem is that caching plugins don’t seem to account for the oembed URLs that look like <your-site.com>/wp-json/oembed/1.0/embed?url=https%3A%2F%2Fyour-site.com%2Fyour-page%2F. These do query (and take down) the database during the Mastodon traffic storm. For me, that means lots of HTTP 500 responses:

Caching to the rescue!

Thankfully, these oembed URLs don’t change unless your post does, so they’re safe to cache for at least a few minutes.

I use Caddy as a reverse proxy to a WordPress Docker container, so I added the following block to my site’s config, after rebuilding Caddy with the cache-handler plugin (docs):

glyphy.com {
  # ... other configs ...
  route /wp-json/oembed/* {
    cache {
      stale 10m
      ttl 5m
      badger {
        path /tmp/badger_glyphy.com
      }
    }
  }
}

This stores a copy of the oembed responses on disk and returns it for five minutes without actually hitting the WordPress installation itself, which is both quicker and avoids taking it down. I tested this out. The access log records a few hits against the oembed URL followed by complete silence as Caddy returns a cache response to the rest of the requests. It’s kept this tiny site up during the test.

You can check if it’s working by looking at response headers in your browser’s developer tools. There should be something like:

cache-status: Souin; hit; ttl=291; key=GET-https-yoursite.com-/wp-json/oembed/1.0/embed?url=https://yoursite.com/your-post-url/

I haven’t messed around with any of the more advanced settings of the cache system yet, but there’re some useful options there, such as one that reduces the maximum size of the cache file on disk.

If you’re not using Caddy, you may have another caching mechanism available. The idea is just to configure the cache for the /wp-json/oembed/* URLs for a few minutes, which is generally how long the Mastodon traffic flood lasts. If you’re using a CDN it may already do this for you.

TL;DR Follow the instructions on https://github.com/faevourite/mastodon-oracle-cloud-free-tier.

When you join Mastodon you lend trust to whoever runs your server, including trust that they moderate the content, keep the service well-maintained, and back up its data. This is easy if you know the people running it! If no one in your friends/family group is running a server already, and you don’t mind the administrative responsibility that services like Masto.host save you, you can host your own without breaking the bank by leveraging Oracle Cloud’s generous free tier.

I did this recently to set up my personal instance! In addition to the reasons above, I wanted to own my data, have a @glyphy.com handle, and be able to make small tweaks to Mastodon itself (like adding custom server emoji). The last two reasons are admittedly all in service of my vanity. Mastodon can verify my identity via links in my profile, and I have writer’s block when it comes to the custom emoji (typist’s block?). But personalization is the soul of IndieWeb and I’m here for it.

While I could just create a free account on Oracle Cloud, spin up a 4-core 24GB ARM-based compute instance (free tier limit) using the console admin UI, and follow the official Mastodon installation instructions, I wanted something more maintainable and automated. If (when?) I mess up my instance beyond repair I’d like to be able to recover quickly.

Here’s what I ended up using to accomplish this:

• Docker for running everything
  • Core components: Mastodon apps, Postgres DB, and Redis cache
  • Caddy, to serve everything over TLS, with a certificate provisioned using its Cloudflare integration
  • Backups via Kopia
  • Healthchecks.io and Newrelic for monitoring (free tiers)
• Ansible to install and configure all of the above
• Terraform to provision the underlying cloud infrastructure
• Cloudflare (free tier again) to manage DNS and provide some bot protection
• Sendgrid (you guessed it, free tier) for Mastodon emails, such as password recovery
• Pushover for cron job failure notifications
  • This is the only thing that’s paid here (optional, though). $5 lifetime per device. It’s more than paid for itself over the years.

I put all the scripts and manifests together into this GitHub repository, along with instructions on how to get it all running.

Below are just some notes about the different choices, mostly so I myself remember them when they invariably turn out to be wrong.

Docker

I don’t want to deal with random OS portability issues or package conflicts. Docker also has a nice side effect of requiring me to think about where things will be stored. An inventory mandate.

One thing I’m worried about is the difficulty of future updates that require some manual steps in a specific order. Docker-compose’s “bring everything up together” behaviour spells danger here. I don’t mind a little bit of downtime on this personal instance, but it may be a bigger deal otherwise.

Caddy

Serves as few purposes:

1. Multiplexes the “web” and “streaming” containers over the same domain
2. Enables compression
3. Sets aggressive caching headers
4. With its Cloudflare DNS module it can provision a TLS cert, which means that Cloudflare->Mastodon traffic is also over HTTPS, and I can just block port 80 entirely

Kopia

As far as backup software goes, it’s relatively young, but I like it. I recently switched to it for some personal backups. It’s like a fancier restic. I point it at Google Drive via rclone (baked into Kopia’s Docker image). If/when I run of storage there I may move to Backblaze B2.

Healthchecks

This is a recent discovery. It’s like a dead-man’s switch for cron jobs, and has its own copious free tier. I have it integrated with Pushover and email, but it supports many other notification systems.

Ansible/Terraform

I first tried using Ansible to provision the infrastructure, but (slowly) realized why Terraform is preferred for this sort of work. The latter keeps track of state. With Ansible, I think I would’ve had to save the IDs of every piece of infra somewhere, and then read it back on startup, to avoid having it try to re-create what already exists.

Cloudflare

I turned on its proxying for my Mastodon domain. I was worried at first since they seem to mess with traffic initiated by systems, but so far I haven’t noticed any problems.

I don’t feel great about using a service that supports right-wing extremists. I’d like to move away to one that’s less harmful. For now, I just promise myself not to give them any of my money. I don’t mind turning off their bot protection, and Caddy is able to automatically refresh a TLS cert, so they’re really only managing DNS for me right now.

Sendgrid

Their customer service is atrocious and the product is stale. But I already had an account, so that’s what I went with. I turn off all email notifications from Mastodon itself except for password resets.

Overall, I’m very happy with how this all turned out. Maybe one day OCI will clamp down on its free tier or my account will get more popular (unlikely) and I’ll be forced to pay up for more infra, which is ok with me. I also like the idea of being the only one to blame when something goes wrong with my own instance.

It’s basically magic when it works!

I generally like the built-in diff in the JetBrains suite of IDEs. The one I use these days is GoLand, but I believe they all support adding an external diff tool. Since difftastic is a console app, here’s what I had to do on my Mac:

1. brew install difftastic # install the tool
2. Install ttab using their brew instructions. This allows GoLand to launch a new tab in iTerm and run the difft command there. Otherwise, using the External Diff Tool in Goland would appear to do absolutely nothing, as the output of the tool isn’t displayed natively.
3. Configure the external diff tool using the instructions for GoLand.
   1. Program path: ttab
   2. Tool name: Difftastic (but can be anything you like)
   3. Argument pattern: -a iTerm2 difft %1 %2 %3
      1. The “-a iTerm2” is to ensure that iTerm is used instead of the default Terminal app.

Now you can click this little button in the standard GoLand diff view to open up the structural diff if needed:

Ideally the diff would be integrated into GoLand, but I don’t mind it being an extra click away, since difftastic doesn’t work reliably in many situations (particularly large additions or refactorings).

Recording and alerting rules exist in a rule group. Rules within a group are run sequentially at a regular interval, with the same evaluation time.

Recording Rules

I read that a while ago, but at the time it wasn’t clear why it mattered. It seemed that groups were mostly intended to give a collection of recording rules a name. It became clear recently when I tried to set up a recording rule in one group that was using a metric produced by a recording rule in another group.

The expression for the first recording rule was something like this:

(
  sum(rate(http:requests[5m]))
  -
  sum(rate(http:low_latency[5m]))
)
/
(
  sum(rate(http:requests[5m]))
)

The result:

It’s showing a ratio of “slow” requests as a value from 0 to 1. Compare that graph to one that’s based on the raw metric, and not the pre-calculated one:

The expression is:

(
  sum(rate(http_requests_seconds_count{somelabel="filter"}[5m]))
  -
  sum(rate(http_requests_seconds_bucket{somelabel="filter", le="1"}[5m]))
)
/
(
  sum(rate(http_requests_seconds_count{somelabel="filter"}[5m]))
)

The metrics used here correspond to the pre-calculated ones above. That is, http:requests is http_requests_seconds_count{somelabel="filter"}, and http:low_latency is http_requests_seconds_bucket{somelabel="filter", le="1"}. The graphs are similar, but the one using raw metrics doesn’t have the strange sharp spikes and drops.

I’m not sure what’s going on here exactly, but based on the explanation from the docs it’s probably a race between the evaluation of the two groups resulting in inconsistent number of samples used for http:requests and http:low_latency. Maybe one has one less sample than the other at the time they’re evaluated for the first group’s expression, which I think could show up as spikes.

Whatever the cause the solution is simple: if one recording rule uses metrics produced by another make sure they’re in the same group.

I was trying to add this site to Indieweb ring last night and found that it couldn’t validate the presence of the previous/next links, even though they were clearly in the footer of every page. I cleared the WordPress and Cloudflare caches without success.

Since Indieweb ring runs on Glitch, which is a large public service, I suspected that maybe Cloudflare was blocking their traffic. Sure enough, checking my http access logs I couldn’t find any requests from Glitch, and switching nameservers to my web host resulted in a successful check:

This was happening even with the “Bot Fight” option set to off, “Security Level” set to “Essentially Off” and a disabled “Browser Integrity Check” option.

[side note] I love that my autogenerated site identifier for Indieweb ring is a person worried about taking pictures and writing. Accurate.

[Edit 2023-05-21] The webring doesn’t use emoji anymore 😢

Error running '<my test>':
Debugging programs compiled with go version go1.17.8 darwin/amd64 is not supported. Use go sdk for darwin/arm64.

Wait, amd64? I had installed the arm64 go package after the switch to M1, so why did it think I was running on amd64?

Turns out I had used homebrew on my old Intel Mac to install and use a newer version of my shell (bash). Because amd64 homebrew installs amd64 binaries the shell itself was running under amd64 and any app it ran would think the CPU architecture was also amd64. Running arch in the shell confirmed my suspicion.

The solution was to switch to the Mac’s built-in bash shell, reinstall homebrew, reinstall bash, and set iTerm to use /opt/homebrew/bin/bash instead of the default /bin/bash . I followed these instructions for switching to arm64 homebrew and kept the old Intel homebrew around aliased to oldbrew as suggested to still have access to amd64-only apps.

In her post Sandi writes about how the Sunk Cost Fallacy plays into the hesitation we as developers feel when encountering perplexing abstractions. Part of the recommendation is to consider that

“It may have been right to begin with, but that day has passed.”

I think this is often how we think about organizational decisions. We make new ones all the time, and often they alter or completely reverse those that came before, even if people who made the initial choices aren’t available for consultation anymore.

I suspect it’s easier for us to undo organizational decisions than code decisions because the former are made in a more visible and social environment. It’s easy to skip fixing the wrong abstraction when that choice may only be seen by one reviewer as part of an unrelated change. Perhaps this is where pair or mob programming can help. More eyes on the wrong abstraction at the right time could be all that’s needed to address it.

Thinking about code as a decision log could also help. Removing the abstraction is just another event. Events are bound to a specific moment in time, and maybe today is the right moment for an event that reverses some of those previous decisions.