I’m running the majority of my workloads on Kubernetes these days, including my Mastodon instance. For many years I’ve been running Mastodon in read-only mode for the container filesystem. However, I somehow missed the Sidekiq container.
This became obvious, when someone threw a question to
#mastoadmin about it.
Is it possible to make
#mastodonwrite its temporary files to somewhere else other than
/opt/mastodon, for those unaware, is the working directory and application directory of Mastodon in the official Mastodon containers. Why would the app use the current working directory instead of
/tmp/ for temporary files?
It turns out, it doesn’t. At least not by default and when the container filesystem is writeable. However, when adjusting the
securityContext of the container to contain
readOnlyRootFilesystem: true, media processing fails, as it tries to write temporary files.
No problem, that’s why
emptyDir exist. Mounting an
emptyDir volume to
/tmp should solve it, however, it doesn’t.
Instead of using
/tmp, suddenly mastodon tries to write its temporary files to
/opt/mastodon or rather the current working directory. Somehow there was a fallback to the current working directory, somewhere in the stack and it was triggered. Diving down into the stack, it became apparent, that it’s actually just the
paperclip dependency that causes the issue, and even further down, it turns out, it’s Ruby itself.
It turns out, Ruby assesses temp directories for suitability before using them:
One of the criteria is that it’s world writeable with a sticky bit or not world writeable at all. And here
emptyDir comes back into play.
emptyDirs in Kubernetes are creating world writeable directories without sticky bit and therefore this check fails and ruby falls back to its fallback directory, the current working directory.
So in Ruby
emptyDir is unsuitable for temporary files created using the native API.
Today I scrolled through
#mastoadmin and came back across the conversation about the problem. And today, I knew how to solve it! Meet Generic Ephemeral Volumes!
Generic Ephemeral Volumes are Kubernetes Persistent Volume Claims, that only exist for the lifetime of a Pod and are only attached to this Pod. PVCs have the nice benefit, that other than
emptyDir they aren’t mounted with world-writeable permissions, but rather just your regular
So the solution to the problem of
emptyDir being unsuitable for ruby temporary files is to use generic ephemeral volumes, to create PVCs that can be mounted under
/tmp and work like a
/tmp directory. Be aware, they do not share the exact attributes and capability of your regular
/tmp directory, but for this use-case, they are good enough.
PS: If you are curious about how I’m running Mastodon, you can have a look at the Helm Chart.
Following up on the conversation mentioned above, I dove into the kubelet source code¹ ² ³, to figure out how the kubelet creates it’s
emptyDir. There I learned that the behaviour with forced
0777 only exists for directory-based
emptyDir. Therefore if you don’t want to use Generic Ephemeral Volumes, you can also just use in-memory
- name: tmp emptyDir: medium: Memory
And after figuring that out, I decided to check the issues tracker, turns out, it’s a well-known bug, that is probably becoming a