DevPod on Kubernetes: turning devcontainer.json into a persistent remote workspace

DevPod is an open source workspace manager for reproducible development environments across Docker, Kubernetes, SSH hosts, and several cloud backends. This note documents a full Kubernetes-based remote development setup with DevPod, including persistent volume strategy, custom images, file sync, IDE integration, and the GPU issues that tend to burn the most time.

DevPod, from Loft Labs, separates environment definition from the infrastructure that runs it. The developer describes the environment in devcontainer.json, including the base image, toolchain, ports, and lifecycle hooks. DevPod then creates and manages the matching workspace on the selected Provider.

Three terms matter more than anything else:

• Provider: the infrastructure backend. DevPod supports Docker, Kubernetes, SSH, and several cloud platforms.
• Workspace: an isolated development environment instance, usually backed by a container or VM on the provider.
• devcontainer.json: a Dev Container specification file that defines the image, lifecycle hooks, port forwarding, and editor behavior.

Compared with GitHub Codespaces or Gitpod, DevPod is a client-side tool rather than a hosted platform. On a self-managed Kubernetes cluster, that means you keep control over networking, storage, security policy, and node placement.

When Kubernetes is the provider, DevPod creates a Pod to host the workspace. Most setups end up with three files:

1. devcontainer.json, which defines the image, workspace directory, forwarded ports, and lifecycle commands.
2. pod-manifest.yaml, which carries the Kubernetes-native parts such as security context, resource limits, and volume mounts.
3. An orchestration script such as devpod.sh, which wraps devpod up, file sync, and environment bootstrap. That script is usually the glue that makes the workflow tolerable.

A typical flow looks like this:

What matters is how devpod stop behaves. It removes the Pod but keeps the PVC. The next devpod up recreates the Pod and reattaches the same volume, so the data survives Pod recreation.

The simplest way to split environments is to keep a separate Pod manifest for each one and switch them in a wrapper script:

This lets each environment define its own node selectors, quotas, and security policy while still sharing one devcontainer.json and one base image.

Where you mount the PVC decides what survives a Pod rebuild.

Mount the PVC at the container's $HOME, for example /root. In most setups, that is the least painful option. There are a few reasons to prefer it:

• The IDE server side, such as VS Code Server or Cursor Server, installs itself under ~/.vscode-server or ~/.cursor-server. Those directories land on persistent storage automatically.
• Toolchain state such as ~/.nvm and ~/.local/bin does not need extra symlink work.
• Shell files such as ~/.bashrc also persist, so environment setup happens once instead of on every Pod restart.

If the PVC is mounted somewhere else, such as /workspace, you usually end up adding symlinks or reinstalling tooling whenever the Pod comes back.

DevPod manages the whole workspace lifecycle through the devpod CLI. These are the commands that tend to matter in daily use.

Add and configure the provider first:

stop only removes the Pod. Everything on the PVC, including extensions, toolchain state, and source code, stays in place. The next up recreates the Pod and reattaches the volume, so the environment comes back quickly.

The Kubernetes provider accepts extra parameters through --provider-option:

Option	Description
DISK_SIZE	PVC size, for example 50Gi or 300Gi.
POD_MANIFEST	Path to the custom Pod manifest.
KUBERNETES_NAMESPACE	Target namespace for workspace Pods.

devcontainer.json is the core Dev Container file. It defines the image, lifecycle hooks, forwarded ports, editor customization, and the rest of the workspace contract. DevPod fully supports that specification. The file usually lives at .devcontainer/devcontainer.json.

A full example for remote development on Kubernetes:

You can define the base container either by pointing directly at an image or by building one from a Dockerfile.

The image field accepts any OCI image, including Docker Hub, GHCR, or a private registry. For remote development on Kubernetes, a prebuilt image usually saves trouble. Baking the whole toolchain into the image cuts startup time from minutes to seconds.

If you need to customize the image, use build:

context defaults to ".", which means the directory that contains devcontainer.json. Setting it to ".." lets the Dockerfile reference files from the project root.

workspaceFolder is the directory the IDE opens by default after it connects. On Kubernetes, it usually makes sense to point it at the PVC mount, for example /root, so the workspace path and the persistent path are the same thing.

workspaceMount controls how local source code gets mounted into the container. It is useful in local Docker workflows. In remote Kubernetes workflows, it is often better to leave it empty. DevPod v0.6.x has a known issue in #1885 where .devpodignore can be ignored during streaming upload, which means a large workspace, including venv and node_modules, can get pushed in full. A custom rsync step gives you much better control.

The Dev Container spec defines six lifecycle hooks, in this order:

Each hook accepts three forms:

• String: executed through /bin/sh.
• Array: executed directly without a shell, which is safer.
• Object: multiple named commands executed in parallel, useful when several services need to start together.

A few practical rules help here:

• If all tools are already in the image, set onCreateCommand to "true" and skip it.
• postStartCommand is a good place for startup checks or light warmup.
• The waitFor field decides which phase must finish before the IDE attaches. The default is "updateContentCommand".

You can declare extensions and settings under customizations.vscode, and they are applied automatically when the IDE connects:

Extensions listed under extensions install automatically on first attach. With the PVC mounted at $HOME, you only pay that cost once. Settings defined here override local editor settings, which helps keep behavior consistent across a team.

Ports listed in forwardPorts are forwarded automatically after the IDE connects. When a service starts inside the container, you can usually hit it on local localhost without extra setup.

portsAttributes lets you define a label and behavior per port:

onAutoForward controls the first reaction when DevPod sees the port: "notify" shows a notification, "openBrowser" opens a browser, "silent" forwards quietly, and "ignore" does nothing. otherPortsAttributes sets the default for ports you did not list explicitly.

The Dev Container spec splits environment variables into two layers:

• containerEnv: set on the container itself, visible to all processes, and fixed for the life of that container.
• remoteEnv: only visible to IDE-launched processes such as terminals, tasks, and debuggers. This layer can reference ${containerEnv:VAR} and does not require a container rebuild when changed.

Both fields also support ${localEnv:VAR}, which reads an environment variable from the host, for example ${localEnv:HOME}.

Dev Container Features are reusable Dockerfile fragments distributed as OCI artifacts. The features field lets you add tools without editing the base image directly:

You can browse the available features at containers.dev/features. For Kubernetes-based remote development, though, baking tools into the base image is usually better than paying installation time on every new workspace. Features fit local Docker prototypes better than long-lived remote workspaces.

A few fields change how the container behaves at runtime:

Field	Default	Description
overrideCommand	true	Overrides the image command with an infinite loop so the container stays alive. This default usually makes sense for custom development images.
shutdownAction	stopContainer	What happens when the IDE closes. Options include stopContainer and none. For Kubernetes, none is often the better choice.
init	false	Uses tini as PID 1 to reap zombie processes.
privileged	false	Enables privileged mode. In Docker workflows this can be set here. In Kubernetes, it belongs in the Pod manifest.
containerUser	root or the Dockerfile USER	The user for all container operations.
remoteUser	same as containerUser	The user for IDE terminals and tasks. It can differ from containerUser.

String values in devcontainer.json can use these predefined variables:

Variable	Meaning
${localEnv:VAR_NAME}	Host environment variable, with optional default value syntax: ${localEnv:VAR:default}
${containerEnv:VAR_NAME}	Container environment variable, available only inside remoteEnv
${localWorkspaceFolder}	Workspace path on the host
${containerWorkspaceFolder}	Workspace path inside the container
${devcontainerId}	Stable unique identifier for the container

You can point the Dev Container image field at any public image, but for remote development on Kubernetes it is usually worth building a dedicated base image with the toolchain, language runtimes, and system libraries locked into image layers.

That pays off in a few ways:

• The Pod is usable as soon as it starts. You do not wait for onCreateCommand to install half the environment.
• Environment consistency improves because everyone shares the same image instead of replaying installation steps in slightly different conditions.
• When the Pod is rebuilt, the toolchain comes back with it. You are not depending on package manager availability at workspace creation time.

Good layering makes build caching much more effective. Put low-churn tools in lower layers and faster-moving pieces in upper layers. End each RUN block with apt-get clean && rm -rf /var/lib/apt/lists/* to keep layers smaller, and use --no-install-recommends to avoid pulling in packages you do not need.

The following example builds a development image with Python 3.11, common system tools, and the NVIDIA CUDA runtime:

Several design choices here matter:

• Add PPAs and GPG keys in layer 1, before update-alternatives changes the default Python. If you switch Python first, add-apt-repository can fail with No module named 'apt_pkg' because the apt_pkg binding expects the system Python.
• Keep NVIDIA tools and CUDA libraries in separate layers. That way a driver update only rebuilds one layer.
• Install nvidia-utils-xxx-server, not nvidia-utils-xxx. On Ubuntu, the latter can be a transitional dummy package without the actual nvidia-smi binary.
• Pick cuda-libraries-12-8, roughly 1.2 GB, instead of the full cuda-toolkit-12-8, which is closer to 10 GB. Most development environments need the runtime more often than the full compiler and debugger stack.

Once the image already contains the full toolchain, devcontainer.json becomes much simpler:

Setting onCreateCommand to "true" means there is nothing left to install at first startup. The Pod is ready immediately after creation.

The Pod manifest is the core Kubernetes-side configuration. It controls the things DevPod cannot express through devcontainer.json.

DevPod renders the Pod manifest as a template before it creates the Pod. These placeholders are commonly used:

Variable	Meaning
{{.WorkspaceId}}	Workspace name, often reused as the Pod name and label value.
{{.Image}}	Image declared in devcontainer.json.

Remote development containers often need looser permissions than production containers. These are the settings that come up most often:

Setting	Use	Risk
privileged: true	Docker-in-Docker, device access, debugging tools	Full access to host kernel capabilities
SYS_ADMIN	mount and cgroup operations	Medium
SYS_PTRACE	strace, gdb, and similar debugging	Low
NET_ADMIN	Network debugging and iptables work	Medium
hostNetwork: true	Direct use of the host network stack, which avoids CNI overhead	Port conflicts and loss of network isolation
hostPID: true	Inspect host processes for system-level debugging	Loss of process isolation

Loosen permissions only where the workspace actually needs them, and keep these Pods isolated to dedicated namespaces or nodes so they do not interfere with production workloads.

Set requests low enough to keep scheduling realistic, and limits high enough to leave room for bursts. Development environments rarely sit at peak usage all day, but builds and test runs can spike hard for a short time.

DevPod includes a built-in sync path through devpod up, and it works fine for small projects. On large multi-repo workspaces, with dozens of subprojects and millions of files, two problems show up fast:

• The first sync can take a very long time, and exclusion control is limited.
• DevPod may try to upload the entire workspaceFolder, including directories you do not want remotely, such as node_modules and .git.

The usual way around this is to launch DevPod with --ide none, skip automatic sync, and then run your own rsync command with explicit include and exclude rules.

Even with --ide none, DevPod still tries to sync the local directory that matches workspaceFolder during devpod up. If that directory is large, the initial startup can still crawl. One workaround is to create a temporary empty directory and use that for the initial workspace creation:

The most useful flags here are:

• -az: archive mode plus compression. Do not add --progress when you have a large number of small files. The extra output can slow the SSH stream badly enough to trigger a broken pipe.
• --copy-unsafe-links: dereferences symlinks that point outside the synced tree. In multi-repo setups this is useful because shared directories linked from elsewhere often do not resolve correctly on the remote side.
• --exclude: keep anything noisy or disposable out of the remote workspace. .vscode/sessions.json changes constantly and tends to fight with remote state, so it should stay out.

VS Code and Cursor run remote development by installing a server-side component inside the container. The local editor talks to that server through SSH.

The server build has to match the local IDE version, usually by commit hash. The installation flow is usually:

1. Read the current commit hash from the local IDE.
2. Download the matching server bundle.
3. Transfer and unpack it to ~/.cursor-server/cli/servers/Stable-{commit}/ on the remote side.

The wrapper script should make installation idempotent:

Extensions live under ~/.cursor-server/extensions/ or ~/.vscode-server/extensions/. If the PVC is mounted at $HOME, those extensions persist automatically.

A common mistake is wiping the whole ~/.cursor-server directory during a server reinstall. That blows away every installed extension. The safer cleanup target is the server binary directory only:

When you first prepare a remote environment, it can be faster to sync already installed local extensions than to redownload everything from the marketplace:

After the sync, check for broken symlinks. Some extensions include links to a local Node.js path that does not exist remotely. Those need to be replaced with real files.

DevPod SSH sessions can drop when left idle because the server's SSH daemon, a firewall, or a load balancer between the client and the pod times out the connection. The standard fix is to enable SSH keepalive on the client side:

ServerAliveInterval 60 sends a keepalive packet every 60 seconds. ServerAliveCountMax 10 allows up to 10 consecutive missed responses before the client closes the connection. That combination keeps the tunnel alive through typical idle timeouts and handles pauses of up to roughly 10 minutes.

For sessions opened through scripts, add the options as flags:

For Cursor remote connections, the keepalive must be in ~/.ssh/config rather than a command flag, because Cursor manages the underlying SSH process itself and does not expose extra flags to the user.

The first IDE attach to a fresh workspace often takes anywhere from 30 seconds to several minutes because the IDE still has to:

• Establish the SSH tunnel, which adds some overhead through DevPod's SSH layer.
• Download and install the server if it is not already present.
• Initialize the installed extensions.

Later connections are much faster because the server and extensions are already sitting on the PVC.

GPU access on Kubernetes depends on several moving parts, including the host driver, the device plugin, and the container runtime hook. If any one of them is wrong, the container will come up without usable GPU devices.

The NVIDIA Device Plugin runs as a DaemonSet on GPU nodes and registers the extended resource nvidia.com/gpu with Kubernetes. A Pod requests GPUs by declaring the count in resources.limits:

The scheduler places the Pod on a node with enough GPU capacity, and the device plugin injects the actual device nodes such as /dev/nvidia0.

Requesting GPU resources is not enough. Kubernetes also has to know which container runtime class should handle GPU device setup. That happens through the Pod's runtimeClassName field:

If you omit runtimeClassName, the Pod may still get GPU quota, but the runtime will not call NVIDIA's prestart hook. The result is simple: no /dev/nvidia* devices inside the container. This is one of the most common failure modes.

privileged: true does not mean AppArmor is unconfined. On nodes with AppArmor enabled, a privileged container can still be blocked by the default profile, such as cri-containerd.apparmor.d, when it tries to access GPU device nodes.

The fix is to declare an unconfined AppArmor profile in the Pod annotations:

Here devpod is the container name. The annotation must match it exactly.

It is tempting to set NVIDIA_VISIBLE_DEVICES=all in the Pod manifest to expose every GPU. In a setup that already uses runtimeClassName: nvidia, that usually backfires. A manually set value can interfere with the device plugin's own injection logic.

The NVIDIA container runtime behaves like this:

• If NVIDIA_VISIBLE_DEVICES comes from the device plugin, the runtime mounts exactly the devices that value names.
• If the manifest hardcodes NVIDIA_VISIBLE_DEVICES=all, that value overrides the plugin-managed one and can break the mapping step.

The safer approach is to leave NVIDIA_VISIBLE_DEVICES alone and let the device plugin manage it. Keeping NVIDIA_DRIVER_CAPABILITIES=all is fine if the container needs full driver capability access.

nvidia-smi is the fastest way to confirm GPU visibility. There is one common trap when you install it inside the container: on some Linux distributions, packages named nvidia-utils-xxx are only transitional dummy packages. They install successfully but do not include the real nvidia-smi binary.

On Ubuntu 22.04, the reliable path is:

1. Add ppa:graphics-drivers/ppa.
2. Install nvidia-utils-xxx-server, with the -server suffix.

If changing the image is inconvenient, one temporary workaround is to mount host driver tools and libraries into the container with hostPath:

After startup, add /host/usr/lib/x86_64-linux-gnu to LD_LIBRARY_PATH and call /host/usr/bin/nvidia-smi directly. It works, but it is still a workaround. The long-term fix is to bake the required driver tools into the image.

If nvidia-smi returns Failed to initialize NVML: Unknown Error, check things in this order:

1. AppArmor. Confirm the Pod annotation is unconfined, and inspect the actual container profile with cat /proc/1/attr/current.
2. Device nodes. Check whether ls /dev/nvidia* returns anything. If the files exist but opening them returns EPERM, the cgroup device filter is the problem, not the driver.
3. Runtime class. Confirm the Pod spec sets runtimeClassName: nvidia and that the cluster actually has that RuntimeClass.
4. Environment variables. Verify that NVIDIA_VISIBLE_DEVICES was not overridden manually.
5. Driver versions. Make sure the user-space NVIDIA libraries in the container are compatible with the host kernel driver.
6. containerd privileged_without_host_devices. If the cluster uses nvidia-container-runtime as the default runtime and this flag is false, privileged pods that do not request nvidia.com/gpu will see device files in /dev but be blocked by the eBPF cgroup program. See the next section.

A development pod sometimes skips the nvidia.com/gpu resource request entirely, for example when the node already runs an inference service and the workspace wants to share the hardware without holding a scheduler slot. That approach works until the cluster enables GPU time-slicing.

A common part of the time-slicing setup is replacing the default containerd runc binary with nvidia-container-runtime:

When this is in place and the containerd setting privileged_without_host_devices is false, privileged containers no longer inherit host /dev automatically. The nvidia-container-runtime attaches an eBPF cgroup program that controls device access. For pods without a device plugin allocation, that program blocks /dev/nvidiactl and friends at the open syscall level.

The device files appear in ls /dev because the runtime still creates their directory entries. But opening them returns EPERM, and NVML fails immediately:

The error is at the kernel cgroup level, not in the userspace library. The same block applies even if you run the host's own nvidia-smi binary via chroot /host nvidia-smi because the eBPF program acts on any process in that container's cgroup.

The fix is one line in /etc/containerd/config.toml:

After updating the file, containerd must be restarted. If you cannot SSH directly to the node, a temporary privileged Job that mounts the host root is the standard approach. Use an image that is already cached on the node and avoid pulling from a registry:

The pod must be recreated after containerd restarts. Stop the workspace with devpod stop before applying the config change. The PVC is not touched by any of these steps.

Symptom	Cause	Fix
Pod enters Dead or Failed state	OOM, node issues, or a bad manifest	Run devpod stop, fix the manifest, then run devpod up again. The PVC stays intact.
SSH exits with code 255	The Pod is not ready yet, or the SSH tunnel dropped	Check Pod state and retry after it reaches Running. If server installation was interrupted, rerun the installation step manually.
rsync reports Broken pipe	Progress output flooded the SSH channel	Use rsync -az without --progress or --info=progress2.
add-apt-repository fails with No module named 'apt_pkg'	The default Python was switched before repository setup	Add all PPAs before calling update-alternatives.
IDE extensions disappear after a Pod rebuild	The reinstall script removed the extensions directory	Delete only the cli/ subtree and keep extensions/.
nvidia-smi: command not found	A transitional dummy package was installed	Install nvidia-utils-xxx-server from ppa:graphics-drivers/ppa.
NVML Unknown Error	AppArmor, runtime class, device injection, environment override, or cgroup eBPF block from privileged_without_host_devices = false	Try opening /dev/nvidiactl in Python. EPERM means cgroup block. Set privileged_without_host_devices = true in containerd config and restart. Otherwise debug: AppArmor, device nodes, runtimeClassName, then environment variables.
/dev/nvidia* does not exist	Missing runtimeClassName: nvidia or a broken device plugin	Confirm the RuntimeClass exists and the device plugin DaemonSet is healthy.