The fundamental value proposition of Kubernetes is that it can provide an abstraction layer for distributed applications on any infrastructure. This depends on the intersection of two partial myths:
- Kubernetes consistently provides every essential service that distributed applications need, and
- Kubernetes can run equally well on different cloud footprints, in a datacenter, or on a single node.
There’s some reflection of the truth in these myths. Vanilla upstream Kubernetes provides many important primitives for distributed applications — but not everything — and individual Kubernetes distributions typically bundle services to address the gaps. While it is technically possible to run Kubernetes on a single node by installing a specialized distribution, most of the solutions are rough around the edges, and if you want true portability, you’ll need to run the same Kubernetes distribution on your workstation (or laptop) and in your datacenter.
Since I’m more interested in developing tools that could be ported to a variety of Kubernetes distributions than I am with developing an application that is absolutely reproducible across multiple footprints in a single organization without additional effort, I have the flexibility to choose any single-node distribution of Kubernetes for local use.
I’ve been impressed with the setup and user experience of microk8s for a long time and used to run it (in a VM) on my old MacBook. In this post, I’ll explain how I used microk8s to set up a data science development environment on my workstation, complete with GPU acceleration, Kubeflow, and a notebook image with RAPIDS.ai preloaded.
I then installed microk8s with snap. There are a few options for versions and channels, but as of late March 2021 the most stable for me was Kubernetes 1.20 (more on this in a bit).
microk8s ships with Calico enabled, and there is a longstanding bug in Calico that prevents it from finding network interfaces that contain
lo anywhere in their names. Since the wireless interface in my workstation is called
wlo2, I needed to change Calico’s environment to get it to work:
With CUDA 11.0 installed, I was able to enable GPU support in microk8s:
I could then verify that the pods started successfully:
and, once they had, that my node had been labeled properly:
You’ll want to see an allocatable resource of type
nvidia.com/gpu in that output, like this (replace “2” with the number of GPUs your workstation has installed):
I could then launch a simple job to verify that I was able to schedule pods to run on the GPU:
Kubeflow and notebooks
Now I was able to install Kubeflow itself (feel free to specify your favorite password in these instructions):
Once Kubeflow was up, I created a persistent volume to enable shared storage between my notebook servers and the host system:
In my case, this created a persistent volume called
willb-share so that I could mount
k8s-share from my home directory as a data volume on a Kubeflow notebook server.
The next step was to get RAPIDS.ai set up in a Kubeflow notebook image, since Kubeflow no longer ships a RAPIDS image. I could have installed individual libraries in a notebook container but there was an easier option. Since the RAPIDS project publishes a variety of Docker images, I could pick one of those as a starting point and ensure that the resulting image would be usable by Kubeflow, like in this Dockerfile:
The only interesting part of that recipe is
RUN ldconfig, which I found necessary so that the Python notebook kernels could find CUDA. I built that image locally and pushed it to an accessible repository so that I could use it while launching a new notebook server from the Kubeflow dashboard.
I often prefer to access my workstation remotely even if I’m at my desk. For running a regular Jupyter notebook server from the command line, this is just a matter of binding to
0.0.0.0 or setting up an SSH tunnel.2 Accessing services running on a single-node Kubernetes deployment is slightly more complicated, though.
First up, all requests will need to go through a load balancer (in this case, Istio). We can access Istio through an external IP, and we can find out which IP that is by inspecting the ingress gateway service:
However, simply connecting to my workstation and forwarding traffic to port 80 of Istio’s IP didn’t do me a lot of good; I found that I also needed to be able to connect to other cluster IPs in order to access the services Istio was exposing.3 To access all of these IPs remotely, we have a couple of options:
- Set up a dynamic proxy over SSH. By connecting with
ssh -D 9999 workstationand then configuring a local proxy to point to
localhoston port 9999, we can access anything that’s accessible from
workstation. This is an easy way to smoke-test a deployment but it isn’t an ideal long-term solution because it requires you to maintain a ssh connection to your workstation and it proxies everything though the workstation unless you explicitly configure the proxy.4 Relying on a dynamic proxy like this can also lead to confusing errors when the SSH connection is down and may be difficult or impossible to configure when using cellular data on a mobile device.
- Use a VPN-like service to relay traffic to given subnets. I use Tailscale for a personal VPN and configured my workstation as a relay node for traffic to cluster IP addresses. This means that if I can access cluster IPs from my workstation, I can also access them from any computer connected to my Tailscale account (whether or not I’ve connected over SSH first). This was very easy and it’s also possible to do with upstream WireGuard.
Once we’re correctly forwarding traffic, connecting to the Istio load balancer will show us the Kubeflow dashboard; clicking the different links on that page (e.g., to create a new notebook server) will send requests to the appropriate internal services.
Challenges and false starts
Sometimes knowing what didn’t work is more useful than knowing what did. In this section, I’ll briefly cover some problems I encountered along the way so you’ll know what to look out for.
Device plugin errors
With microk8s 1.19, I was unable to get the device plugin pod to run successfully and always got some variant of this error in my logs:
Loading NVML Failed to initialize NVML: could not load NVML library. If this is a GPU node, did you set the docker default runtime to nvidia?
This is a confusing error because microk8s uses containerd and not Docker. While many people seem to have run in to this error online, none of the recommended solutions worked for me. (I also tried specifying a newer version of the device plugin container image, which was also not successful.)
GPU operator errors
The beta release of microk8s 1.21 uses the NVIDIA GPU operator to manage GPU drivers. As of mid-March 2021, the GPU operator is not intended to work on nodes that already have GPU drivers installed, which makes it more suitable for provisioning new nodes or VMs and adding them to a cluster (its intended use case, to be fair) than for enabling GPU support for a single-node Kubernetes on a workstation.5
I was able to enable GPU support in microk8s 1.21 by first removing CUDA and GPU drivers from my system, but this was an unpalatable hack since I’d prefer to be able to manage system dependencies with a native package manager (and also to use the GPU and CUDA outside of Kubernetes). I also noticed that the GPU operator failed to start after I had rebooted my system, presumably because it had installed the drivers before and they loaded on boot.
Image pull failures
After installing microk8s 1.20, the Calico pod failed due to an image pull timeout. I was able to explicitly pull it using the bundled
ctr tool before restarting the pods:
While I wouldn’t recommend single-node Kubernetes to most machine learning practitioners (it still requires a lot of interaction with Kubernetes proper to get to a productive state or troubleshoot problems), Kubernetes provides some useful primitives for managing resources, isolating jobs, and making work reproducible. Furthermore, developing ML tools on Kubernetes ensures that they’ll be consumable in multiple contexts. The combination of microk8s and Kubeflow provides a relatively-painless way to get to a productive discovery environment with RAPIDS and GPUs. In future posts, I’d like to look at using my single-node Kubernetes deployment to orchestrate other machine-learning and data processing workloads.
Be warned that I’m almost certainly doing some basic administration tasks suboptimally – while I used Debian at a consulting gig in the late 1990s and briefly used Ubuntu in the public cloud in graduate school, my main Linux distributions have been RPM-based for over 25 years. I chose Ubuntu for this application because it offered frictionless installation of GPU drivers – but the long support cycle vis-à-vis other community Linux distributions is also a plus. ↩
This didn’t make a lot of sense to me, but individual Kubeflow dashboard components were exposed with wildcard DNS hostnames pointing to cluster IPs – and, if I couldn’t connect to the cluster IPs, it manifested as unusual “Page not found” errors from the Kubeflow dashboard. ↩
This is a totally sensible design decision since a single-node Kubernetes deployment is not anywhere near the primary audience for a tool like the GPU operator. However, the upcoming release of the GPU operator will support this workstation use case by allowing users to skip driver installation; microk8s will incorporate this fix as well. ↩