Kubernetes volumes with rclone (the hacky way)

» Post Updates

2024-09-30: Added a second approach with using a sidecar container.
2024-09-30: Added a third approach with using nfsmount.

» Introduction

Kubernetes has a wide list of various CSI-drivers for mounting remote storage. I was looking for mounting Hetzner Cloud Object Storage, but unfortunately, the given options were not to my satisfaction. There are several options for mounting S3-compatible storage, but few of them offer transparent encryption and if they do, you are stuck with the tool. When I have to be stuck with a tool, why not a very popular one I can use almost everywhere? When I can mount volumes with rclone and its crypt module in Kubernetes, I can use the same settings for mounting the bucket on my other machines, and that was my goal. Unfortunately, there is no rclone CSI-driver yet, so I went with the hacky solution of directly using the host path of the Node.

There where two articles which guided me the way for this post: Kubernetes shared storage with S3 backend and Mounting S3 bucket in docker containers on kubernetes.

With the following example, we should be able to use all rclone supported remotes, such as S3, SFTP, SMB, etc.

» Approach 1: DaemonSet

With the DaemonSet you will be able to mount the same data in multiple pods across different namespaces.

» Deployment

» Namespace

First, let us create an own rclone namespace:

1
2
3
4
apiVersion: v1
kind: Namespace
metadata:
  name: rclone

The secret contains a rclone configuration as you would normally configure it. When using the encryption, you must point the remote to the unencrypted config entry.

» Secret

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
apiVersion: v1
kind: Secret
metadata:
    name: rclone
    namespace: rclone
stringData:
    rclone.conf: |
        [hetzner]
        type = s3
        provider = Other
        access_key_id = <ACCESS_KEY>
        secret_access_key = <SECRET_KEY>
        endpoint = https://<REGION>.your-objectstorage.com

        [hetzner-encrypted]
        type = crypt
        remote = hetzner:/<BUCKET>
        password = <RCLONE_PW>
        password2 = <RCLONE_PW_SALT>        

» DaemonSet

The DaemonSet will execute the rclone mount command on every node of your Kubernetes cluster, so no matter on which node a pod starts, the mount will be there.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: rclone
  namespace: rclone
  labels:
    k8s-app: rclone
spec:
  selector:
    matchLabels:
      name: rclone
  template:
    metadata:
      labels:
        name: rclone
    spec:
      containers:
        - name: rclone
          image: rclone/rclone:1.68.1
          command: ["rclone"]
          args:
            - "mount"
            - "hetzner-encrypted:/<MY_ENCRYPTED_FOLDER>"
            - "/data/<BUCKET>"
            - "--allow-non-empty"
          securityContext:
            privileged: true
          volumeMounts:
            - name: config 
              mountPath: /root/.config/rclone/
            - name: my-bucket
              mountPropagation: Bidirectional
              mountPath: /data/<BUCKET>
              readOnly: true
      volumes:
        - name: config
          secret:
            secretName: rclone
        - name: my-bucket
          hostPath:
            path: /var/mnt/rclone/<BUCKET>

Now one or more rclone Pods should be running which mounted the S3 bucket to your Nodes host path.

» Example Deployment

Finally, we can use this host post in other Pods to access our S3 bucket. Those pods don’t need to be in the same rclone namespace, as they are not referencing anything there, the only reference is the volume from the host path.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: alpine
  name: alpine
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alpine
  template:
    metadata:
      labels:
        app: alpine
    spec:
      containers:
        - image: alpine
          name: alpine
          args:
            - "sleep"
            - "infinity"
          volumeMounts:
            - mountPath: /my-bucket
              name: my-bucket
              mountPropagation: Bidirectional
          securityContext:
            privileged: true
      volumes:
        - name: my-bucket
          hostPath:
            path: /var/mnt/rclone/<BUCKET>

When you shell into the container, you should be able to see your unencrypted files in the /my-bucket directory.

» Troubleshooting

» Transport endpoint is not connected

When your pod failes to start with CreateContainerError, check the events at the bottom after executing:

1
kubectl describe pod -n rclone rclone-XYZ

When you see the follwing error:

1
Error: failed to generate container "xyz" spec: failed to generate spec: failed to stat "/var/mnt/rclone/<BUCKET>": stat /var/mnt/rclone/<BUCKET>: transport endpoint is not connected

It can happen that the mount was not properly unmounted when playing with the settings and restarting the pods from the DaemonSet. If so, shell into your node and unmount it manually (umount <PATH>).

It should be possible to fix this with a preStop lifecycle handler (inspired by kube-rclone which I found after writing the first version of this post).

» Approach 2: Sidecar Container

With the sidecar container you can mount the data only in this specific Pod by using an emptyDir volume.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mydeployment
  labels:
    name: mydeployment
spec:
  selector:
    matchLabels:
      name: mydeployment
  template:
    metadata:
      labels:
        name: mydeployment
    spec:
      containers:
        - name: rclone
          image: rclone/rclone:1.68.1
          command: ["rclone"]
          args:
            - "mount"
            - "hetzner-encrypted:/<MY_ENCRYPTED_FOLDER>"
            - "/data/<BUCKET>"
            - "--allow-non-empty"
          securityContext:
            privileged: true
          volumeMounts:
            - name: config
              mountPath: /root/.config/rclone/
            - name: my-bucket
              mountPropagation: Bidirectional
              mountPath: /data/<BUCKET>
              readOnly: true
        - name: other-container
          image: myimage
          volumeMounts:
            - name: my-bucket
              mountPropagation: Bidirectional
              mountPath: /<BUCKET>
          securityContext:
            privileged: true
      volumes:
        - name: config
          secret:
            secretName: rclone
        - name: my-bucket
          emptyDir: {}

» Approach 3: nfsmount

I’ve figured out that sharing the mount in Kubenetes performs very poorly when seeking in bigger files. I’ve tried various available caching options but with no success. I’m not sure if it is the decryption or something else. After spending several hours, I tried it with rclone nfsmount.

Due to lazyness, I will just paste the relevant parts of the manifest.

We adjust our rclone mount container to use nfsmount instead. I had to specify the mount options manually to add nolock.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
- name: rclone
  image: rclone/rclone:1.68.1
  command: ["rclone"]
  args:
    - "nfsmount"
    - "hetzner-encrypted:/<MY_ENCRYPTED_FOLDER>"
    - "/data/<BUCKET>"
    - "--addr=:37297"
    - "--option=port=37297,mountport=37297,tcp,nolock"
    - "-v"
    - "--read-only"
    - "--vfs-cache-mode=full"
  volumeMounts:
    - name: rclone-config
      mountPath: /root/.config/rclone/
    - mountPath: /data/<BUCKET>
      name: my-bucket
      readOnly: true
      mountPropagation: Bidirectional
  securityContext:
    privileged: true

This increased the performance drastically for my use case and is now how I would have expect it from the beginning. Unfortunately, this approach also comes with the disadvantage that NFS does not support inofity, so your applications won’t be able to detect changes based on this feature.

» Summary

When you try to access the data without the crypt module from rclone (e.g. with the minio client or the Hetzner Cloud Console), you will only see encrypted files and filenames.

With rclone we have a very versatile tool which does not only allow us to mount S3 compatible remotes, but all other supported protocols, too and with all available features such as encryption. Especially with the encryption feature, I didn’t want to use a tool which only runs on Kubernetes, but I can use it on other machines too, I just have to copy the rclone config.

While examining the deployment manifests, you have probably already seen a big bummer: We are running the containers in privileged mode. This is necessary for mounting on the host path, as well as for using the Bidirectional mount propagation. Privileged means, the container has full access to your node! This is a huge disadvantage and security concern, so you have to weigh up whether it is worth it for you.