Kubernetes Operator

The BrewFS Operator provides a Kubernetes-native way to deploy the Redis + RustFS backend stack and to create managed BrewFS mount workloads. The current API group is storage.brewfs.io/v1alpha1 and contains two custom resources:

The Operator is not a CSI driver. It does not provide dynamic PV/PVC provisioning, CSI lifecycle integration, or in-place injection into an existing workload. It creates and manages its own mount and consumer workloads.

Prerequisites

• A Linux Kubernetes cluster with /dev/fuse on target nodes.
• Cluster policy that permits the mount workload to run privileged with SYS_ADMIN.
• Local host storage permitted when using hostMountPath.
• A pullable BrewFS image and Operator image.
• kubectl with Kustomize support.

The operator uses FUSE and, when exposing a host mount, mount propagation. Review Pod Security, admission, node selection, and hostPath policies before enabling it in a production cluster.

Install the Operator

Clone the repository and apply either the base controller or the full Kubernetes example overlay:

git clone https://github.com/brewfs/brewfs
cd brewfs/operator/brewfs-operator

# CRDs, RBAC, namespace, and the controller only
kubectl apply -k manifests

# Or: controller plus example BrewFSCluster and BrewFSMount resources
kubectl apply -k overlays/kubernetes

For a local minikube environment, the repository also includes:

kubectl apply -k overlays/minikube

Check that the controller is ready before applying custom resources:

kubectl -n brewfs-system get deployment brewfs-operator
kubectl get crd brewfsclusters.storage.brewfs.io brewfsmounts.storage.brewfs.io

1. Create a backend stack

BrewFSCluster creates a single Redis deployment and service, a single RustFS deployment and service, a RustFS credentials Secret, a data PVC, a bucket initialization Job, and a BrewFS configuration ConfigMap named <cluster-name>-brewfs-config.

apiVersion: storage.brewfs.io/v1alpha1
kind: BrewFSCluster
metadata:
  name: demo
spec:
  redis: {}
  rustfs:
    bucket: brewfs-data
    storageSize: 20Gi
  mountConfig:
    mountPoint: /mnt/brewfs
    chunkSize: 67108864
    blockSize: 4194304

Apply it and wait until the generated configuration is available:

kubectl apply -f cluster.yaml
kubectl get brewfscluster demo -o yaml
kubectl get configmap demo-brewfs-config

The BrewFSCluster status reports phase, message, generated Redis and RustFS service names, the bucket, and the configuration ConfigMap. Create a mount only after the cluster reaches Ready and status.configMap is populated.

2. Create a node-level mount

Use a DaemonSet with hostMountPath to mount once on each selected node. The Bidirectional mount propagation mode allows the FUSE mount performed in the mount container to appear at the host path.

apiVersion: storage.brewfs.io/v1alpha1
kind: BrewFSMount
metadata:
  name: demo-mount
spec:
  clusterRef:
    name: demo
  workloadKind: DaemonSet
  image: ghcr.io/ivanbeethoven/brewfs:latest
  imagePullPolicy: Always
  mountPath: /mnt/brewfs
  hostMountPath: /var/lib/brewfs/mounts/demo
  mountPropagation: Bidirectional
  configPath: /run/brewfs/config.yaml
  statePath: /var/lib/brewfs
  logLevel: brewfs=info
  nodeSelector:
    kubernetes.io/os: linux
  tolerations:
    - operator: Exists

kubectl apply -f mount.yaml
kubectl get brewfsmount demo-mount -o yaml
kubectl get daemonset demo-mount-mount

Without hostMountPath, the mount exists only inside the managed mount Pod. This is a good first step for FUSE and backend debugging. With hostMountPath, the mounted directory is exposed on the host node and can be consumed through a matching hostPath.

3. Let the Operator create a consumer workload

When hostMountPath is set, consumer can create a new application workload that receives the same path at consumer.mountPath. The Operator supports consumer Deployment, DaemonSet, and StatefulSet; a StatefulSet also receives a headless Service.

apiVersion: storage.brewfs.io/v1alpha1
kind: BrewFSMount
metadata:
  name: demo-mount
spec:
  clusterRef:
    name: demo
  workloadKind: DaemonSet
  image: ghcr.io/ivanbeethoven/brewfs:latest
  hostMountPath: /var/lib/brewfs/mounts/demo
  mountPropagation: Bidirectional
  consumer:
    workloadKind: Deployment
    mountPath: /data
    containers:
      - name: app
        image: busybox:1.36
        command: ["/bin/sh", "-ec"]
        args: ["while true; do date; ls -al /data; sleep 30; done"]
        resources:
          requests:
            cpu: 100m
            memory: 128Mi

The consumer spec supports labels and annotations, init containers, multiple containers, extra volumes, environment variables, resources, probes, security context, image pull secrets, node selectors, tolerations, and selected PodSpec settings. It is still a managed new workload: move the needed template fields into spec.consumer; do not expect the operator to mutate an existing application resource.

Operational status and troubleshooting

Inspect the custom resource status first:

kubectl get brewfscluster,brewfsmount
kubectl describe brewfscluster demo
kubectl describe brewfsmount demo-mount
kubectl -n brewfs-system logs deployment/brewfs-operator

Current limitations

• The API is v1alpha1; treat fields and behavior as subject to change.
• Redis and RustFS are reconciled as single-instance services, not a high-availability backend topology.
• The Operator does not implement CSI, dynamic provisioning, or a per-node runtime service.
• It does not modify existing Deployments, StatefulSets, or Pods in place.
• Host mounts depend on cluster security policy and node-level FUSE/mount propagation support.