JuiceFS Operator
JuiceFS provides an Operator, which is a controller designed specifically for Kubernetes environments to automate the management of JuiceFS' distributed cache clusters, cache preloading, and data synchronization, making it easier to use JuiceFS in container environments.
Install the JuiceFS Operator
Install Helm and add the JuiceFS Helm chart repository:
helm repo add juicefs https://juicedata.github.io/charts/
helm repo update
Before installation, read values.yaml to learn all available configuration items. If modifications are needed, create a new values file (for example, values-mycluster.yaml) and include the parts you want to modify. If you need to install the operator on multiple Kubernetes clusters, create separate values files for each cluster configuration.
# Modify values-mycluster.yaml as needed
helm upgrade --install juicefs-operator juicefs/juicefs-operator -n juicefs-operator --create-namespace -f values-mycluster.yaml
You can use kubectl wait to wait until the operator is ready:
kubectl wait -n juicefs-operator --for=condition=Available=true --timeout=120s deployment/juicefs-operator
Update JuiceFS Operator
If you need to update the Operator, you can use the following commands:
helm repo update
helm upgrade juicefs-operator juicefs/juicefs-operator -n juicefs-operator --reuse-values
Due to Helm's limitations, CRDs are not updated together when upgrading, so please manually update the CRDs after updating the Operator:
export CHART_VERSION=$(helm show chart juicefs/juicefs-operator | grep appVersion | awk '{print $2}')
kubectl apply -f https://raw.githubusercontent.com/juicedata/juicefs-operator/refs/tags/v${CHART_VERSION}/dist/crd.yaml
Once the JuiceFS Operator is installed, you can start creating and managing cache groups. The operations introduced in the following sections can be completed through both the CSI Dashboard (version 0.25.3 or above) and kubectl. Choose the method you prefer. To simplify the documentation examples, only the kubectl method will be introduced.

Cache group
Enterprise users can use the JuiceFS Operator to create and manage distributed cache clusters. Compared to other deployment methods, the JuiceFS Operator is more convenient to use (supporting both GUI and CLI) and also supports advanced features such as different node configurations, smooth scaling, and automatic cache cleaning.
Create a cache group
Refer to the following example to save the cache group configuration as a YAML file (for example, juicefs-cache-group.yaml). This example deploys a distributed cache on all nodes with the juicefs.io/cg-worker: "true" label (you can set any label you like). For more configuration options, refer to the Cache group configurations section.
apiVersion: v1
kind: Secret
metadata:
name: juicefs-secret
namespace: juicefs-cache-group
type: Opaque
stringData:
name: juicefs-xx
token: xx
access-key: xx
secret-key: xx
# envs: '{"BASE_URL": "http://<IP or HOST>/static"}'
---
apiVersion: juicefs.io/v1
kind: CacheGroup
metadata:
name: cachegroup-sample
namespace: juicefs-cache-group
spec:
secretRef:
name: juicefs-secret
cacheGroup: juicefs-cache-group-cachegroup-sample # Custom cache group name, default is `${NAMESPACE}-${NAME}`
worker:
template:
nodeSelector:
juicefs.io/cg-worker: "true"
image: juicedata/mount:ee-5.4.4-d52f3e9
opts:
- cache-size=204800
- free-space-ratio=0.01
- group-weight=100
cacheDirs:
- type: HostPath
path: /mnt/cache
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 1
memory: 1Gi
Then create the cache group using the kubectl apply command:
kubectl apply -f juicefs-cache-group.yaml
If the Kubernetes nodes do not have the juicefs.io/cg-worker: "true" label, add this label:
kubectl label node node1 juicefs.io/cg-worker=true
Check cache group status
Use the following command to check the cache group status and confirm that the cache group is in the "Ready" state:
kubectl get cachegroups -n juicefs-cache-group
NAME CACHE GROUP PHASE BACK UP WAITING DELETED CACHE SIZE CACHE CAPACITY READY AGE
cachegroup-sample juicefs-cache-group-cachegroup-sample Ready <none> <none> <none> <none> 1/1 10s
The status columns show the scaling progress:
BACK UPis the number of Workers in the temporary join phase (commission or group-backup), not the number of cache replicas.WAITING DELETEDis the number of Workers migrating their cache before deletion.CACHE SIZEandCACHE CAPACITYare the cache usage and capacity reported by the Workers. They may be empty until the first statistics sample is collected.
Use the cache group
After completing the above steps, a JuiceFS distributed cache cluster has been started in Kubernetes, with the cache group name juicefs-cache-group-cachegroup-sample. To allow the JuiceFS client of the application to use this cache cluster, the JuiceFS client needs to join this cache group and add the --no-sharing mount option. This way, the JuiceFS client of the application joins the cache group but does not participate in cache data construction, avoiding cache data instability caused by frequent client creation and destruction.
For dynamic configuration, modify the mount options as shown below. For more information on how to adjust mount configurations, see Mount options.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: juicefs-sc
mountOptions:
- cache-group=juicefs-cache-group-cachegroup-sample
- no-sharing
Add and delete cache nodes
JuiceFS Operator supports smooth scaling of cache nodes, ensuring that adjustments do not significantly impact cache hit rates. With JuiceFS Operator, do not delete cache Worker Pods directly; change the cache group membership and let the Operator migrate the cache. It uses the --commission and --decommission mount options. For how these options work and their version requirements, see Cache group member change.
In the Create a cache group example, Kubernetes nodes must have the juicefs.io/cg-worker: "true" label. Therefore, adding or deleting cache nodes involves adding or removing this label from Kubernetes nodes. If the cache group uses spec.replicas, adjust the replica count instead; the same smooth scaling logic applies. For example, use the kubectl command to add or delete nodes:
# Add nodes
kubectl label node node1 juicefs.io/cg-worker=true
kubectl label node node2 juicefs.io/cg-worker=true
# Delete nodes
kubectl label node node1 juicefs.io/cg-worker-
When the node membership changes, JuiceFS Operator smoothly adds or removes Workers:
-
Add nodes. The Operator creates new Workers and temporarily adds mount options to support a smooth join:
- With a JuiceFS Enterprise Edition 5.4.4 or later cache image, it uses
--commission. The metadata service must be 5.4.3 or later. When the new Worker does not yet have the data it is responsible for, read requests are served by the node that owned the data before scaling. This reduces reads from object storage. - If the cache node image is older than 5.4.4 but version 5.1 or later, the Operator falls back to
--group-backup. Images older than 5.1 do not receive an automatically added join option. - The join option is temporary.
spec.backupDurationcontrols how long it remains in effect (1 hour by default). In current versions, the Operator may also end the join phase early if the Worker's cache receive rate falls below the rebalance idle threshold. - The Operator removes the join option by restarting the Worker with updated mount options. This relies on the online mount option update capability introduced in version 5.4.3. However, the Operator currently uses this mechanism only with Enterprise Edition 5.4.4 or later images. Workers running older images are recreated instead.
- When multiple nodes are added at once, the Operator may create Workers in batches. It waits for the current join batch to finish before creating the next batch.
- Background data rebalancing starts about 10 minutes after the membership change, consistent with the normal behavior of a cache group.
- With a JuiceFS Enterprise Edition 5.4.4 or later cache image, it uses
-
Remove nodes. The Operator drains the cache on a Worker before deleting it:
- If the cache node image is Enterprise Edition 5.4.4 or later, the Operator restarts the Worker with
--decommission. For older images, it falls back to--group-weight=0. - The Operator waits until the Worker's cache migration rate falls below the rebalance idle threshold or
spec.waitingDeletedMaxDurationexpires (1 hour by default). It then deletes the Worker. - If the Worker has no cached data, is not ready, or is the last ready Worker, the Operator deletes it directly.
- If the cache node image is Enterprise Edition 5.4.4 or later, the Operator restarts the Worker with
To clean the cache directory when a Worker is deleted, set spec.cleanCache. See Clean cache when deleting a node.
The rebalance idle threshold defaults to 1 MiB/s (1,048,576 B/s). The Operator uses it to determine whether cache migration has finished during a join or removal. You can change it with the Operator's --cache-group-rebalance-idle-rate flag (specified in bytes per second). This is an Operator-level setting, not a field of the CacheGroup resource.
In most cases, do not manually add --commission, --group-backup, or --decommission to worker.template.opts. The Operator manages these temporary options for smooth joins and removals. If you explicitly configure any of these options, the Operator does not automatically add other options for a smooth join.
The following example sets the temporary join phase to 30 minutes and the maximum wait time before deleting a Worker to 2 hours:
apiVersion: juicefs.io/v1
kind: CacheGroup
metadata:
name: cachegroup-sample
namespace: juicefs-cache-group
spec:
backupDuration: 30m
waitingDeletedMaxDuration: 2h
To use --commission and --decommission automatically, use a JuiceFS Enterprise Edition 5.4.4 or later image for the cache Workers. The cache group examples in this document use a 5.4.4 image.
Cache group configurations
All supported cache group configurations can be found in the complete example.
Specify Worker Replicas Added in v0.6.0
You can specify the number of worker replicas in the cache group by setting the spec.replicas field:
- The worker management mode cannot be changed:
replicasmust be set when aCacheGroupis created. It cannot be added later to an existingCacheGroupthat does not have this field, nor can it be removed once set. However, you can change its value to scale the number of Workers. - When using this method, ensure that Pod IPs are fixed and the cache disk can follow Pod migration to other nodes, otherwise it may lead to cache penetration.
- The
worker.overwritefield will not be applicable in this mode, meaning different nodes cannot have different configurations.
apiVersion: juicefs.io/v1
kind: CacheGroup
metadata:
name: cachegroup-sample
namespace: juicefs-cache-group
spec:
replicas: 3 # Specify to create 3 worker replicas
worker:
template:
nodeSelector:
juicefs.io/cg-worker: "true"
image: juicedata/mount:ee-5.4.4-d52f3e9
opts:
- cache-size=204800
- free-space-ratio=0.01
- group-weight=100
cacheDirs:
- type: VolumeClaimTemplates
volumeClaimTemplate:
metadata:
name: jfs-cache
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: <your-storage-class-name>
This way, you can precisely control the number of workers in the cache group instead of relying on the number of node labels.
Affinity and anti-affinity Added in v0.7.2
By default, the JuiceFS Operator deploys workers on all nodes that match the nodeSelector without following Node and Pod affinity and anti-affinity rules.
Starting from version v0.7.2, the JuiceFS Operator supports enabling scheduling functionality through the spec.enableScheduling field.
For example, to deploy cache groups in different zones.
Only process the requiredDuringSchedulingIgnoredDuringExecution rule.
apiVersion: juicefs.io/v1
kind: CacheGroup
metadata:
name: cachegroup-sample
namespace: juicefs-cache-group
spec:
secretRef:
name: cachegroup-sample-secret
enableScheduling: true
worker:
template:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: juicefs.io/cache-group
operator: In
values:
- cachegroup-sample
topologyKey: "topology.kubernetes.io/zone"
Update strategy
When updating the cache group configuration, you can specify the update strategy for the worker nodes under the cache group using the spec.updateStrategy field.
Currently supported strategies are:
RollingUpdate(default): This is the default update strategy. When using theRollingUpdatestrategy, after updating the cache group template, the old Worker Pods will be terminated, and new Worker Pods will be automatically created. The number of updates at a time follows thespec.updateStrategy.rollingUpdate.maxUnavailableconfiguration, which defaults to 1.OnDelete: When using theOnDeletestrategy, after updating the cache group template, new Worker Pods will only be created when you manually delete the old Worker Pods.
apiVersion: juicefs.io/v1
kind: CacheGroup
metadata:
name: cachegroup-sample
namespace: juicefs-cache-group
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
Cache directory
The cache directory can be set using the spec.worker.template.cacheDirs field. Supported types are HostPath, PVC and VolumeClaimTemplates Added in v0.6.0.
apiVersion: juicefs.io/v1
kind: CacheGroup
metadata:
name: cachegroup-sample
namespace: juicefs-cache-group
spec:
worker:
template:
nodeSelector:
juicefs.io/cg-worker: "true"
image: juicedata/mount:ee-5.4.4-d52f3e9
cacheDirs:
- type: HostPath
path: /var/jfsCache-0
- type: PVC
name: juicefs-cache-pvc
# v0.6.0 and above support VolumeClaimTemplates
- type: VolumeClaimTemplates
volumeClaimTemplate:
metadata:
name: jfs-cache
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: <your-storage-class-name>
Use a block device as a cache disk Added in v0.9.0
Cache directories of the PVC and VolumeClaimTemplates types can use Kubernetes raw block volumes. The StorageClass and its CSI Driver must support raw block volumes. HostPath only supports directories and cannot be used in this mode.
When referencing an existing PVC, both spec.volumeMode in the PVC and cacheDirs[].volumeMode must be set to Block.
spec:
worker:
template:
cacheDirs:
- type: PVC
name: block-cache-pvc
volumeMode: Block
format: true
When using VolumeClaimTemplates to dynamically create a block volume, set volumeMode: Block in the PVC template:
spec:
worker:
template:
cacheDirs:
- type: VolumeClaimTemplates
format: true
volumeClaimTemplate:
metadata:
name: block-cache
spec:
volumeMode: Block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: <your-storage-class-name>
For multiple workers, use VolumeClaimTemplates to create a separate PVC for each worker, or use worker.overwrite to assign a different PVC to each node.
format is only valid for block volumes and defaults to false. If the block device does not contain a recognizable file system, the worker exits without modifying the device by default. When set to true, if the Operator does not detect a recognizable file system or other signatures, it formats the device as ext4. Signature detection cannot prove that a device is empty, so unrecognized data may still be erased. Enable this option only for a dedicated cache disk that you have confirmed can be erased.
Specify different configurations for different nodes
Cache nodes may have heterogeneous configurations (for example, different cache disk sizes). In this case, you can specify different configurations for different nodes using the spec.worker.overwrite field:
apiVersion: juicefs.io/v1
kind: CacheGroup
metadata:
name: cachegroup-sample
namespace: juicefs-cache-group
spec:
worker:
template:
nodeSelector:
juicefs.io/cg-worker: "true"
image: juicedata/mount:ee-5.4.4-d52f3e9
hostNetwork: true
cacheDirs:
- path: /var/jfsCache-0
type: HostPath
opts:
- group-weight=100
overwrite:
- nodes:
- k8s-03
# You can also use nodeSelector
# nodeSelector:
# kubernetes.io/hostname: k8s-02
opts:
- group-weight=50
cacheDirs:
- path: /var/jfsCache-1
type: HostPath
- path: /var/jfsCache-2
type: HostPath
Mount options
Mount options can be set using the spec.worker.template.opts field. Refer to the documentation for all mount options.
apiVersion: juicefs.io/v1
kind: CacheGroup
metadata:
name: cachegroup-sample
namespace: juicefs-cache-group
spec:
worker:
template:
nodeSelector:
juicefs.io/cg-worker: "true"
image: juicedata/mount:ee-5.4.4-d52f3e9
opts:
- group-weight=100
Cache group name
The JuiceFS Operator generates default cache group names in the
${NAMESPACE}-${NAME} format. If you want to customize the cache group name, you can set it using the spec.cacheGroup field:
apiVersion: juicefs.io/v1
kind: CacheGroup
metadata:
name: cachegroup-sample
namespace: juicefs-cache-group
spec:
cacheGroup: jfscachegroup
Clean cache when deleting a node
When deleting a node, you can specify whether to clean the cache using the spec.cleanCache field:
apiVersion: juicefs.io/v1
kind: CacheGroup
metadata:
name: cachegroup-sample
namespace: juicefs-cache-group
spec:
cleanCache: true
Delete a cache group
Use the following command to delete the cache group. All worker nodes under the cache cluster will be deleted:
kubectl delete cachegroup cachegroup-sample -n juicefs-cache-group
Warmup Cache Group
Operator supports creating a WarmUp CR to warmup the cache group.
apiVersion: juicefs.io/v1
kind: WarmUp
metadata:
name: warmup-sample
namespace: juicefs-cache-group
spec:
cacheGroupName: cachegroup-sample
# If targetsFrom is not configured, the entire file system is warmed up by default
targetsFrom:
files:
- /a
- /b
- /c
# Options for the juicefs warmup command; do not include the leading "--"
# See https://juicefs.com/docs/cloud/reference/command_reference/#warmup
options:
- threads=50
spec.targetsFrom supports the following three ways of providing a file list. Only one is used at a time:
files: List paths directly in theWarmUpresource.configMap: Specify thenameandkeyof the ConfigMap entry that contains the file list.filePath: Specify the path to an existing file list in the JuiceFS file system.
The legacy spec.targets field is deprecated. Use spec.targetsFrom.files instead.
To run cache warmup periodically, set the policy to Cron:
spec:
policy:
type: Cron
cron:
schedule: "*/5 * * * *"
suspend: false
Warm up an external cache group Added in v0.8.1
If the distributed cache group is not managed by a CacheGroup resource, specify the JuiceFS authentication Secret through spec.secretRef and explicitly set spec.image. The WarmUp resource and Secret must be in the same namespace. In this case, set cacheGroupName directly to the name of the distributed cache group to join.
apiVersion: juicefs.io/v1
kind: WarmUp
metadata:
name: warmup-external
namespace: juicefs-cache-group
spec:
cacheGroupName: existing-cache-group
secretRef:
name: juicefs-secret
image: juicedata/mount:ee-5.4.4-d52f3e9
mountOptions:
- no-update
options:
- threads=50
targetsFrom:
files:
- /dataset
If the image is hosted in a private registry, configure pull credentials through spec.imagePullSecrets. Any additional Secrets declared through configs in the authentication Secret are also mounted into the WarmUp job.
Sync
Operator supports quickly creating a distributed Sync task.
For example, to sync data from OSS to JuiceFS, you can refer to the following example:
apiVersion: juicefs.io/v1
kind: Sync
metadata:
name: sync-test
namespace: default
spec:
# Expected number of workers, default is 1
# meaning single-node synchronization
replicas: 3
options:
- debug
- threads=10
image: registry.cn-hangzhou.aliyuncs.com/juicedata/mount:ee-5.4.4-d52f3e9
from:
external:
uri: oss://sync-test.oss-cn-hangzhou.aliyuncs.com/sync-src-test/
# Two ways to provide the credentials, choose either value or valueFrom
accessKey:
value: accessKey
secretKey:
valueFrom:
secretKeyRef:
name: sync-test-secret
key: secretKey
to:
juicefs:
path: /sync-test/demo2/
token:
valueFrom:
secretKeyRef:
name: sync-test-secret
key: token
volumeName: sync-test
For each of from and to, choose one endpoint type from external, juicefs, or juicefsCE. The source and destination URIs must either both end with / or both omit the trailing /.
Starting from version v0.5.0, data synchronization for community edition JuiceFS is supported:
Currently, synchronization between community edition and enterprise edition is not supported.
apiVersion: juicefs.io/v1
kind: Sync
metadata:
name: sync-ce-test
namespace: default
spec:
replicas: 3
image: juicedata/mount:ce-v1.4.0
from:
external:
uri: oss://sync-test.oss-cn-hangzhou.aliyuncs.com/sync-src-test/
# Two ways to provide the credentials, choose either value or valueFrom
accessKey:
value: accessKey
secretKey:
valueFrom:
secretKeyRef:
name: sync-test-secret
key: secretKey
to:
juicefsCE:
metaURL: redis://127.0.0.1/1
path: /sync_test/
Specify source files and mount additional volumes
Set filesFrom on the source endpoint to synchronize only the specified paths. As with WarmUp, this field supports files, configMap, and filePath as file-list sources, but it cannot be configured on the destination endpoint. When JuiceFS Enterprise Edition is used as the source endpoint, filesFrom requires client image version 5.1.10 or later. When JuiceFS Community Edition is used as the source endpoint, it requires version 1.3.0 or later.
spec:
from:
external:
uri: oss://sync-test.oss-cn-hangzhou.aliyuncs.com/sync-src-test/
filesFrom:
files:
- images/
- videos/example.mp4
Each endpoint also supports mounting a ConfigMap, Secret, HostPath, or PVC into the sync Pods through extraVolumes. This can be used to provide configuration files or access a local file:// endpoint.
Configure sync Pods
Use resources to configure resources for both manager and worker Pods. Starting from v0.8.0, you can use managerResources and workerResources to configure the two types of Pods separately. If these fields are set together with resources, managerResources and workerResources override the corresponding resource configuration for their respective Pods.
Use env to inject environment variables into both manager and worker containers. Added in v0.8.3
spec:
managerResources:
requests:
cpu: 1
memory: 1Gi
workerResources:
requests:
cpu: 2
memory: 2Gi
env:
- name: HTTP_PROXY
value: http://proxy.example.com:8080
For more supported options, refer to the example.
Use Kerberos to access HDFS Added in v0.8.0
When an external endpoint URI uses the hdfs:// scheme, you can set krb5Principal and provide a keytab using either krb5Keytab or krb5KeytabBase64. Each field supports either value or valueFrom. The two keytab fields cannot be used together. The Operator also does not support HDFS-to-HDFS synchronization.
spec:
from:
external:
uri: hdfs://namenode.example.com:8020/source/
krb5Principal:
value: [email protected]
krb5KeytabBase64:
valueFrom:
secretKeyRef:
name: hdfs-credentials
key: keytab-base64
Sync Progress
You can view the sync progress using the following command:
➜ kubectl get sync -w
NAME PHASE REPLICAS PROGRESS AGE
sync-test Preparing 3 12s
sync-test Progressing 3 19s
sync-test Progressing 3 7.40% 26s
sync-test Progressing 3 45.50% 38s
sync-test Completed 3 100% 50s
Sync Cleanup
Delete the corresponding CRD to clean up all resources, or set automatic cleanup after the task is completed by setting spec.ttlSecondsAfterFinished.
Scheduled Data Synchronization
Operator supports creating a CronSync resource for scheduled data synchronization.
apiVersion: juicefs.io/v1
kind: CronSync
metadata:
name: cron-sync-test
namespace: default
spec:
# Suspend jobs
# Will not affect jobs that have already started
suspend: false
# The number of successful jobs to keep, default is 3
successfulJobsHistoryLimit: 3
# The number of failed jobs to keep, default is 1
failedJobsHistoryLimit: 1
# Concurrency policy, default is Allow
# - Allow: Allow concurrent runs of jobs
# - Forbid: Forbid concurrent runs, skip the new job if previous is still running
# - Replace: Replace the currently running job with a new one
concurrencyPolicy: Allow
# ref https://wikipedia.org/wiki/Cron
schedule: "*/5 * * * *"
syncTemplate:
spec:
replicas: 2
from:
...
to:
...

