Skip to main content

Distributed cache

By default, JuiceFS Client cache is exclusive to the client mount point only, when many clients need to access the same data set repeatedly, distributed cache is a great way to share cache data and improve performance. Distributed cache is best suited for deep learning model training based on GPU cluster. By caching the training dataset into memory or SSD on all nodes, JuiceFS provides high-performance data access capabilities, so that the GPU capability is not bottlenecked by data access.

Architecture​

When distributed caching is enabled, client members will form a "distributed cache group," which is actually a consistent hashing ring. Each cached data block's storage location is calculated using consistent hashing and shared within the group. Every group member is essentially a normal JuiceFS mount point client, with its group name specified by the --cache-group option. All clients with the same group name make up a "cache group," and all cache sharing only happens within the same group: when a client initiates a read request, the data block is obtained directly from the member node. If the data is not yet cached on the node, the node will download the data from object storage and store it in the cache disk.

distributed cache architecture

As illustrated above, node D is trying to read file a (consisting of 4 data blocks, a1-a4), assuming by consistent hashing that a1-a4 are to be located on nodes A-D, so every member node will access object storage to fetch the corresponding data block. Upon download completion, nodes A-C will send the blocks back to node D to conclude this read request.

Network environment and data security

Currently, distributed caching assumes high-performance, security private network. If however, hosting on public internet is required, here are some important considerations:

  • Public internet typically yields far worse network quality compared to the object storage service, make sure you benchmark in advance, and if latency & throughput isn't ideal, consider giving up on distributed caching and directly download from object storage instead;
  • Distributed caching service comes with no authentication mechanism, serving a public network poses data security risks, so network ACL should be properly managed.
IORPC protocol

Starting from JuiceFS Enterprise Edition 5.2, the IORPC protocol provides a high-performance RPC framework for cache group communication, supporting connection multiplexing and multiple cache access modes (direct I/O, splice, etc.).

RDMA acceleration

Starting from JuiceFS Enterprise Edition 5.3, RDMA (Remote Direct Memory Access) can be used to accelerate data transfer between cache group nodes, significantly reducing latency and CPU overhead.

Service discovery​

In the distributed architecture, our metadata service is used as a service-discovery: every group member client will listen on a local port to serve distributed cache, and register its access address with the metadata service to share cache with its peers. Clients will discover its peers within the same cache group, and then communicate to share cache data.

In a cache group, read & write are carried out via peer communication, if there's any failure during this process, the client will give up on the cache group and execute locally instead. Still using the above diagram as an example: when D is trying to read a file, and request towards A fails (e.g. network timeout), hence block a1 is unavailable, then D will download a1 directly from object storage to complete this read.

If communication with a cache node fails multiple times, the node will be considered unreachable and will be removed from the cache group.

info

When faced with cache group failures, JuiceFS Client always tries stay serviceable, this means if client fails to communicate with its peers, many requests might penetrate to the object storage, hindering performance. In order to ensure a high performance distributed cache service, pay attention to:

Scaling and data rebalancing​

When a node joins or leaves a cache group, data is migrated to neighboring nodes on the hashing ring. To prevent unintended fluctuations, the JuiceFS cache group waits for 10 minutes after a membership change before starting migration. If a cache provider node reboots, all the cache read requests directed to that node fall through to object storage until the mount point recovers.

For planned cache group scaling, JuiceFS Enterprise Edition 5.4 provides two mount options that reduce requests falling through to object storage:

  • --commission (5.4.3 and later): Before background migration starts, read requests for data assigned to the new node are served by the node that owned the data before scaling. Then, the new node caches the data locally.
  • --decommission (5.4.1 and later): The node keeps serving reads from its local cache, stops adding new cache data for regular reads and writes, and starts migrating cached data to other members about 10 seconds after remounting.

See Cache group member change for the full procedures.

Membership changes only affect a small portion of cached data. The larger the cache group, the smaller the impact of adding or removing a single node on overall hit ratio. JuiceFS also uses virtual nodes to ensure data is evenly distributed, preventing hot spots during migration. For more details on consistent hashing and virtual nodes, see this article.

Fault tolerance​

If an error occurs while sending a cache block to a consumer, JuiceFS retries only once. This prevents transient network jitter from having an outsized impact on cache group performance. Other failures are not retried (for example, if a cache group server node fails to access object storage). Therefore, when a cache group request fails, the consumer request falls through to object storage directly. The cache group request timeout is defined by --remote-timeout. The default is 65 seconds, a very generous value that makes cache requests less likely to fail during network congestion.

However, for latency-sensitive, high-performance clusters, a 65-second timeout may actually increase file system latency when the network is congested. If the object storage service can handle the load and absorb some traffic, consider reducing the timeout (for example, use --remote-timeout=1 when internal network bandwidth is high). This allows requests to fall through to object storage faster under cache group congestion, improving system response time.

From the consumer's perspective, if access to a member node fails frequently (specifically, 31 consecutive failures), the node is "removed" from that consumer's view. The consumer effectively excludes the problematic member from its local member list. The client log records an event similar to:

<INFO>: remove peer x.x.x.x:xxx after 31 failure in a row [peer.go:532]

Note that removing a cache group member does not mean it is removed from service in the entire system. It only means the member is unavailable to that single consumer. Therefore, it is possible for different consumers to have different views of the available member list in the cache group architecture. In the diagram below, due to uneven node load and request errors, Client_1 removes Node C while Client_2 removes Node A, resulting in different cache group membership as seen by Client A and Client B:

cache group remove peer

Although the consumer has removed some peers, Nodes A, B, and C are still healthy and can communicate with the metadata service. From the metadata service's perspective, they remain valid cache group members. The consumer will continue trying to reconnect to removed nodes, and if the node load improves and it can respond normally, the consumer will reconnect to the full member list, accompanied by a log such as:

<INFO>: add peer x.x.x.x:xxx back after 829.247µs [peer.go:538]

If a network anomaly causes a true network partition where the client can only communicate with some cache group members and times out with others, the asymmetric cache group service architecture shown above is no longer temporary. It becomes a long-term state. In that case, according to the data rebalancing mechanism described in the previous section, if a provider node remains removed for more than 10 minutes without reconnecting, the consumer recalculates its local consistent hashing ring based on the current cache group topology and triggers a rebalancing request. Cache group nodes then redistribute data according to the new hash ring.

In the diagram above, if the cache group topology differs between Client_1 and Client_2, each consumer rebalances based on its own view of the consistent hashing ring. After rebalancing, nodes A, B, and C are still members of the same cache group, but due to the network partition they belong to two different consistent hashing rings simultaneously. As a result, some blocks may be stored redundantly on two nodes, allowing them to continue serving both Client_1 and Client_2 during the partition.

Removing cache group members reduces the cache group's service capacity. Therefore, in production environments, you should use the highest-spec network interfaces possible when building cache clusters, to avoid cache request failures caused by network issues.

Group backup​

By default, JuiceFS' distributed cache feature caches data only once without creating additional replicas. This is the most efficient setup as long as cache group nodes do not change frequently and internal network communication is stable.

However, in certain scenarios, enabling the group backup feature for JuiceFS clients can improve the stability of cache services. Starting from JuiceFS 5.1, you can enable the group backup feature by specifying the --group-backup parameter during client mount. Here is how it works:

When --group-backup is enabled, if a cache miss occurs, the client calculates the group backup node corresponding to the data block using the cache key and forwards the request to that node for handling. The backup node then checks whether the data is cached locally and either returns the data directly or requests it from object storage. In contrast to a cache group without backup, the cache block is processed by two cache group members before reaching the client. Therefore, ideally the cache block is stored on both member nodes.

Assuming no special circumstances, the architecture of the group backup feature is as follows:

cache group backup

In the diagram:

  1. The client requests the data block b1. The cache group's consistent hash algorithm determines that b1 is located on Node A.

  2. Node A attempts to serve the request but does not have b1 cached locally. Then, it computes that Node B is the backup node for b1 and forwards the request to Node B.

  3. Node B also misses the cache, so it fetches the data from object storage.

If any of the steps from "client → A → B → object storage" find the data already cached, the request is returned early without further access to the next step. Therefore, the group backup feature does not strictly guarantee two local replicas for every data block. Factors like insufficient cache space, eviction of data, or early cache hits can result in fewer than two replicas.

The target scenarios for the group backup feature include:

  • Frequent changes in cache group members, where maintaining stable cache hit rates is important.
  • Cache group expansion, where stable hit rates are desirable during the process.
  • Unreliable or unstable network quality between cache group nodes, with frequent disconnections, where improving hit rates is crucial.

On the other hand, the following scenarios are not suitable for enabling the group backup feature or must avoid keeping it enabled for extended periods:

  • Cache space is already insufficient. Enabling backup will increase space usage, further degrading cache service capacity.
  • The group backup feature is not designed to increase throughput. This is a common misconception. Cache group throughput is limited by inter-node bandwidth and cache disk capacity, and adding more replicas does not improve throughput. If you are concerned about hotspots, JuiceFS already distributes cache data across nodes using consistent hashing to balance access pressure, and adding more replicas will not further distribute the load.

Cache data priority​

Users might want to associate different importance to different cache data, so that when capacity is full, the less important cache are evicted first to save space for higher priority data. Use --cache-priority for this purpose.

This option can be specified in different subcommands, read below example for more.

# Add this option directly to a mount point, then all cache built through
# this mount point will carry the specified priority.
# This is especially helpful in Kubernetes CSI, as you can associate different priority
# to different PV, to distinguish imporance of application cache data
juicefs mount myjfs /jfs --cache-group=mygroup --no-sharing --cache-priority=3

# Add this option to the warmup command, and the data built via warmup will have the specified priority
juicefs warmup /jfs/big-dir --cache-priority=3

Although cache priority is usually used against distributed cache clusters, it works the same in a single node scenario, you can use the warmup command with different priority to built cache of different importance.

Using distributed cache​

A distributed cache group is composed of multiple individual JuiceFS clients, to set up a cache group, simply run the mount command as shown below, in a set of nodes within the same VPC:

# Replace $VOL_NAME with the actual file system name, --cache-group is the user-defined group name
juicefs mount $VOL_NAME /jfs --cache-group=mygroup

Cluster networks can be complicated with nodes attaching multiple interfaces of different performance levels. To achieve the highest performance, it is recommended to allocate the best performing network device to the cache group:

# --group-ip represents the network device assigned to the cache group. It is not necessary to specify the full IP address; a prefix can be used effectively as a CIDR.
# For example, with the high-performance network interface IP being 10.6.6.1, its CIDR is 10.6.0.0/16.
# Specifying --group-ip=10.6.0.0 will smartly attach the corresponding network interface.
juicefs mount $VOL_NAME /jfs --cache-group=mygroup --group-ip=10.6.0.0

Upon a successful mount, check the client log or directly view TCP connection to verify:

# Search for the word "peer" to see distributed cache related content
# Like <INFO>: Peer listen at 172.16.255.181:36985 [peer.go:790]
grep peer /var/log/juicefs.log

# Check both established and listening TCP sockets to verify that the cache group is working
ss -4atnp | grep jfsmount

Now that the cache group has been built, but it's still empty, in order to improve the first time read performance, the warm up command is often run in advance to cache all the data that needs to be used:

# Execute the warmup in any of the group member yields the same result
# In large scale scenarios, increase concurrency via -c, to speed things up
juicefs warmup /jfs/train-data -c 80

Additionally, you can also specify the --fill-group-cache option (disabled by default), so that data blocks uploaded to object storage are also sent directly to the cache group, so that every write operation contributes to cache. It is important to note that sending data blocks to the cache group may fail, so there is no guarantee that they will be cached.

Some important considerations when building a cache group, which also applies to dedicated cache clusters (introduced in the next section):

  • Use a homogeneous set of nodes (at least same disk size for cache storage) for cache group, because the consistent hashing algorithm currently used by JuiceFS assumes all nodes have the same cache storage size. Expect waste on disk usage if you were to use nodes with different disk size for JuiceFS cache group.
  • Cache group members are connected to each other at least with 10 Gigabit Ethernet, if multiple network interfaces are installed, use --group-ip to specify the network device with larger bandwidth.
  • Cache group members need to ensure high-speed access to the object storage service, low internet speed will result in a poorer use experience.

Dedicated cache cluster​

In a distributed cache group, all clients contribute to cache, but if the cache group exists within a cluster that's constantly scaling up and down (like Kubernetes), cache utilization is not ideal because cache nodes are frequently destroyed and re-created. In this case, consider deploying a dedicated cache cluster on a set of nodes (or containers) so that the computation cluster have access to a more stable cache.

dedicated cache cluster architecture

While application clients will join this cache group, and obtain data from the group, they'll enable the --no-sharing option, this option literally means that clients will not share their own cache data, and only take from the group. This eliminates ephemeral clients from the cache group and improves stability for the distributed cache group.

As can be seen, a dedicated cache cluster is essentially a distributed cache group, but cache providers and users are different JuiceFS clients. Thus in this scenario, we usually use the phrase "cache cluster" to refer to this stable, permanent cache group, and the word "client" to refer to the application clients that's actually using the cache service, even though they are all JuiceFS clients, just mounted with different options.

Building a dedicated cache requires a cache group, in Kubernetes this is usually deployed as DaemonSet or StatefulSet (see example in our docs), taking the cache group demonstrated in the above section as an example, clients can connect to this group by running:

# Application clients will mount with the --no-sharing option,
# they will join the cache group, but do not provide cache service to other members
juicefs mount $VOL_NAME /jfs --cache-group=mygroup --no-sharing

# You can warm up the data from --no-sharing mount points as well, the result is the same as warming up in any other group members
juicefs warmup /jfs/train-data -c 80

After clients have been set up like above, they will use the cache group in a "only take, never give" manner, but note that even for them, the multi-level cache design still applies: data sent from the remote cache group will be cached locally, in the client cache directory. If you wish to disable client cache altogether, and direct all file access to the cache cluster, you can disable the client cache through --cache-size=0 option:

# Disable client cache to direct all read to the cache cluster
juicefs mount $VOL_NAME /jfs --cache-group=mygroup --no-sharing --cache-size=0

Disabling client cache in order to maximally utilize cache cluster disks, and save client disk space. If client disk comes with poor performance, you can also use this method to increase IO.

Some other --no-sharing caveats:

  • If the client node does not hit the cache, the cache group will be responsible for downloading and caching the data, and then provide it to the client. Unless peer communication timeouts or fails, clients themselves do not directly download files from object storage.
  • When multiple clients simultaneously access a file that's currently uncached, only a single cache penetration takes place, the corresponding cache group member will handle this download request and then return the result to the clients, there will not be wasted, repeated requests.

Multi-level cache architecture​

In JuiceFS distributed caching, multi-level caching consists of two complementary cache hierarchies:

  • Cache layers between an application client and a cache group: An application client uses the kernel page cache, in-process memory cache, local disk cache, and distributed cache group in sequence.
  • Cache layers between cache groups: A client can configure a primary cache group and a second cache group with --cache-group and --second-group. If the primary cache group misses, the client continues to query the second cache group.

When workloads run across multiple data centers, availability zones, or compute clusters, you can deploy a small, high-performance primary cache group within each compute cluster and a larger second cache group at a regional hub. This setup preserves the client-side cache hierarchy while allowing requests that miss the primary cache group to be served by the second cache group before falling back to object storage.

dedicated cache cluster multi-level cache

Added in v5.2.15 JuiceFS Enterprise Edition supports two-level distributed caching. Use --cache-group to specify the primary cache group and --second-group to specify the second cache group.

tip

Starting with JuiceFS Enterprise Edition 5.3.9, clients can also read prefetched data blocks from the second cache group and backfill them into the primary cache group.

Cache layers between clients and cache groups​

The multi-level cache architecture used for single-node caching also applies to distributed caching. In addition to the application client's kernel page cache, in-process memory cache, and local disk cache, a distributed cache group provides a shared remote cache layer.

Distributed caching does not simply extend the single-node cache path to remote nodes:

  • Application clients read data through FUSE, so FUSE accesses populate the kernel page cache on the client.
  • Cache group nodes do not read data through a JuiceFS FUSE mount when serving client requests. Instead, they read data directly from the cache disk and send it to the requesting client. This can populate the kernel page cache of the local file system that stores the cache files.
  • Data read from a cache group can still enter the application client's local cache. Unless --cache-size=0 is specified, --no-sharing only prevents the client from sharing its local cache with the cache group; it does not disable local caching.

If --cache-size=0 is specified on an application client, JuiceFS still reserves 100 MiB of in-process memory for caching and can use the kernel page cache on the FUSE path. Repeated reads can therefore still hit the memory cache or page cache, but file data is not persisted to the client's local cache disk.

tip

Starting with JuiceFS Enterprise Edition 5.0, when a client deletes a file, the cache group that the client has joined actively removes the corresponding cached data, even if the client uses --no-sharing.

How two-level distributed cache groups work​

After a second cache group is configured, read requests look up data in the following order:

  1. The application client's kernel page cache
  2. The application client's in-process memory cache and local disk cache
  3. The primary cache group
  4. The second cache group
  5. Object storage
Application read
│
▼
Client kernel page cache
│ Miss
▼
Client in-process memory / local disk cache
│ Miss
▼
Primary cache group (--cache-group)
│ Miss
▼
Second cache group (--second-group)
│ Miss
▼
Object storage

The primary and second cache groups maintain independent consistent-hash rings. The client discovers members of both groups, but it still uses the group specified by --cache-group as its primary cache group. Configuring --second-group does not make the client a data-serving member of the second cache group.

When the primary cache group misses and the second cache group returns the data successfully, the primary cache group attempts to write the data block to its local cache. Subsequent reads of the same data block can then be served directly by the primary cache group without accessing the second cache group again. The second cache group therefore acts as an upstream cache layer for the primary cache group. This reduces pressure on object storage while the primary cache group is cold and helps a newly created primary cache group gradually build up hot data.

If the second cache group also misses, a member of the second cache group reads the data from object storage. As the data is returned to the client, both the second and primary cache groups typically attempt to cache it, and the data may also enter the application client's cache layers. However, cache writes can fail because of insufficient space, concurrency limits, or node failures. Multi-level caching therefore does not guarantee that every data block is always stored in every cache layer.

Multi-level cache groups are not cache replicas

--second-group and --group-backup serve different purposes:

  • --second-group establishes an upstream-downstream relationship between two independent cache groups. It is intended for tiered caching, cross-cluster cache sharing, and cache migration.
  • --group-backup selects a backup node within the same cache group. It is intended to reduce cache-hit fluctuations caused by membership changes or node failures.

Configuring a second cache group does not automatically duplicate existing cached data and cannot replace cache replication.

Use cases​

Tiered cache capacity​

You can deploy a smaller, high-performance primary cache group within a compute cluster and a larger second cache group at a regional hub:

  • The primary cache group uses local NVMe devices to provide low-latency, high-throughput access for the current workload.
  • The second cache group uses higher-capacity SSDs to retain shared hot data for multiple compute clusters.
  • Object storage serves as the final persistent storage layer.

The primary cache group only needs to hold the hot working set of the current compute cluster. After hot data is evicted from the primary group, subsequent reads can retrieve it from the second group if it is still cached there, instead of accessing object storage directly.

Share a common cache across compute clusters​

Multiple compute clusters can each deploy an independent primary cache group and use the same regional cache group as their second cache group. For example:

Compute cluster A: cache-a ─┐
├─> regional-cache ─> Object storage
Compute cluster B: cache-b ─┘

In this architecture, the primary cache groups of different compute clusters are independent, so changes in one group do not affect the others. Shared data can be reused through the second cache group, reducing repeated downloads from object storage and cross-region traffic.

Migrate or rebuild a cache group smoothly​

When replacing cache nodes, resizing a cache group, or rebuilding a cache cluster, temporarily configure the old cache group as the second cache group and the new group as the primary cache group:

New cache group (primary) -> Old cache group (second) -> Object storage

The workload continues to access the new cache group. When the new group misses, it reads the data from the old group and backfills it into the new group. Hot data gradually migrates to the new cache group as it is accessed by real workloads. After object-storage passthrough traffic and reads from the second cache group decrease and stabilize, remove --second-group and then remove the old cache group.

For adding or removing nodes within the same cache group, follow the cache group member change procedures instead. The two-group procedure here is for migrating between independent cache groups.

This is an access-driven, gradual migration. It does not proactively migrate all data from the old cache group. To build a complete target cache before switching over, use juicefs warmup to warm up the required dataset.

Configuration​

Assume that you want to deploy the following two cache levels:

  • Primary cache group: hot-cache
  • Second cache group: warm-cache

First, mount the file system on the persistent cache nodes in each group. Run the corresponding command on every member of each cache group, and adjust the cache directory, capacity, and network interface for your environment.

Every primary cache group node must also set --second-group=warm-cache so that it can access the second cache group on a cache miss. Do not configure hot-cache as the second group of warm-cache, as this would create a circular dependency.

# Primary cache group nodes
juicefs mount $VOL_NAME /hot-cache \
--cache-group=hot-cache \
--second-group=warm-cache \
--cache-dir=/data/hot-cache \
--cache-size=102400 \
--group-network=eth0

# Second cache group nodes
juicefs mount $VOL_NAME /warm-cache \
--cache-group=warm-cache \
--cache-dir=/data/warm-cache \
--cache-size=1024000 \
--group-network=eth0

Then specify both cache groups when mounting the file system on an application client:

juicefs mount $VOL_NAME /jfs \
--cache-group=hot-cache \
--second-group=warm-cache \
--no-sharing

For short-lived application clients that scale frequently, --no-sharing is recommended to prevent application nodes from joining the primary cache group and changing its topology. The application client still checks its kernel page cache, in-process memory cache, and local disk cache first, then accesses the primary and second cache groups in order.

To disable the local disk cache on application clients, add --cache-size=0:

juicefs mount $VOL_NAME /jfs \
--cache-group=hot-cache \
--second-group=warm-cache \
--no-sharing \
--cache-size=0

The client still uses a 100 MiB in-process memory cache and the kernel page cache, but file data is not persisted to the client's local cache disk.

Parameter constraints

Observe the following constraints when using --second-group:

  • --cache-group must also be configured. If --second-group is used alone, the client ignores it.
  • The primary and second cache group names must differ. Otherwise, the client ignores the second cache group.
  • Both cache groups must belong to the same JuiceFS file system. The client internally prefixes cache group names with the file system name.
  • Network connectivity is required between the primary and second cache groups and between application clients and both cache groups.
  • Cache group listeners do not provide authentication. Deploy them on a trusted private network and restrict access with security groups, firewalls, or network policies.

Warm-up and eviction​

Warm up the primary cache group​

When juicefs warmup is run against an application mount point configured with two cache groups, it fills the primary cache group:

juicefs warmup /jfs/train-data -c 80

If the second cache group already contains the target data, the warm-up task reads the data from the second group and backfills it into the primary group, reducing object-storage access.

If neither cache group contains the target data, the request falls through from the second cache group to object storage. As the data returns, both the second and primary cache groups attempt to populate their caches. Cache population is asynchronous and best effort: the second cache group may drop a cache write if its write queue is congested, while a cache write accepted by the primary group may still fail during the subsequent disk flush because of insufficient space or disk errors. Therefore, a successful warm-up request does not mean that the data block has been written to the cache disks of both groups. In a fully cold-cache scenario, warming up the primary group can still increase space consumption and network traffic in both groups. Use cache checks, monitoring metrics, and client logs to verify the actual cache state.

Starting with JuiceFS Enterprise Edition 5.3.9, regular prefetch operations can also read complete data blocks from the second cache group and asynchronously backfill them into the primary group. Earlier client versions can still fall back to the second cache group, but some prefetch requests may not use the second group.

Evict data from a specific cache group​

juicefs warmup --evict affects only the primary cache group joined by the mount point on which the command is executed. It does not follow --second-group to evict data from the second cache group.

# Evict the target data from the primary cache group
juicefs warmup /jfs/train-data --evict

To also evict data from the second cache group, run the command separately on a mount point where that group is configured as --cache-group:

# /warm-cache is mounted with --cache-group=warm-cache
juicefs warmup /warm-cache/train-data --evict

This prevents an eviction initiated by an application client from unexpectedly clearing the upstream shared cache.

Failure handling​

The second cache group primarily handles cache misses in the primary cache group. It is not a failover cluster for the primary cache group.

The client queries the second cache group only when the primary cache group explicitly returns a cache miss. If the primary group cannot respond normally because of a network timeout, node overload, or another error, the request may follow the distributed cache's existing failure-handling path and fall through to object storage instead of automatically failing over to the second cache group.

Multi-level cache groups therefore do not replace the following measures:

  • Provide a reliable, low-latency network for the primary cache group.
  • Configure --remote-timeout according to workload latency requirements.
  • Continuously monitor cache group errors, object-storage passthrough traffic, and workload read latency.
  • When you need to reduce the impact of membership changes, follow the cache group member change procedures. Choose --commission, --decommission, --group-backup, or weight adjustments according to your client version.

If a client repeatedly fails to access a member of the second cache group, it temporarily removes that member from its own view and continues trying to reconnect. The primary and second cache groups maintain separate membership lists. Membership changes in one group do not trigger data rebalancing in the other.

Monitoring and verification​

After configuration, use client logs to verify that members of both cache groups have been discovered:

grep -E "add peer|remove peer|peer" /var/log/juicefs.log

The add peer ... into group ... log entry contains the actual cache group name and can be used to distinguish the primary and second cache groups. The client automatically prefixes the cache group name with the file system name, so the name in the log may differ slightly from the command-line value.

Use the following metrics and observations to verify that multi-level caching behaves as expected:

  • JuiceFS client read latency and throughput
  • juicefs_remotecache_errors: cache group communication errors
  • juicefs_remotecache_redirect and juicefs_remotecache_redirectBytes: redirected cache group requests and traffic
  • Cache capacity, cache hits, and disk throughput on nodes in both groups
  • Object-storage download traffic, request volume, and error rate
  • Network throughput and latency between the primary and second cache groups

Use the following verification sequence:

  1. Clear the cache or use test data that has never been read, and record object-storage traffic with completely cold caches.
  2. Warm up the test data in the second cache group and verify that its cache usage increases.
  3. Read the data through an application mount configured with both cache groups and verify that the primary cache group is gradually backfilled.
  4. Repeat the read and verify that most requests hit the application client or primary cache group, while traffic to object storage and the second cache group decreases significantly.
  5. Test a primary cache miss, a primary cache node failure, and a second cache node failure separately, and verify the resulting behavior and latency boundaries.

Cache group mixed deployment​

If you are in a scenario where the performance of the local disk is poor, disabling the local cache as described in the previous section is suitable. However, if you are facing the opposite situation and wish to maximize the utilization of local cache disks for better I/O performance and reduce network communication between nodes, consider mixing and deploying two JuiceFS clients on all nodes: one for the cache cluster and the other serving as the application end client.

mix-deployed cache group architecture

Such mixed deployment offers the following benefits:

  • The high performance disks are used both as distributed cache and local cache, not wasting any I/O capability;
  • All nodes join the same cache group, all file reads are handled by the cache group, which only download from object storage once, effectively consolidating object storage requests and reduce the number of API calls.

Mixed deployment puts cache cluster and --no-sharing clients on a same set of nodes. Assuming a 1TB (1000000MB) disk on each node, use the following mount commands:

# Cache cluster mount point
# --cache-size=500000 limit cache size to 0.5TB
# --free-space-ratio=0.1 limit cache usage to 90% of total disk space
juicefs mount $VOL_NAME /distributed-cache --cache-group=mygroup --cache-dir=/data/distributed-cache --cache-size=500000 --free-space-ratio=0.1

# Application mount point
# --cache-size=1000000 limit cache size to 1TB
# --free-space-ratio=0.01 limit cache usage to 99% of total disk space
juicefs mount $VOL_NAME /jfs --cache-group=mygroup --cache-dir=/data/local-cache --cache-size=1000000 --no-sharing --free-space-ratio=0.01

Even though the two mount point share the same cache disk, they manage their own cache data, and require different cache directories (--cache-dir). Do not use the same cache directory for two mount points or there will be conflicts. This obviously creates redundancy because data will be cached both in the cache cluster and clients, this redundancy is necessary for increasing performance. But note that the above commands have already been tuned to favor local I/O: using smaller --free-space-ratio and larger --cache-size for local clients, this ensures that when cache disks are full, distributed cache is evicted in favor of local client cache.

Mixed deployment can be flexibly tuned according to your specifications, for example, if your environment comes with an excessive amount of memory, you can also use this memory as cache disks to gain even higher I/O:

# Cache cluster mount point
# Adjust --cache-size according to actual available memory
juicefs mount $VOL_NAME /distributed-cache --cache-size=4096 --cache-dir=/dev/shm --cache-group=mygroup

# Application mount point
# Use a minimal --free-space-ratio to maximize cache disk usage
juicefs mount $VOL_NAME /jfs --cache-size=102400 --cache-dir=/data --cache-group=video-render --no-sharing --free-space-ratio=0.01

Maintenance​

Remove/replace cache disk​

Starting from version 5.1, the JuiceFS client supports the group backup feature. When this feature is enabled, the client forwards cache miss requests to other nodes for assistance. This allows users to handle disk removal or replacement operations more smoothly. Therefore, for clients running version 5.1 or above, it is recommended to follow the steps below:

info

The steps below assume that only one copy of data is being cached, so --group-backup is used temporarily and turned off once the process is completed. However, if your cluster has already enabled dual replicas, all mount points have the --group-backup parameter enabled by default. Therefore, you can skip the steps related to group backup below.

  1. Copy the data from the disk to be removed to other disks. If the available space on a single disk is insufficient to hold the data to be migrated, you can group the data and copy it to other disks based on available capacity.

  2. Adjust the following mount parameters:

    • Add the --group-backup parameter. This ensures that after the mount point restarts, the client forwards cache miss requests to other clients for assistance.
    • Modify the --cache-dir parameter to remove the old disk.
    • Since removing the disk changes the total cache capacity of the node, adjust --group-weight to lower the weight and prevent inefficient utilization caused by inconsistent space between nodes.
  3. After modifying the parameters, seamlessly restart the JuiceFS client to apply these changes.

  4. After the restart, the client rescans the cache directories and rejoins the cache group. If the weights were adjusted, the cache group rebalances the data.

  5. After rebalancing the data, to avoid the group backup feature consuming too much space, remove the --group-backup parameter and smoothly restart the mount point again.

If you are still using a version earlier than 5.1, the JuiceFS client also supports migrating data across multiple disks. However, since the group backup feature is not supported, you may experience more significant cache hit rate fluctuations. Follow these steps:

  1. Copy the data from the disk to be removed to other disks. If the available space on a single disk is insufficient to hold the data to be migrated, you can group the data and copy it to other disks based on available capacity.

  2. Adjust the following mount parameters:

    • Modify the --cache-dir parameter to remove the old disk.
    • Since removing the disk changes the total cache capacity of the node, you should adjust --group-weight to lower the weight and prevent inefficient utilization caused by inconsistent space between nodes.
  3. After modifying the parameters, seamlessly restart the JuiceFS client to apply these changes.

  4. After the restart, client will scan local cache data and rebalance within the cache group according to the new weight.

Steps are largely the same for disk replacement. Just copy the data to the new disk, change the mount option when needed, and then re-mount.

Cache group member change​

JuiceFS Enterprise Edition 5.4 introduces two mount options for planned cache group scaling:

ScenarioOptionEffect
Adding a cache node (scaling out)--commissionThe new node joins the cache group smoothly. Before background migration starts, read requests for data owned by the new node are served by the node that owned the data before scaling. Then, the new node caches the data locally.
Removing a cache node (scaling in)--decommissionThe node leaves the cache group smoothly. It continues serving read requests from its local cache while migrating cached data to other members at full speed.

Use these options only on cache group nodes, that is, mount points that provide shared caching to the group. Application clients mounted with --no-sharing do not need these options: --commission is ignored when --no-sharing is set, and --decommission has no effect on clients that do not provide shared caching. The two options are mutually exclusive. If both are set, neither takes effect.

Version requirements​

ComponentVersion requirementNotes
All cache group nodes--commission: 5.4.3 and later; --decommission: 5.4.1 and laterEvery member of the cache group must meet the version requirement. Versions 5.4.4 and later are recommended because they reduce cache misses during scaling and improve request forwarding and warmup handling during removal.
Metadata service5.4.3 and later when using --commissionOlder versions do not return an error, but they treat the new node as a regular cache group member, resulting in less effective smoothing during scaling.
Application client with --no-sharingNo requirementThese clients do not need either option. You can upgrade them together with the cache group when convenient.
tip

Before using these options, upgrade the metadata service first, then upgrade the cache group nodes. Follow the seamless restart procedure to avoid interrupting workloads.

Smooth scaling in version 5.4 and later​

Both options reduce reads from object storage and fluctuations in the cache hit ratio during planned scaling. Each option applies only to the cache node on which it is set.

Add cache nodes Added in v5.4.3​
  1. Prepare the new node. Match its hardware specifications as closely as possible to those of the existing cache nodes, especially cache disk capacity and inter-node network bandwidth. Make sure it can access the cache group over the high-speed internal network. Configure its cache directory and capacity consistently with the existing nodes.

  2. Mount the new node with --commission, leaving its other cache group options unchanged:

    juicefs mount $VOL_NAME /jfs --cache-group=mygroup --cache-dir=/data/cache --cache-size=102400 --commission
  3. Monitor the migration for at least 30 minutes, until the metrics are stable:

    • Cache usage on the new node keeps increasing.
    • Inter-node migration traffic is visible.
    • Object storage read traffic and latency do not increase significantly, which indicates that few requests are falling through to object storage.
    • The overall cache hit ratio remains stable.
  4. After migration stabilizes, perform a seamless restart on the new node without --commission to return it to normal operation.

Notes:

  • Adding one node at a time is recommended, but not required. If you need to add multiple nodes, add them within the same maintenance window to avoid repeated data migration. Then monitor the cache group as a whole.
  • Avoid overlapping this operation with other cache group changes or node maintenance.
  • Background migration starts about 10 minutes after the new node joins. Until then, read requests for data not yet cached on the new node are served by the node that owned the data before scaling. Then, the new node caches the data locally without manual intervention.
  • --commission does not create an extra cache replica. Data is migrated from the previous owners to the new node.
  • You can keep --commission enabled for a long time, but the node remains in the scaling state. Remove the option once migration has stabilized.
--commission and --group-backup​

--commission and --group-backup are not mutually exclusive, but they serve different purposes during scaling:

Aspect--commission--group-backup
ScopeHandles read requests from other cache group members for data assigned to the new node but not yet cached thereAlso handles the new node's own local reads and tends to create a cache replica
Data sourceThe node that owned the data before scaling, which usually has the data cachedA hash-selected backup node, which may not have the data
Space usageNo extra replica; data is migrated from the old ownerTends to store two cache copies, using additional space
VersionCache node and metadata service 5.4.3 and laterClient 5.1 and later, with no additional dependency

In most cases, use --commission alone. Add --group-backup if the new node also serves application reads, or if you want additional fault tolerance and a higher hit ratio during scaling; keep in mind the extra space usage and remove --group-backup after warm-up stabilizes. When adding several nodes at once, --commission has a clearer advantage.

Remove cache nodes Added in v5.4.1​
  1. Choose a low-peak window. Migration runs at full speed by default and consumes inter-node network bandwidth and disk I/O.

  2. Perform a seamless restart on the node with --decommission, keeping its other mount options unchanged:

    juicefs mount $VOL_NAME /jfs --cache-group=mygroup --cache-dir=/data/cache --cache-size=102400 --decommission

    Then, the node:

    • Continues to serve read requests, using its local cache whenever possible.
    • Does not add new cache data for regular reads and writes. Incoming cache writes from peers are forwarded when a target is available; otherwise they are rejected.
    • Migrates its local cache to other cache group members at full speed, starting about 10 seconds after remounting.
  3. Observe the migration progress:

    • Local cache usage on the node keeps decreasing and eventually approaches 0.
    • Cache usage on the other nodes increases and then stabilizes.
    • Object storage traffic and access latency remain normal.
    • The overall cache hit ratio remains stable.
  4. Remove the node safely. After the cache is mostly migrated, stop the process or remove the node through your deployment platform.

Notes:

  • Do not force-stop the process (for example, with kill -9) before migration completes. Otherwise, the data that has not been migrated falls through to object storage.
  • Migration runs at full speed by default. If you are concerned about inter-node bandwidth usage, set JFS_FILL_GROUP_THREADS before starting the mount on this node. The default is 20 while the node is decommissioning; reduce it to 1–2 to limit migration concurrency and speed.
  • If a node fails and must be removed urgently, remove it directly and use Evaluate the impact of member node changes to estimate the workload impact.
  • In version 5.4.3 and earlier, warmup requests sent to a node in the --decommission state may fail. This is fixed in version 5.4.4 and later: when a decommissioning node cannot find a forwarding target, it holds the warmup data locally and migrates it later instead of rejecting the request. On affected versions, finish or pause warmup before decommissioning the node, or contact technical support for an available workaround.
  • Remove one node at a time so that the remaining group has enough capacity and the impact is predictable. Wait for the current node to finish migrating before starting the next one.
  • To replace the whole cache group, use the two-level cache migration procedure instead of removing nodes one by one.
Kubernetes deployments​

If the cache group is managed by JuiceFS Operator, do not add --commission or --decommission manually to individual Pods. Adjust the cache group membership and let the Operator handle the migration. For details, see Add and delete cache nodes.

For a cache group deployed as a DaemonSet, set its update strategy to OnDelete before changing mount options, so that existing nodes are not recreated when the template changes. Then add or remove the target node from the node selector and delete only that node's Pod to apply the new option. See Scale cache nodes smoothly.

Scaling checklist​
PhaseChecks
Before scalingVersions meet the requirements; node specifications and network are ready; add one node at a time unless multiple nodes need to be added together.
During scalingThe new node's cache usage increases; object storage traffic does not increase significantly; overall hit ratio is stable.
After scalingMigration is stable; perform a seamless restart without --commission.
Before removalChoose an off-peak period; confirm that the migration bandwidth and disk I/O are acceptable.
During removalThe target node's cache usage decreases; other nodes' cache usage increases; object storage traffic and latency remain normal.
After removalThe cache is nearly drained; stop the node; confirm that the overall hit ratio is stable.
Cache node scaling FAQ​

How long should --commission stay enabled?

There is no fixed duration. Use the metrics to determine when migration is complete: cache usage on the new node stops growing, inter-node migration traffic drops, and object storage read traffic and latency return to baseline. This usually takes tens of minutes to several hours, depending on the data volume and inter-node bandwidth. Once the metrics have stabilized, perform a seamless restart without --commission. The node retains its original weight, so no second migration is triggered. You can keep the option enabled for a long period, but the node remains in the scaling state. Remove the option once migration has stabilized.

Does a short restart of a cache node trigger data migration?

No. If the node recovers within about 10 minutes, the cache group topology remains unchanged, and no data migration is scheduled. Requests may briefly fall through to object storage while the node is down. Normal cache service resumes automatically after the node rejoins.

Is the old --group-weight=0 removal method still supported?

Yes, but newer versions should use --decommission. Setting the node's weight to 0 still triggers cache migration, but the node stops serving cache to the group during removal. As a result, more requests fall through to object storage. By contrast, --decommission keeps the node serving reads from its local cache while data is being migrated, resulting in less fluctuation in the cache hit ratio.

Original procedure for older versions​

tip

If possible, first upgrade the metadata service to version 5.4.4 or later, and then upgrade the cache nodes to version 5.4.4 or later. After upgrading, use --commission and --decommission for cache group scaling. The minimum required versions are 5.4.3 for --commission and 5.4.1 for --decommission. The --commission option also requires the metadata service to be version 5.4.3 or later. If you continue to use --group-weight=0 for node removal, the node stops serving cache to the group during removal, causing more requests to fall through to object storage.

If you cannot upgrade the cache nodes yet, use the original procedure below.

For planned maintenance events involving the addition of nodes, it is recommended to use the group backup feature of JuiceFS 5.1 to reduce cache hit rate fluctuations during migration. For planned maintenance events involving node removal, it is recommended to adjust the cache group weight to minimize cache hit rate fluctuations.

info

The steps below assume that only one copy of the data is cached, so the --group-backup parameter is used temporarily and turned off after use. However, if your cluster has already enabled dual replicas, since the data is already cached in two copies, the impact of adding or removing nodes is minimized, and no additional operations are needed.

If a cache group is deployed within Kubernetes, there's no easy way to adjust mount options for a single client, rendering below procedure undoable. You must first exclude a node from Kubernetes deployment, and mount directly to host using JuiceFS Client, only then can you adjust mount options (and seamless restart) for this single node, and migrate cache data via --group-weight.

When you plan to add nodes, follow these steps:

  1. Mount JuiceFS on the new node and add it to the cache group. Be sure to include the --group-backup parameter on the new node. Since the cache disk on this new node is empty, the group backup feature helps excessive access to object storage from this new node.
  2. After running for some time, observe the object storage penetration traffic. Once it stabilizes at a low level, remove the --group-backup parameter from the new node to avoid excessive space usage by group backup replicas.

For node removal scenarios, JuiceFS clients support adjusting the weight of member nodes via the --group-weight parameter. This parameter can be used not only to form heterogeneous cache clusters (cache groups with nodes of different capacities) but also to explicitly migrate cached data during planned node removal. Follow these steps:

  1. Add the --group-weight=0 parameter to the node scheduled for removal:

    # Set the weight of the node for removal to 0 to explicitly trigger cache data migration.
    juicefs mount myjfs /jfs --cache-group=mygroup --group-weight=0

    Notes:

    • After setting the weight to 0, data migration begins 10 seconds after remounting (unlike during node restart or shutdown, where migration starts after 10 minutes). During this data redistribution, cache misses may occur when data is accessed, leading to cache penetration. Refer to the next section for details on how to evaluate the impact of such changes.
    • Before data migration completes, accessed data may cause cache penetration. If you want to minimize penetration, avoid reducing the weight to 0 all at once. Instead, gradually lower the weight within an acceptable range.
  2. During the ongoing data migration, monitor the migration progress by checking the cache data size (blockcache_bytes) or cache write traffic (remotecache_putBytes) in the monitoring dashboard. You can also observe object storage traffic and I/O latency to assess the current cache penetration and system performance. Once the local cache size on the node scheduled for removal drops to 0 or near 0, and the cache data on other nodes stabilizes, you can remove the node.

Evaluate the impact of member node changes​

Taking node restart as an example, this section outlines the steps to evaluate the impact on the cache group when there is a member change. Node restart seldom exceeds 10 minutes, thus a simple restart will not cause data migration (see architecture). But during downtime, data on the restarting node cannot be accessed, so a restart will definitely fluctuate the cache hit ratio. To actually quantify the impact, we should head to the file system monitor page to do the math.

Assuming the node expecting maintenance is A, open up the monitor page and first pay attention to the overall cache group capacity, which can be viewed directly from the Distributed Cache section:

distributed cache overall metrics

As shown in Number of Distributed Cache Requests, the current number of cache requests is around 4,300 per minute.

Next, find out the actual request load for our node A, filter this node out using the top right All Clients menu, and select node A to focus on its local cache request load:

single node metrics within the distributed cache group

As shown in the figure Average Size of Cached Blocks, node A contains approximately 13TiB of cache data. Keep in mind that the monitor page is already filtered to display only node A. The current Cache Hits value is 130 per minute. Therefore, when this node goes offline, the affected requests are 130 out of 4300, or 3%. That is to say, during node restart, approximately 3% of read requests will penetrate to the object storage, served by its neighbors in the hashing ring. When A goes back online and rejoins the cache group within 10 minutes, the hashing ring topology is kept intact so no data migration is scheduled (except for the few penetrated requests, which are downloaded and cached in its neighbors).

From the cache group's perspective, a node restart involves a quick exit and rejoining. If you are planning a permanent eviction, the evaluation process is largely the same: use the formula provided above to calculate the request proportion and local cache size to determine the impact of this eviction. With --decommission, the node continues serving reads during migration and starts migrating about 10 seconds after remounting, so the estimated passthrough ratio is an upper bound. With --commission, missing blocks are served by the previous owner instead of object storage, so the actual impact is also lower than the estimate.

If the overall capacity is insufficient (also follow the below Observation and monitor section to check current usage and whether a scaling is required), and you decide to add a new node into the cluster, the cache group starts migrating data about 10 minutes after its membership list stabilizes to maximize utilization of available disk space and I/O capacity. During this migration, access to affected data will result in cache penetration. With --commission, cache misses during this period are served by the previous owners, reducing object storage reads.

Assuming a total of 500TiB of data is distributed across 40 cache group member nodes and it's expected to grow to 50 nodes, depending on different network bandwidths, the estimated data rebalancing duration will be:

tip

The "Theoretical balancing time" is calculated assuming that the network bandwidth can be fully utilized, which is impossible in real world scenarios, so the calculation results are for reference only.

Cache group total sizeCurrent number of nodesExpected number of nodesNetwork bandwidthTheoretical balancing time
500TiB405010Gbps2h26m
500TiB405025Gbps58m

The algorithm used to reach this conclusion:

  1. Before expansion, each node holds 12.5TiB of data (500 / 40 = 12.5). After expansion, the estimated node data size will be 10TiB (500 / 50 = 10). Each node expects to move 2.5TiB of data (12.5 - 10 = 2.5). This means the total migration size is 100TiB (40 * 2.5 = 100).
  2. Physical nodes are translated into virtual nodes in the hashing ring. Even though only 10 physical nodes are added, the resulting ring topology will see hundreds (to thousands) of new virtual nodes. The existing 40 nodes will transfer these 100TiB of data evenly to the 10 newcomers.
  3. Assuming the peer network bandwidth being 10Gbps, each new node will receive 10TiB of migration data. The estimation of time spent is 2h26m (10TiB / 10Gbps = 2h26m).

Observation and monitor​

Import our Grafana Dashboard and it already includes distribute cache related panels:

  • Remote Cache Requests
  • Remote Cache Throughput
  • Remote Cache Latency

The most important thing to keep in mind about distributed cache hit ratio: a drop in client's cache hit ratio, doesn't necessarily indicate penetration against the cache group. When a client access file data that's not cached locally, it'll fetch all the data blocks from the cache group. Hence, client's miss becomes cache group members' hit. So next time when you noticed a drop in hit ratio, check for any obvious object storage traffic spikes, this'll tell you if cache group has been penetrated.

To calculate overall hit ratio of a cache group using Prometheus query language:

# Depending on your environment, metric prefix may need substitution
# For on-prem deployments, Grafana dashboard "volume overview" already contains this query
1 - (sum(rate(mount_get_bytes{owner="", subdir="$name", cache_group!="", cache_group=~"${cache_group:raw}"}[1m])) / sum(rate(mount_read_bytes{owner="", subdir="$name", cache_group!="", cache_group=~"${cache_group:raw}"}[1m])))

The denominator of the above query is read_bytes, i.e. FUSE read traffic, which may contain various non-disk reads. If you wish to accurately represent hit ratio, you need to calculate hits / (hits + object_get) where hits is the local cache hits on all member nodes, and object_get being the penetrated object storage requests.

Some common maintenance based on monitoring:

  • Check the "Block Cache Hit Ratio" and "Object Requests" for signs of cache penetration, if there's indeed unexpected cache penetration, check "Block Cache Size" for the current usage, and see if the cache disks need an upgrade.
  • Low cache group storage storage will cause block eviction, observe this using juicefs_blockcache_evict and other relevant metrics. See Metrics for more.
  • Check requests latency for cache group server nodes, i.e. Remote Cache Latency (server), if any node are running with high latency, there may be a black sheep in the group, you need to examine this node and probably add a --no-sharing option to prevent it from serving other members.
  • When scaling cache nodes, monitor juicefs_blockcache_bytes on the target node, juicefs_remotecache_putBytes and juicefs_remotecache_sendBytes for migration traffic, cache group request volume and errors (juicefs_remotecache_get, juicefs_remotecache_send, and juicefs_remotecache_errors), object storage read traffic and latency, and inter-node network throughput.

Troubleshooting​

If a cache group doesn't yield expected performance, before going into troubleshooting, we recommend:

  • If possible, enable DEBUG logging for all cache group members, and pay special attention to the word peer, as this word usually appears in the log text when the cache group member changes.
  • Check distributed cache related metrics, under on-premises deployments, you can directly view "Remote cache" related panels in Grafana.

Is cache capacity enough, data completely warmed up?​

When using dedicated cache cluster, data is usually warmed up in advance, if there's still massive object storage penetrations, it may indicate insufficient cache capacity, and the cluster cannot hold all of the data blocks.

When using juicefs warmup, a frequently encountered situation to keep in mind is that even the command succeeds, file data isn't necessarily completely stored locally. The warmup intrinsically only cares about the download process, you're responsible to ensure a sufficient cache capacity (both in the cache disk, and the --cache-size option) so that cache blocks aren't evicted. For a single node cache disk, you can manually run something like du --max-depth 1 -h /var/jfsCache to verify the actual size, but for a cache group, you'll have to check the Block Cache Eviction Rate monitor for signs of eviction. An empty panel indicates that there's no eviction, which means the warmup runs without a problem. But if there's eviction happening (shown below), it means capacity is limited and cannot hold all cache data.

cache eviction monitor

As explained above, a warmup success is necessary condition for a 100% hit ratio, but not a sufficient one, because eviction can happen due to a variety of reasons. But on the other hand, if warmup doesn't succeed (return a non-zero code), then some data blocks does not successfully download, continue reading for troubleshooting steps.

juicefs warmup works against the specified mount point, add --verbose to the mount command, and check client log for errors, like:

<ERROR>: xxx could be corrupted: chunk x:x:xxx is not available: read header: EOF

This means cache group members encountered errors when downloading the data blocks, there're various causes for this, enable --verbose as well on cache group member nodes, and check their logs for signs of error.

Are there multiple network interfaces in cache group member nodes? Is the correct one being used?​

For nodes with multiple network interfaces, if those network cannot connect to each other, or has poor bandwidth, then it might be unfit for distributed cache use. Check network connectivity using commands like:

# Enter a cache group member node (or container)
# For member nodes, JuiceFS Client listens on a local port, for peer communication
# pprof listens at 6060 and 6070 by default, the port used for distributed cache is usually the largest one
lsof -PiTCP -sTCP:LISTEN | grep jfs

# Enter a client node (or container), and test network connectivity against distributed cache member node
telnet [member-ip] [port]

If it's required that you change the bind network interface for cache group, use the --group-ip option in the mount command to specify desired IP address. Note that CIDR prefix is also supported, e.g. --group-ip=172.16.0.0 to match with 172.16.0.0/16, this eliminate the need to specify different mount commands for different cache group members, making things easier to manage.

Is the intranet bandwidth of cache group members limited?​

It is recommended to connect the nodes of JuiceFS distributed cache with at least 10 Gigabit network. If you build a cache group under a low-bandwidth network, such as a Gigabit network, the following errors may occur:

# Peer network bandwidth too low, or not connected at all
<INFO>: remove peer 10.8.88.242:40010 after 31 failure in a row [peer.go:532]

# Peers are connected, but network quality is bad, causing peers to be removed from group following immediate rejoin
<INFO>: add peer 10.8.88.242:40010 back after 829.247µs [peer.go:538]

# Failure to download data blocks from peer
<WARNING>: failed to get chunks/6C/4/4020588_14_4194304 for 10.6.6.241:38282: timeout after 1m0s [peer.go:667]
<ERROR>: /fio_test/read4M-seq.2.0 could be corrupted: chunk 1:0:4020660 is not available: read header: read tcp 10.8.88.241:34526->10.8.88.242:40010: i/o timeout [fill.go:235]

# Download speed too slow
<INFO>: slow request: GET chunks/6E/4/4020590_0_4194304 (%!s(<nil>), 105.068s)

Make sure cache group peers are connected using 10 Gigabit Ethernet, use tools like iperf3 if you need to measure.

If you cannot easily improve cache group hardware (network and disk), be sure to use JuiceFS 5.1 or later. In previous versions, the JuiceFS client set cache group header timeouts to 10 seconds and body timeouts to 1 second. JuiceFS 5.1 extends these timeouts to 65 seconds and enhances concurrency control. This enables distributed caching to perform more effectively under network congestion.

Is there a black sheep within the cache group?​

Cache group's high performance really requires that all peers are equipped with network devices of similar bandwidth, and cache disks of similar available capacity. If any of the member nodes comes with poor network performance, it'll systematically hinder the overall performance.

cache group black sheep

As shown above, assuming an equal distribution of cache data between the three nodes, but one comes with poor network (could be potentially caused by a faulty --group-ip setting, binding the wrong net interface, or this node comes from a different network environment, not suitable for this cache group at all), then obviously, 1/3 of the cache requests will be served by this slow node, systematically degrading the cache group performance.

In a larger scale cache cluster, the black sheep problem can cause obscure performance issues difficult to debug, for example if there's only one bad node among all 200 members, the total affected requests percentage may be low enough to escape debugging. In this case, check Remote Cache Latency (server) to see if there's any members with abnormally high latency, as they are likely the culprit.

cache group black sheep monitor

As demonstrated above, one bad apple among the cache group can worsen the overall performance, in the Remote Cache Latency (server) panel, sort by the Send latency and find if the top ones are abnormally high, if that's indeed the case, check the problematic nodes and consider add a --no-sharing option to prevent them from serving cache requests.

Can the cache group access the object storage service at high speed?​

If cache group members cannot download data from object storage fast enough, you'll get similar problems listed above. Use a lower download concurrency to try to achieve a more stable warmup process, like juicefs warmup --concurrent=1.

Is client load too high?​

High client load can cause low hit rates for distributed cache. For example, client enabled --fill-group-cache option, so that processed data is contributed back to the distributed cache while written to the file system. If these clients run into performance problems due to high load, writes to JuiceFS file system will still succeed, but cache data exchange with peers will likely fail due to timeouts (cache write failures are not retried), in this case, --fill-group-cache doesn't work well, read requests on relevant files will cause penetration to object storage, resulting in a low cache hit rate.