Redis Cluster and High Availability: Setting Up a Fault-Tolerant Cache for Microservices

Introduction — Why Redis Cluster Matters in a Microservices Architecture

Microservices architecture involves dozens or even hundreds of independent services, each potentially accessing a shared cache. A single Redis instance in such an environment becomes a bottleneck: it cannot scale horizontally, offers no fault tolerance, and is limited by the memory of a single server.

Redis Cluster addresses all these problems at once: data is automatically distributed across multiple nodes (sharding), each master node has replicas for automatic failover, and clients can redirect requests when the topology changes. For high-traffic microservices requiring fault-tolerant Redis with millisecond latency, this is the standard production solution.

Redis Sentinel vs Redis Cluster: When to Use Which

The two primary mechanisms for Redis high availability are Redis Sentinel and Redis Cluster. They solve different problems.

• Redis Sentinel — a monitoring and automatic failover system for a single master. Sentinel watches the master, elects a new leader from replicas upon failure, and notifies clients. Data is not sharded; the entire dataset is stored in one place. Best suited for moderate data volumes and simple scenarios.
• Redis Cluster — a horizontally scalable mode with automatic data sharding across multiple masters. Each master has its own replicas. Cluster supports up to 1,000 nodes and provides linear scaling for both read and write throughput. Best suited for large data volumes, high RPS, and horizontal scaling requirements.

When to choose Sentinel: your dataset fits on a single machine, you need simple failover, and there's no need to scale writes.

When to choose Cluster: your data doesn't fit on one server, you need horizontal scaling, high write RPS, or a microservices architecture with multiple independent data domains.

Redis Cluster Architecture: Sharding, Slots, and Replication

Hash Slots

Redis Cluster divides the key space into 16,384 slots (hash slots). Each key is mapped to one of these slots using a CRC16 hash. Slots are distributed evenly across master nodes. For example, in a three-master cluster:

• Master 1: slots 0–5460
• Master 2: slots 5461–10922
• Master 3: slots 10923–16383

When nodes are added or removed, slots migrate without stopping the cluster — this process is called resharding.

Replication

Each master node can have one or more replicas. Replication is asynchronous. When a master fails, the cluster automatically performs a failover: a replica is promoted to the new master. This requires a quorum — a majority of master nodes must agree. The recommended minimum is 3 masters + 3 replicas (6 nodes total).

Intra-Cluster Commands

If a key resides on a different node, Redis returns a MOVED or ASK response, and the client redirects the request. Smart clients cache the cluster topology and route requests directly to the correct node.

Step-by-Step Redis Cluster Setup in Docker and Kubernetes

Docker Setup

Create a configuration file for each node. Example redis-7001.conf:

port 7001\ncluster-enabled yes\ncluster-config-file nodes-7001.conf\ncluster-node-timeout 5000\nappendonly yes\nbind 0.0.0.0\nprotected-mode no

Create similar files for ports 7002–7006. Start the containers:

for port in 7001 7002 7003 7004 7005 7006; do\n  docker run -d --name redis-$port \\\n    --net host \\\n    -v $(pwd)/redis-$port.conf:/usr/local/etc/redis/redis.conf \\\n    redis:7.2 redis-server /usr/local/etc/redis/redis.conf\ndone

Initialize the cluster:

redis-cli --cluster create \\\n  127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 \\\n  127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 \\\n  --cluster-replicas 1

The --cluster-replicas 1 flag assigns one replica per master. Redis CLI will distribute nodes automatically.

Redis Cluster Setup in Kubernetes

For Kubernetes, use a StatefulSet and a Headless Service so that each Pod has a stable DNS name.

apiVersion: v1\nkind: Service\nmetadata:\n  name: redis-cluster\nspec:\n  clusterIP: None\n  selector:\n    app: redis-cluster\n  ports:\n    - port: 6379\n      targetPort: 6379\n    - port: 16379\n      targetPort: 16379\n---\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n  name: redis-cluster\nspec:\n  serviceName: redis-cluster\n  replicas: 6\n  selector:\n    matchLabels:\n      app: redis-cluster\n  template:\n    metadata:\n      labels:\n        app: redis-cluster\n    spec:\n      containers:\n        - name: redis\n          image: redis:7.2\n          ports:\n            - containerPort: 6379\n            - containerPort: 16379\n          command: ["redis-server"]\n          args:\n            - --cluster-enabled\n            - "yes"\n            - --cluster-config-file\n            - /data/nodes.conf\n            - --cluster-node-timeout\n            - "5000"\n            - --appendonly\n            - "yes"\n          volumeMounts:\n            - name: data\n              mountPath: /data\n  volumeClaimTemplates:\n    - metadata:\n        name: data\n      spec:\n        accessModes: ["ReadWriteOnce"]\n        resources:\n          requests:\n            storage: 1Gi

After deploying, initialize the cluster by connecting to one of the Pods:

kubectl exec -it redis-cluster-0 -- redis-cli --cluster create \\\n  $(kubectl get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 {end}') \\\n  --cluster-replicas 1

Working with Redis Cluster from Go and PHP Clients

Go: go-redis Library

The go-redis library supports Redis Cluster out of the box via redis.NewClusterClient.

package main\n\nimport (\n    "context"\n    "fmt"\n    "github.com/redis/go-redis/v9"\n)\n\nfunc main() {\n    rdb := redis.NewClusterClient(&redis.ClusterOptions{\n        Addrs: []string{\n            "redis-cluster-0.redis-cluster:6379",\n            "redis-cluster-1.redis-cluster:6379",\n            "redis-cluster-2.redis-cluster:6379",\n        },\n        // Automatically refresh slots on topology changes\n        RouteByLatency: true,\n        ReadOnly:       true, // read from replicas\n    })\n\n    ctx := context.Background()\n    err := rdb.Set(ctx, "user:1001", "Alice", 0).Err()\n    if err != nil {\n        panic(err)\n    }\n\n    val, err := rdb.Get(ctx, "user:1001").Result()\n    if err != nil {\n        panic(err)\n    }\n    fmt.Println("user:1001 =>", val)\n}

The ReadOnly: true option routes read requests to replicas, reducing load on masters. RouteByLatency selects the nearest node based on latency.

PHP: Predis and phpredis

The two most popular PHP clients are Predis and the phpredis extension.

Predis:

<?php\nrequire 'vendor/autoload.php';\n\n$client = new Predis\\Client(\n    [\n        ['host' => 'redis-cluster-0.redis-cluster', 'port' => 6379],\n        ['host' => 'redis-cluster-1.redis-cluster', 'port' => 6379],\n        ['host' => 'redis-cluster-2.redis-cluster', 'port' => 6379],\n    ],\n    [\n        'cluster' => 'redis',\n        'parameters' => [\n            'password' => null,\n        ],\n    ]\n);\n\n$client->set('order:555', json_encode(['status' => 'pending']));\n$value = $client->get('order:555');\necho $value;

phpredis (extension):

<?php\n$redis = new RedisCluster(null, [\n    'redis-cluster-0.redis-cluster:6379',\n    'redis-cluster-1.redis-cluster:6379',\n    'redis-cluster-2.redis-cluster:6379',\n]);\n\n$redis->set('session:abc123', 'user_data');\necho $redis->get('session:abc123');

An important note for PHP: when using multi-key operations (MGET, MSET), all keys must hash to the same slot. To achieve this, use hash tags — a portion of the key enclosed in curly braces: {user:1001}:profile, {user:1001}:settings. The cluster hashes only the part inside the braces, guaranteeing the same slot assignment.

Handling Failover and Reconnection on the Client Side

Failover in Redis Cluster takes 5–15 seconds (depending on cluster-node-timeout). During this window, some requests will fail. The client must handle these situations gracefully.

Error Handling Strategies

• Retry with exponential backoff: on receiving a CLUSTERDOWN or LOADING error, retry the request with increasing intervals (100ms, 200ms, 400ms...).
• Circuit Breaker: if the error rate exceeds a threshold, temporarily stop sending requests to Redis and use a fallback (e.g., an empty response or a database query).
• Topology refresh: upon receiving a MOVED response, the client should immediately fetch the current topology via CLUSTER SLOTS or CLUSTER SHARDS.

Retry example in Go:

func getWithRetry(ctx context.Context, rdb *redis.ClusterClient, key string) (string, error) {\n    var val string\n    var err error\n    for i := 0; i < 3; i++ {\n        val, err = rdb.Get(ctx, key).Result()\n        if err == nil {\n            return val, nil\n        }\n        if errors.Is(err, redis.Nil) {\n            return "", nil // key not found — not an error\n        }\n        time.Sleep(time.Duration(100*(i+1)) * time.Millisecond)\n    }\n    return "", fmt.Errorf("redis unavailable after retries: %w", err)\n}

Monitoring Redis Cluster: Metrics, Alerts, and Tools

Key Metrics

• cluster_state — must be ok. Any other value requires an immediate alert.
• cluster_slots_fail — number of slots in FAIL state. Must be 0.
• connected_slaves — number of connected replicas per master.
• used_memory vs maxmemory — memory utilization. Alert at 80% usage.
• instantaneous_ops_per_sec — current RPS.
• rejected_connections, evicted_keys — signs of overload.
• keyspace_hits / keyspace_misses — cache hit rate. Should stay above 90%.

Monitoring Tools

• Redis Exporter + Prometheus + Grafana — the standard stack. Deploy oliver006/redis_exporter as a sidecar or standalone Deployment, configure scraping in Prometheus, and use ready-made Grafana dashboards (e.g., ID 763).
• redis-cli cluster info — quick manual diagnostics: redis-cli -c -h redis-cluster-0.redis-cluster cluster info.
• redis-cli --cluster check — cluster integrity check: redis-cli --cluster check redis-cluster-0.redis-cluster:6379.
• RedisInsight — the official GUI tool from Redis Ltd. with topology visualization, slow query analysis, and memory inspection.

Prometheus Alert Example

groups:\n  - name: redis_cluster\n    rules:\n      - alert: RedisClusterDown\n        expr: redis_cluster_state != 1\n        for: 1m\n        labels:\n          severity: critical\n        annotations:\n          summary: "Redis Cluster is not in OK state"\n      - alert: RedisMemoryHigh\n        expr: redis_memory_used_bytes / redis_memory_max_bytes > 0.85\n        for: 5m\n        labels:\n          severity: warning\n        annotations:\n          summary: "Redis memory usage exceeds 85%"

Anti-Patterns and Common Mistakes with Redis Cluster

1. Using commands not supported in Cluster mode. Commands that operate on multiple keys from different slots (MGET, SUNION, EVAL with multiple keys) will throw an error. Solution: use hash tags to group keys into the same slot, or redesign the logic.
2. Storing "hot" keys on the same slot. If a single key receives 90% of the load, its master becomes a bottleneck. Solution: manually shard hot keys (user:counter:1, user:counter:2...).
3. Ignoring MOVED/ASK responses. Basic clients without Cluster support will receive errors when accessing slots hosted on a different node. Always use client libraries with native Redis Cluster support.
4. Running without replicas. Launching a cluster with no replicas (--cluster-replicas 0) means losing any master leads to data loss and cluster degradation. In production, always have at least one replica per master.
5. Setting cluster-node-timeout too low. An overly aggressive timeout (below 2000ms) can trigger false failovers due to brief network hiccups. Recommended value: 5000–15000ms.
6. Lua scripts accessing keys from different slots. EVAL in Redis Cluster requires all keys to reside in the same slot. Violating this rule will produce a CROSSSLOT error.
7. Not monitoring cluster_state. The cluster can silently enter a FAIL state if no alerts are configured. Monitoring is mandatory in production.

Conclusion

Redis Cluster is a mature and reliable solution for building a fault-tolerant cache in a microservices architecture. Automatic sharding across 16,384 slots, built-in failover, and horizontal scalability make it the go-to choice for high-load systems where a single Redis instance is no longer sufficient.

When properly configured — with the right number of replicas, well-designed clients in Go or PHP, comprehensive monitoring via Prometheus and Grafana, and thoughtful use of hash tags — Redis Cluster provides a solid foundation for caching in Kubernetes environments and distributed systems.

Key production recommendations: at least 3 masters + 3 replicas, cluster-node-timeout of no less than 5000ms, monitoring of cluster_state and used_memory, client-side retry logic, and hash tags for multi-key operations. Following these principles will give you a stable, scalable, and fault-tolerant Redis for your microservices.