Redis Cluster MOVED error

Install / Update Setup
, ,
1

Description of the issue

I installed the ERPNext with Helm and also I installed bitnami/redis-cluster chart for redis-queue and redis-socketio on my K8S cluster. ERPNext worker nodes throwing an error after the connected to Redis Cluster.

Is ERPNext not supporting Redis Cluster?

Context Information

erpnext 13.13.0
frappe 13.13.0
redis-cluster 6.2.6
bitnami/redis-cluster helm version 7.0.8
frappe/erpnext helm version 3.2.31

Expected result

All ERPNext pods running without throw exception.

Stacktrace / full error message

Attempt 1 to connect to mariadb:3306
Attempt 1 to connect to redis-queue:6379
Attempt 1 to connect to redis-cache-master:6379
Attempt 1 to connect to redis-socketio:6379
Connections OK
Traceback (most recent call last):
  File "/home/frappe/frappe-bench/commands/worker.py", line 12, in <module>
    main()
  File "/home/frappe/frappe-bench/commands/worker.py", line 7, in main
    start_worker(queue, False)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 155, in start_worker
    Worker(queues, name=get_worker_name(queue)).work(logging_level = logging_level)
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/rq/worker.py", line 550, in work
    self.register_birth()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/rq/worker.py", line 286, in register_birth
    if self.connection.exists(self.key) and \
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/redis/client.py", line 1581, in exists
    return self.execute_command('EXISTS', *names)
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/home/frappe/frappe-bench/env/lib/python3.7/site-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ResponseError: MOVED 10476 10.23.2.124:6379
2

Check your redis cluster configuration. I’ve not setup and used redis cluster on my own.

I’ve used AWS ElastiCache redis cluster and it worked as expected.

did you search about this error?

redis.exceptions.ResponseError: MOVED 10476

try if searching helps.

3

Hi,

I already checked my redis-cluster configuration. It’s working with another application.
Also when I was searching the error I saw to the error about Redis Client. The moved error mean is server redirect you to another redis node.

On AWS, did you give all redis hosts to your redis url like redis://redis1:6379,redis2:6379,redis3:6379 or give one load balancer name like redis://redis:6379?

4

It was just single hostname. Nothing separated by comma.
I used databases feature of redis instead of redis1, redis2, redis3

e.g.
redis://redis:6379/0
redis://redis:6379/1
redis://redis:6379/2

redis_* keys in common_site_config.json, I think are fed to redis python client / nodejs socketio.

So you can try things.

The only problem I think may happen is with healthcheck script https://github.com/frappe/frappe_docker/blob/main/build/frappe-worker/healthcheck.sh