Issues with redis faced on ERPNext 12

Frappe Framework Bug
1

Hi,

I’m facing this issue with ERPNext 12 running on VM (Allotted 2GB RAM and 2 Cores - Xeon processors on the host), and I haven’t been able to figure out a solution for it. While I’m usually using ERPNext (No development, just like a normal user), it stops working. Ultimately nothing works and I have to restart my VM machine.

Also, it’s happening a lot lately that after making some changes in .py when I fire bench restart, or even when opening a new DocType, say Sales Invoice, ERPNext hangs and stops loading. Since I’m using a reverse proxy, it shows me 502 Proxy Error after a few minutes.

Now, I got to restart the server to fix it. And it happens every time I make some changes in the code and it’s getting really annoying.

Then I saw worker.error.log, it showed issues with resources, all of them are now fixed by using the following (Except redis.exceptions.ConnectionError: Error 111 connecting to localhost:11000. Connection refused):

LimitNOFILE=65536 in /etc/systemd/system/redis.service
vm.overcommit_memory = 1 and net.core.somaxconn=1024 in /etc/sysctl.conf

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi

in /etc/rc.local

fs.inotify.max_user_watches=524288 in /etc/sysctl.conf

Here’s an excerpt from worker.error.log:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/redis/connection.py", line 484, in connect
    sock = self._connect()
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/redis/connection.py", line 541, in _connect
    raise err
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/redis/connection.py", line 529, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 97, in <module>
    main()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 18, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/scheduler.py", line 161, in start_worker
    start_worker(queue, quiet = quiet)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 147, in start_worker
    Worker(queues, name=get_worker_name(queue)).work(logging_level = logging_level)
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/rq/worker.py", line 502, in work
    self.register_death()
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/rq/worker.py", line 300, in register_death
    p.execute()
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/redis/client.py", line 2894, in execute
    return execute(conn, stack, raise_on_error)
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/redis/client.py", line 2749, in _execute_transaction
    connection.send_packed_command(all_cmds)
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/redis/connection.py", line 585, in send_packed_command
    self.connect()
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/redis/connection.py", line 489, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to localhost:11000. Connection refused.
09:18:07 RQ worker 'rq:worker:ubuntu.1785.default' started, version 0.12.0
09:18:07 *** Listening on default...
09:18:07 Cleaning registries for queue: default
09:18:07 RQ worker 'rq:worker:ubuntu.1787.long' started, version 0.12.0
09:18:07 *** Listening on long...
09:18:07 Cleaning registries for queue: long
09:18:07 RQ worker 'rq:worker:ubuntu.1788.short' started, version 0.12.0
09:18:07 *** Listening on short...
09:18:07 Cleaning registries for queue: short

It’s really frustrating as I’m hardly able to do anything. I’m sorry to bother you @rmehta, I just would be really grateful to you if you invest a little bit of your time on this.

Thanks!

2

Can you check if the supervisorctl is up? and also the ports in use

sudo supervisorctl status && netstat -tnlp | grep -i listen
3

Yes, it’s up and the ports are in use. There’s also one other annoying issue, whenever there’s in “Internal Server Error” popup on the desk, I reload the document hoping that it would load but nope. Again, I got to restart by VM.

This is after I did bench restart (ERPNext isn’t loading currently, all I did was open Sales Invoice):

tcp        0      0 127.0.0.1:13000         0.0.0.0:*               LISTEN      1790/redis-server 1
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:11000         0.0.0.0:*               LISTEN      1789/redis-server 1
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      2139/python3    
tcp        0      0 127.0.0.1:12000         0.0.0.0:*               LISTEN      1791/redis-server 1
tcp        0      0 0.0.0.0:8001            0.0.0.0:*               LISTEN      -               
tcp6       0      0 :::9000                 :::*                    LISTEN      2140/node       
tcp6       0      0 :::53                   :::*                    LISTEN      -               
tcp6       0      0 :::22                   :::*                    LISTEN      -               
tcp6       0      0 :::25                   :::*                    LISTEN      -

Please help :sweat_smile:

4

I am sorry, but why do you expect me to invest time in your issue? There are thousands of people on this forum who help if they have time. The software is free, that’s it.

Just because you use (or intend to use) ERPNext, does not mean any one person on this forum is obliged to help you. Its unreasonable and rude to expect it.

4 Likes
5

but open source culture is not a common culture specially if coming from another development practice.

it takes time for me to be familiar and still learning, meanwhile you can just ignore it :)…

6

It appears redis is not running.

Did you try starting redis manually?

7

There also appear to be 2 instances running - you have port 8000 as well as 8001 as active listeners. Maybe you need to reboot your server and get rid of the 2nd one

8

@rmehta, It was just a humble request and it’s nothing like rude or unreasonable I suppose.

I just posted because it wasn’t just my issue, it’s a global issue with ERPNext on VMs, so I thought it would be better if “you” knew it and might be helpful for everyone.

Thanks for replying anyways, and yeah I solved it.

9

Ignorance is a bliss

1 Like
10

Thanks for your help @trentmu, @crafter.

Actually the issue was finally fixed by removing –quiet from all three workers in Procfile.

1 Like
11

That is why this needs to be reinforced otherwise important notifications will also be ignored. Policing is clearly not my favourite job but someone needs to do it.

Culture requires enforcement and ensuring fairness in the community, so people don’t abuse attention.

1 Like
12

What merits consideration and a reminder -

As the grand marshal rmehta weathers a relentless storm of requests from all channels that require his review decisions - who among us can imagine or envy his workload…

1 Like
13

I appreciated your response sir, but I pretty sure that everyone in the world has lots of works to do and comparing them wouldn’t be ethical too. Also, unlike some of course, most of the people handle workload and maintain a good sense of general decency.

14

A solution was found, so I’m closing this thread.

1 Like