Updating ERPNext on AWS Free Tier hangs

Frappe Framework
1

I have noticed that since the last two updates of v12 latest being
erpnext 12.10.1
frappe 12.8.3
bench update fails at a stage where it is building frappe assets

Building frappe assets...

✔ Built js/modules.min.js
✔ Built js/checkout.min.js
✔ Built js/dialog.min.js
✔ Built js/social.min.js
✔ Built js/web_form.min.js
✔ Built js/chat.js
✔ Built js/list.min.js
✔ Built frappe/css/email.css

I believe this occurs due to some hard limits on the free tier. I am not sure. Can anyone point to which logs to check.

bench.log does not help much

ubuntu@XXXXX:/home/frappe/frappe-bench/logs$ tail bench.log
2020-07-13 12:56:42,047 INFO installing frappe
2020-07-13 12:56:57,410 INFO installing erpnext
2020-07-13 13:55:00,803 INFO pulling frappe
2020-07-13 13:55:05,027 INFO pulling erpnext
2020-07-13 13:55:09,358 INFO installing frappe
2020-07-13 13:55:22,546 INFO installing erpnext
2020-07-27 19:43:42,002 INFO pulling frappe
2020-07-27 19:43:45,630 INFO pulling erpnext
2020-07-27 19:43:51,365 INFO installing frappe
2020-07-27 19:44:10,127 INFO installing erpnext

In case it is due to some hard limit is there a workout.

Last time the project failed like this I had to clear the under maintenance mode flag
bench --site site-name set-maintenance-mode off

This time though that did not happen however, I did not get a message after a new login saying what all things have changed like is common whenever we update.

Edit: My AWS OS is Ubuntu 18.04

2

I have no idea what OS you are running within AWS instance (maybe you can edit your post telling us what OS you are running).

Going straight to your point, I always have problems running ERPNext on t3.micro, especially in building assets.
The suggestion is to upgrade to at least to a t3.small in production.
If you really want to use a t3.micro, my suggestion is to look for memory issues.

Dig into these details (stack overflow, this forum, and bench sources on Github help you to understand better what I listing below):

  1. Try to add more swappiness, something like 60/70. you can set it in sysctl.conf

  2. ERPNext needs vm.overcommit_memory set to 1, but doing that, so the application allocates more memory than it really need. Try to set it to 0 (bench start if you running dev will notice you about this change)

  3. If you installed ERPNext through the easy install process, MariaDB conf it allocates a lot of memory for its own processes, that for a dev environment it’s not needed. Check mariadb conf

1 Like
3

Same issue for me when I try bench build.

I usually reboot the machine and do it again. It takes around 20 minutes to complete then.

4

Try this as well:

1 Like
5

I have experienced bench build performance issues too.
Mostly while building the frappe app :sweat_smile:

Having more RAM (4GB or more) generally helps.

6

Thank you all for the suggestions. I will try and update soon.

7

sudo supervisorctl stop all; Do your stuff, then: sudo supervisorctl start all;

This seemed the most easy to try.
Unfortunately, I have some custom script in there and it crashed when doing this

Updating customizations for Address

Traceback (most recent call last):
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/redis/connection.py", line 484, in connect
    sock = self._connect()
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/redis/connection.py", line 541, in _connect
    raise err
  File "/home/frappe/frappe-bench/env/lib/python3.6/site-packages/redis/connection.py", line 529, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred: 
.
.
.

So looks like with redis turned off that might not work or did I miss something here

9

Before doing supervisor stuffs, try sudo systemctl enable redis && sudo systemctl start redis and bench setup redis

10

Thank you for the post

For this specific point from MariaDB website says

You should set innodb_buffer_pool_size to about 80% of your memory. The goal is to ensure that 80 % of your working set is in memory!

Is that something useful for us in this situation ? Haven’t done DBs for a decade now.
Can I change that to some lower value during update and spare myself the trouble

PS: I also edited my post to include the OS

11

Are you in dev or production?

If you are in dev low it down up to 50% of memory and try what’s happening

12

Says this

ubuntu@XXXXXX:/home/frappe/frappe-bench$ sudo systemctl enable redis
Failed to enable unit: Refusing to operate on linked unit file redis.service
13

Its very low throughput production. Early days.

So I can reduce this for a while during update. Let me try and update

14

Changed innodb_buffer_pool_size from 256M to 128M and restarted mysql with
sudo service mysql restart

Ran bench update and no luck

15

try to modify this in system unit file:

[Unit]
Description=Redis persistent key-value database
After=network.target

[Service]
Type=forking
PIDFile=/var/run/redis/redis.pid
ExecStart=/usr/bin/redis-server /etc/redis.conf
ExecReload=/bin/kill -USR2 $MAINPID
ExecStop=/usr/bin/redis-shutdown
User=redis
Group=redis

[Install]
WantedBy=multi-user.target

and on /etc/redis.conf file I edit this line :

daemonize yes
pidfile /var/run/redis/redis.pid
1 Like
16

Try put overcommit_memory = 0 and swappiness = 70

Btw, ERPNext is made to run on machines with more than 1GB of RAM, until using version 11, you can survive with t3.small, but using v12 or v13 it’s really not enough in a production system

My strong suggestion is that you pass to a bigger instance like t3.small.
Trust me you gonna waste time, and when you find the balance the overall performances of ERPNext will be quite slow.

Another test you can try is using a distro like centos or “play” with overcommit_ratio in order to split the memory between RAM and swap, but from here we gonna walk in the danger zone if you don’t know what to do

https://engineering.pivotal.io/post/virtual_memory_settings_in_linux_-_the_problem_with_overcommit/

17

I guess I will take you advice and move to a bigger machine to save time now which was eventually necessary anyways.

But that was a useful exercise to learn and I guess I will post when I can experiment more.
I still have not done the redis one especially and will update if that helps

Thanks for all your help.