ERPNext Slow with multiple POS Users

[wale]

Hi

When we have about 15 users logged in (mostly POS Users), ERPNext becomes slow. Trying to open anything takes time and the screen Grey’s out for a while as though the system is busy with something… and this happens even during intervals when no sales are getting synced (we use offline POS)

This is happening on an AWS server with 16G RAM and 4 CPUs. There’s enough free RAM and Innodb Buffer Pool size is set to 8GB

What could be the issue and how do we improve this?

Also, do sales coming in from Offline POS transactions get handled as background jobs? If not, shouldn’t they?

Thanks

[felix]

1. Sounds like you need to increase the number of gunicorn workers
2. Are you using general purpose (SSD) storage for the database?

[wale]

Hi @felix

Thanks for your response

1. What number do you suggest?
2. Yes I am

Please advise

[felix]

What number do you suggest?

How many do you have now? The max recommended is 2*(number of cpu cores) + 1, but since you’re also running the DB and other things on the server, you’ll need to readjust. Test some different values and see where the sweet spot is.

[wale]

How many do you have now?

Currently I’ve got 3 gunicorn workers and 3 background workers. Are you suggesting to increase that?

Also, do you know if transactions from the offline POS are being handled as background jobs?

Thanks

[felix]

I don’t believe so. I would increase the number of gunicorn workers to 5 and see if that helps.

[wale]

Hi @felix

Trust you’re doing great. Seems like there’s a correlation between gunicorn workers and CPU usage. Yesterday evening I bumped the number of workers up to 5 as suggested and there seemed to be a significant improvement but as of this morning it became very slow again. I checked the CPU Utilization % and it was practically hitting 100

I’ve adjusted to 4 workers now but performance is still not okay. I’m beginning to consider moving this account off AWS to a platform where I can get good service and higher capacity at less cost. I still think the current capacity should be more than adequate for our use but not sure what else to tweak

Any other suggestions before I jump the boat?

Thanks

[olamide_shodunke]

Am no techie techie but 16GB RAM and 4 CPUs is more than enough for 15 users.

From the experience of @noetico adding more capacity/processing power will not solve the problem when the issue is not basically with the processing power/memory capacity of your server .

I think you should try and get the attention of the ERPNext guys, this is a problem I also see myself contending with and I am really interested in the cause/resolution.

Regards

[clarkej]

Yup to zero in on code bottlenecks use real-time profilers https://docs.python.org/2/library/profile.html

[wale]

Hi @olamide_shodunke

Thanks for your input. I’ll try to reach out and troubleshoot some more for another day or two but you know how it is in the world of retail… everything is urgent!

Cheers!

[wale]

Thanks a lot @clarkej

I’ll look through this and hope it helps shed some light on areas of improvement

Cheers!

[felix]

Any other suggestions before I jump the boat?

If you are using a T instance on AWS, those are unreliable for processing power - they are great for bursts but not so great for 100%+ utilization over time. Look at an M or C instance.

Honestly, if you’re not autoscaling or have abnormal security requirements, you would most likely save a lot of money switching to Digital Ocean, Linode, or Scaleway.

[saidsl]

@wale

Another option you might want to consider is separating your DB and Application on separate servers.

I tested this solution on VM and it worked seamlessly as well offered far better performance as opposed to having the everthing managed on the one server.

[felix]

Another option you might want to consider is separating your DB and Application on separate servers.

The downside of this on AWS is that they don’t allow custom instance sizes like Google Cloud. The DB needs more memory, while the App server needs more CPU. On AWS, if you split the app and db, you’ll most likely overprovision memory on the App server. Once you reach a certain CPU need, this becomes quite expensive.

[saidsl]

@felix

you would most likely save a lot of money switching to Digital Ocean, Linode, or Scaleway

As you suggested earlier and was thinking DO since they offer flexible configurations.

[wale]

Hi @felix @saidsl

Thanks for the input and suggestions. I decided to try a 32GB RAM, 8 CPU Server to see if that would solve the problem. While there’s an improvement, there’s still a significant lag during peak periods. Could it be a fundamental flaw in the design of the POS? (I’m referring to offline POS here as this is what we use)

Why isn’t syncing of transactions from the POS handled as a background job? Is there a technical reason why this isn’t feasible? Other ERPNext users with a decent number of concurrent POS users have also complained of performance issues. I’d like to know if there’s a technical reason for not using background jobs before I open a Github issue for this

Thanks

[felix]

Thanks for the input and suggestions. I decided to try a 32GB RAM, 8 CPU Server to see if that would solve the problem. While there’s an improvement, there’s still a significant lag during peak periods. Could it be a fundamental flaw in the design of the POS? (I’m referring to offline POS here as this is what we use)

Unfortunately, I’m not going to be able to comment further on this because there are too many unknowns and I don’t use the POS. Maybe install an APM solution like NewRelic to see where the slowdowns are. As a test, try disabling the offline mode - do the slowdowns remain?