It is really hard to tell in your case since you are two versions behind, I’m sure many things have been optimized since. why don’t you to import your data on clean v9 version and do real test and benchmarking there.
Yes, I’ll do it, will be tedious lol, but I can’t use v9 non production; I will import just to bring useful feedback here, except the online pos has been perfected, the pos is why I stayed on v6. Its a solid version, serving extremely well just these shops that have grown so large slowing down. Many of our clients have dumped quickbooks and sage for erpnext and they keep loving the feature set.
1 Like
I was shocked at the instability of v9 offline pos and the dependence on an open source, user accessible cache. but once we cross this completely in the community; I can migrate the clients. I will get v9 and confirm the online pos has no dependency on the cache especially for critical things like the invoices themselves or even prices!
Can you confirm at what point you started noticing the slow down in invoice submission, was it the first or third month of operations?
Do you know if this also affects submission of non PoS sales invoices ?
You said one of the affected clients has six locations, are all six locations using one database or do you have each location on separate database ?
If on one database how have you structured access by the individual locations?
Regards
@noetico Install NewRelic APM (you can use their free plan) and see where the slowdowns are. Its been extremely helpful for us.
Ok; started say 6 months after. But that’s because of their rapid growth; we have over 13 more clients with nearly 3 years of transactions even on wireless and they’re still good and going. Just these 2 with very high sales.
Then; it affects submission of invoices, pos or form view, also old PRs an POs take a while, same day; maybe 5 to 10 seconds to submit; I’m particular about pos cus that’s whr speed is critical. It’s really slow on every submission due to the size of data; a lot of queries as @ganas pointed out; the queries haven’t been optimized for bigger businesses with thousands of items. Mind you this is still medium size afaik.
No; ok; the client with 6 locations is not affected; not at all; they’re running strong; in fact theirs is still a v4 installation, running in their own pentium processor server (basic system). With the other branches linking by public ip address. Their total sales is over 1.3 billion naira for the past 3 years; impressive; but the difference is; its not supermarket style retail; its furniture; so they may not have up to 20 to 30 invoices daily; they have high value and slow paced items so everything is cool.
ok thanks, will do. though this is a local installation.
v4 is still a very lovely implementation, solid performance and quality, they also have huge item list with images on many and a lot of stock entries made moving stock around, big accounting figures and transactions, a lot of pending invoices as they have credit sales etc![01 AM|612x500] over 20 cashiers; all running from a pentium processor, windows 7, 4 gb ram, 7200 rpm hdd, this system is a ‘toy’ compared to the one running v6, yet it handles this so well and I don’t even hear from these guys in a whole 9 months. runs like a horse, obviously the best build when you talk about quality and performance. Pic attached:
1 Like
v4 looks so good, so ERP!! why didn’t we just build on it lol lol 
I might have skipped this info from the thread but can you check the slow logs in mariadb.
Thanks Tunde for showing up on this thread. I have potential ERPNext clients that can easily blow past the figures posted by @noetico for his “problematic” database. I am not a techy so I would really like to know what the bottle neck is and see if it is a solveable problem.
Regards
True talk. It’s also worrying to me that ERPNext should be so slow on such a setup. That’s the reason why I asked @noetico to check his slow logs. With that, we can narrow down the areas we should be investigating. At the end of the day, we might simply need to add more indexes to the tables or change to a more efficient query under the hood. But when the slow logs results come out, I believe we should be able to determine the root cause quickly.
Hi @tundebabzy @olamide_shodunke
Trust you’re doing great. I’ve just gone through this thread and quite interested in seeing the end of this. I however believe that the cause of the issue might have been addressed in later iterations. One of the retail setups I manage currently has almost 20 locations with hundreds of transactions daily
Server is a Duo core with 8 Gigs of Ram and it works great! We had an issue recently with serious delays in selecting a Customer on the POS screen but that was promptly handled by the ERPNext team and now it’s back to it’s usual snappy self
BTW @noetico have you checked the Innodb Buffer Pool size? That is one of the usual suspects when things start running slow
Kind regards,
Yeah, I remember the issue which was subsequently fixed by Rohit. @noetico, is the upgraded instance on v9-latest?
Ok; thanks Wale; I think I have done the innodb buffer pool size optimization, will double check and get back to you.
For the site; we have pos profiles so the cashier customer profiles are automatically entered per invoice. No performance issues with that part, just submitting takes a toll. This innodb thing has to do more with ram, but looking at the behaviour, the scripts rely on processing speed and frequency; if you read some of my posts you will see that as you give the server a higher processor it greatly improves. We’re on a 2.9ghz core i7 and its ok.
Here’s how they manage with 10sec+ submission time: as they hit submit; they arrange any change and get th goods packed in, most of the time, it’s submitted even before they are done and shut the cahs drawer.
hi @noetico,
increase innodb_buffer_pool_size to half of you server memory, if this doesn’t give you real benefits:
Can you post the output of iostat during that operation?
Can you see the output of slow queries in mysql?
If the problem is really lack of CPU power (I’m not convinced), and from your SS you are only using on core of the CPU because you use only one connection and not partitioning (and this is ok) so you only gain with CPU upgrade if the CPU clock speed is significantly higher (or a much efficient CPU) you will not have positive effect with an upgrade with more cores of with a similar cpu.
But if the problem is in CPU core speed and not in IO, I think that mysql optimizations with server parameterization,SQL queries (frappe & erpnext enhancements) and better Indexes (also frappe & erpnext enhancements) will make miracles withou any cpu upgrade. IMO mysql High CPU usage reflects bad indexes. To look at this you should really upgrade the erpnext (maybe these indexes benefits I’m talking about are already done in a recent version).
hello Paulo; how do I run this iostat and the slow queries log; haven’t done this in a while? I’ll see where I put the innodb_buffer_size, around 4G but I will confirm, increase and report back.
Regarding the cpus; on the core i7 system we have 2 available for the vm, like I detailed somewhere; at some point we tried with a dell r610 16 core (8 physical) processor server with 180gb ssd hard drive; this machine was slower than the core i7 even though we threw 24gb or ram and 8 cpus to the vm; you will see erpnext using up to 4 cores sometimes yet it was slower; we came to the conclusion that the speed to finish is so crucial as the 2 core i7 cores did a lot better; in fact all this while they’ve been selling on it. In all these also; let me mention that the host os was also a factor. Windows 10 was the best and fastest for the vm
I’m going to try to move the data into a clean v9 install because upgrading just doesn’t work for me!
Long story.
Please guide on the slow queries, I haven’t done this before. Thanks, expecting.