Progress bar display problem due to gunicon workers

bomsy · August 26, 2017, 9:10pm

Hi all,
we currently have a custom app dependent on ERPNext v8.0.44 and Frappe v8.0.64
Every month, we trigger a task to bill about 1000 customers and remind them
every 7 days of unpaid invoices. When this task is triggered, a progress bar is
displayed to enable the user monitor the process.

def update_progress(self,start,stop,task='bill'):
		'''Publish progress to UI.'''
		show_progress = getattr(self,'show_progress',False)
		if show_progress:
			percent = (start*100)/stop
			frappe.publish_progress(percent,self.get_progress_title(stop,task=task))

The issue is when the value for gunicorn workers in common_site_config.json
is set to 1, the progress bar displays and updates as expected for most of the time
but the whole site remains in a sort of ‘task processing’ state ie no one can access the site
untill the task is done, but when the variable is increased from 1(to 2,3 and 4), the task still runs in
the background but the progress bar displays occasionally (sometimes never, until a refresh is done
multiple times on the page)
A solution/explanation for these behaviours would be appreciated.
Thanks

rmehta · August 28, 2017, 6:18am

Why do you want only 1 worker. I think that is the problem. When your worker is busy, it does not allow other web-requests to process.

Progress bar is updated with socketio, so it does not matter.

bomsy · August 28, 2017, 8:24am

Thanks @rmehta

Why do you want only 1 worker

bench setup configures that here automatically

github.com

frappe/bench/blob/develop/bench/config/common_site_config.py#L55


      
          def put_config(config, bench_path="."):
          	config_path = get_config_path(bench_path)
          	with open(config_path, "w") as f:
          		return json.dump(config, f, indent=1, sort_keys=True)
          
          
          def update_config(new_config, bench_path="."):
          	config = get_config(bench_path=bench_path)
          	config.update(new_config)
          	put_config(config, bench_path=bench_path)
          
          
          def get_config_path(bench_path):
          	return os.path.join(bench_path, "sites", "common_site_config.json")
          
          
          def get_gunicorn_workers():
          	"""This function will return the maximum workers that can be started depending upon
          	number of cpu's present on the machine"""
          	import multiprocessing

multiprocessing.cpu_count() is known to often give erratic results

When your worker is busy, it does not allow other web-requests to process.

I figured this.

Setting the number of workers isn’t really a problem,as even when it’s increased, the progress bar doesn’t show unless the page is refreshed multiple times

Progress bar is updated with socketio, so it does not matter.

so, I guess the main issue is that, the progress bar is not being displayed as needed.

I would be grateful if there is a solution to this.

Thanks a million.

rmehta · August 28, 2017, 8:26am

The settings we come up with are based on best guess! You can tweak your settings as you need.

I think your core problem is that your server does not have multiple processes. When you are trying to do parallel tasks, it helps if you have a multi-core CPU!

bomsy · August 28, 2017, 8:34am

I think your core problem is that your server does not have multiple processes

It’s on my local machine “Intel Core i3”.
I really don’t think having a mulit-core is the issue here. I think the issue is why socketio doesn’t update the progress bar when needed even on a multicore.

Thanks @rmehta