[Performance] Load balancing requests between multiple ERPNext servers

We are currently running multiple ERPNext server and load balance request between them. The current load balancing is based on simple URL pattern. All the HTTP requests for Reporting calls are diverted to a secondary server where as others are routed to Primary ERPNext server.

This routing is done in an external nginx webserver running in-front of ERPNext servers -

map $arg_cmd $backend_servers {
  "frappe.desk.query_report.run" secondary_servers;
}

upstream secondary_servers {
  server wra02.ntex.com;
  server wra03.ntex.com;
}

location / {
    if ( $backend_servers ) {
      add_header x-served-from $backend_servers;
      proxy_pass http://$backend_servers;
      break;
    }

    add_header x-served-from "master";
    proxy_pass http://primary_server;
}

While this routing works like a charm for GET calls, we cant use the same for POST calls as the cmd argument is part of the HTTP body.

On option available is to write a Lula script in nginx and parse the body to the get the value of cmd parameter and do the routing. However this is expensive.

We propose a change in frappe framework to pass the cmd as part of HTTP header as well so as to make it easy to retrieve the same in webserver configuration and make routing more powerful

request.js

frappe.request.call = function(opts) {
	frappe.request.prepare(opts);

        ...
        ...
        ...

	var ajax_args = {
		url: opts.url || frappe.request.url,
		data: opts.args,
		type: opts.type,
		dataType: opts.dataType || 'json',
		async: opts.async,
		headers: {
			"X-Frappe-CSRF-Token": frappe.csrf_token,
			"Accept": "application/json",
			**"X-Frappe-CMD": opts.args.cmd**
		},
		cache: false
	};

The above change would make the cmd available in a custom header - `X-Frappe-CMD’. This can serve multiple purposes right from logging to load balancing.

What does dev community think of above?

9 Likes

PR sent

https://github.com/frappe/frappe/pull/7228

We are using these headers (X-Frappe-CMD and X-Frappe-Doctype) to do load balancing across our backend ERPNext servers as well as analysing performance aspects from nginx logs using ELK. We have logged these HTTP headers in nginx log file and same has been fed to ELK. It makes it very easy to know the POST and GET request patterns and time taken by different type of requests.

2 Likes

can u show more about ur way to build multi server frappe docker and load balancing between them. How about redis cache, socketio, should i use internal redis cache container in docker compose for each server or use 1 external redis container for all.