Data Import Tool hangs on ERP Next instance running in Production Mode

[gvyshnya]

Hi guys,

We observe quite a weird issue with Data Import Tool on an instance of ERP Next running in production mode.

The data import tool will freeze (with no further changes to the import progress bar and no error messages shown in UI) and hang indefinitely. At the same time we see the following in web.error.log


[2016-12-12 16:13:15 +0000] [23553] [CRITICAL] WORKER TIMEOUT (pid:23568)
[2016-12-12 16:13:15 +0000] [23568] [INFO] Worker exiting (pid: 23568)
[2016-12-12 16:13:15 +0000] [23585] [INFO] Booting worker with pid: 23585


No other error messages detected in any of different error logs on the backend.

We tried increasing the worker timeout values (actually we doubled it) with no luck.

We were trying to upload csv files with about 5000 records for Address DocType as we observed this problem. The same csv file was processed successfully by Data Import Tool of another ERP Next instance configured to run in Developer Mode.

Has anyone observed the issues like this in past? What is the best way to address it?

Just for the record, both ERP Next instances referred above run the same configuration as follows

• ERPNext: v7.1.23
• Frappe Framework: v7.1.24

[nabinhait]

The server request might be timing out, please try to import in a smaller batches.

[gvyshnya]

@nabinhait: thanks for the suggestion. We will try this approach the next time we are embarked on a massive data import activity.

P.S. For the specific case we had last night, we temporarily enabled Developer mode on the target ERP N instance, uploaded files in the batches of ~5k records and then switched the instance back to Production mode.

[felix]

If it works in dev, it will be a problem in either gunicorn or nginx.

in gunicorn, check --limit-request-line

in nginx, check client_max_body_size

[gvyshnya]

@felix: thanks for the suggestion, we will definitely try it out!

[gvyshnya]

Just to follow up on what we did after @felix and @nabinhait shared their suggestions, here is the summary

• we increased the session timeout on nginx up to 30 min
• we increased the session timeout on gunicorn up to 30 min
• we trippled the number of workers to start by gunicorn vs. its default value

The combination of those three configuration changes helped to cook our noodles, and we were able to load the batch import file with the size of about 5k records (for Address DocType) to an instance of ERP Next with Production mode set on.

Thank you once again for helpful advise.

[sayyucel]

in gunicorn, check --limit-request-line

in nginx, check client_max_body_size

Where are they and how can we check?
could you share the path?

[froldan]

Hello @gvyshnya, can you tell me where did you add those changes?
Thanks!

[clarkej]

May I jump in here…

google ‘gunicorn --limit-request-line’ will get you this http://docs.gunicorn.org/en/stable/settings.html

A forum search on ‘gunicorn’ will get you for eg this Gunicorn Worker how to increase

and for your 2nd case this local search

frappe@erpnext:~/frappe-bench$ find . -name ‘*.conf’ | xargs grep “client_max_body_size”
./frappe-bench.conf: client_max_body_size 50m;
./config/nginx.conf: client_max_body_size 50m;