Bulk API usage kills ERPNext

I was using curl in a loop to insert about 5000 Sales Invoices and to attach a file to each one after insertion.

The process slowed down gradually, then threw the werkzeug error you’ll see in the trace.

The process was started with bench not supervisor. I stopped it with <ctrl>-C

The second trace shows it took that nearly a full minute for the bench to shut down.

Is this solvable with configuration optimizations, or do I need to look at choking my curl loop?

(V12 upgraded to V13. Ubuntu 20.04)

07:02:14 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:14] "POST /api/resource/Sales%20Invoice HTTP/1.0" 500 -
07:02:14 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:14] "POST /api/resource/Sales%20Invoice HTTP/1.0" 500 -
07:02:14 web.1            | Traceback (most recent call last):
07:02:14 web.1            |   File "/home/erpdev/frappe-bench-DYPW/apps/frappe/frappe/middlewares.py", line 16, in __call__
07:02:14 web.1            |     return super(StaticDataMiddleware, self).__call__(environ, start_response)
07:02:14 web.1            |   File "/home/erpdev/frappe-bench-DYPW/env/lib/python3.8/site-packages/werkzeug/middleware/shared_data.py", line 220, in __call__
07:02:14 web.1            |     return self.app(environ, start_response)
07:02:14 web.1            |   File "/home/erpdev/frappe-bench-DYPW/env/lib/python3.8/site-packages/werkzeug/middleware/shared_data.py", line 220, in __call__
07:02:14 web.1            |     return self.app(environ, start_response)
07:02:14 web.1            |   File "/home/erpdev/frappe-bench-DYPW/env/lib/python3.8/site-packages/werkzeug/local.py", line 231, in application
07:02:14 web.1            |     return ClosingIterator(app(environ, start_response), self.cleanup)
07:02:14 web.1            |   File "/home/erpdev/frappe-bench-DYPW/env/lib/python3.8/site-packages/werkzeug/wrappers/base_request.py", line 240, in application
07:02:14 web.1            |     return resp(*args[-2:])
07:02:14 web.1            | TypeError: 'NoneType' object is not callable
07:02:37 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:37] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:37 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:37] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:39 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:39] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -
07:02:39 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:39] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -
07:02:41 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:41] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:41 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:41] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:43 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:43] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -
07:02:43 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:43] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -
07:02:44 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:44] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:44 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:44] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:48 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:48] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -
07:02:48 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:48] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -
^C07:02:53 system           | SIGINT received
07:02:53 system           | sending SIGTERM to redis_cache.1 (pid 127894)
07:02:53 system           | sending SIGTERM to redis_socketio.1 (pid 127895)
07:02:53 system           | sending SIGTERM to redis_queue.1 (pid 127893)
07:02:53 system           | sending SIGTERM to web.1 (pid 127899)
07:02:53 system           | sending SIGTERM to socketio.1 (pid 127896)
07:02:53 system           | sending SIGTERM to watch.1 (pid 127909)
07:02:53 system           | sending SIGTERM to schedule.1 (pid 127913)
07:02:53 system           | sending SIGTERM to worker_short.1 (pid 127925)
07:02:53 system           | sending SIGTERM to worker_long.1 (pid 127928)
07:02:53 system           | sending SIGTERM to worker_default.1 (pid 127938)
07:03:15 system           | watch.1 stopped (rc=-15)
07:03:21 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:23 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:23 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:23 system           | sending SIGKILL to web.1 (pid 127899)
07:03:23 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:29 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:29 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:29 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:29 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:29 system           | web.1 stopped (rc=-15)
07:03:29 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:29 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:29 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:29 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:29 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:29 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:29 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:30 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:30 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:30 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:30 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:30 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:33 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:33 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:33 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:33 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:33 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:33 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:33 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:33 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:34 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:34 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:34 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:34 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:34 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:34 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:34 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:34 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:35 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:35 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:35 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:35 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:35 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:35 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:35 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:35 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:35 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:35 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:35 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:35 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:35 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:36 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:36 system           | schedule.1 stopped (rc=-15)
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:36 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:36 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:36 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:03 redis_socketio.1 | 127900:signal-handler (1605268973) Received SIGTERM scheduling shutdown...
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:36 system           | worker_long.1 stopped (rc=-15)
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | worker_short.1 stopped (rc=-15)
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:07 redis_queue.1    | 127905:signal-handler (1605268973) Received SIGTERM scheduling shutdown...
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_queue.1    | 127905:M 13 Nov 2020 07:02:54.219 # User requested shutdown...
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:04 redis_cache.1    | 127903:signal-handler (1605268973) Received SIGTERM scheduling shutdown...
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 redis_cache.1    | 127903:M 13 Nov 2020 07:02:54.218 # User requested shutdown...
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 redis_cache.1    | 127903:M 13 Nov 2020 07:02:54.339 * Removing the pid file.
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 redis_cache.1    | 127903:M 13 Nov 2020 07:02:54.420 # Redis is now ready to exit, bye bye...
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:37 system           | redis_cache.1 stopped (rc=-15)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_socketio.1 | 127900:M 13 Nov 2020 07:02:54.072 # User requested shutdown...
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_socketio.1 | 127900:M 13 Nov 2020 07:02:54.450 * Removing the pid file.
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_socketio.1 | 127900:M 13 Nov 2020 07:02:54.450 # Redis is now ready to exit, bye bye...
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:37 system           | redis_socketio.1 stopped (rc=-15)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_queue.1    | 127905:M 13 Nov 2020 07:02:54.463 * Removing the pid file.
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_queue.1    | 127905:M 13 Nov 2020 07:02:54.463 # Redis is now ready to exit, bye bye...
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:37 system           | redis_queue.1 stopped (rc=-15)

Use data import via rest api.

Wait for one data import to complete before starting another batch.

5000 individual requests from app may also slow down the system for users.

Would you mind elaborating on that? I’m not sure what you mean.

All steps are done programmatically:

  1. Prepare csv template, create .csv file
  2. Create data import, attach the previous .csv
  3. Start import using whitelisted endpoint.
  4. Poll for completion/failure of data import.
  5. In case of success, update data
  6. In case of failure, show error to user for intervention

Sorry, I didn’t mean for you to go into so much detail. I understand the process and I understand csv files. I was unaware of a whitelisted end point.

So this is what I need?

export const DATA_IMPORT_API_ENDPOINT = '/api/resource/Data Import';
export const LEGACY_DATA_IMPORT_API_ENDPOINT =
  '/api/resource/Data Import Legacy';
export const FRAPPE_START_DATA_IMPORT_API_ENDPOINT =
  '/api/method/frappe.core.doctype.data_import.data_import.form_start_import';
export const FRAPPE_START_LEGACY_DATA_IMPORT_API_ENDPOINT =
  '/api/method/frappe.core.doctype.data_import_legacy.data_import_legacy.import_data';
export const FRAPPE_FILE_ATTACH_API_ENDPOINT =
  '/api/method/frappe.client.attach_file';

Yes, these are the endpoints.

Better still, add your own custom button that redirects the data import functionality to your custom python function that instead enqueues it.

@frappe.whitelist()
def form_start_import_custom(docname): 
	args = { "data_import": docname }
	from frappe.core.doctype.data_import.data_import import start_import
	enqueue(method=start_import, queue='short', timeout=300, **args)
	return

Thanks for the suggestion!

I am already inserting data via the REST API by iterating over dozens of pre-existing shell scripts, so also using the API to initiate bulk imports is only a minor alteration to what I am currently doing.