Bulk API usage kills ERPNext

MartinHBramwell · November 13, 2020, 12:32pm

I was using curl in a loop to insert about 5000 Sales Invoices and to attach a file to each one after insertion.

The process slowed down gradually, then threw the werkzeug error you’ll see in the trace.

The process was started with bench not supervisor. I stopped it with <ctrl>-C

The second trace shows it took that nearly a full minute for the bench to shut down.

Is this solvable with configuration optimizations, or do I need to look at choking my curl loop?

(V12 upgraded to V13. Ubuntu 20.04)

07:02:14 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:14] "POST /api/resource/Sales%20Invoice HTTP/1.0" 500 -
07:02:14 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:14] "POST /api/resource/Sales%20Invoice HTTP/1.0" 500 -
07:02:14 web.1            | Traceback (most recent call last):
07:02:14 web.1            |   File "/home/erpdev/frappe-bench-DYPW/apps/frappe/frappe/middlewares.py", line 16, in __call__
07:02:14 web.1            |     return super(StaticDataMiddleware, self).__call__(environ, start_response)
07:02:14 web.1            |   File "/home/erpdev/frappe-bench-DYPW/env/lib/python3.8/site-packages/werkzeug/middleware/shared_data.py", line 220, in __call__
07:02:14 web.1            |     return self.app(environ, start_response)
07:02:14 web.1            |   File "/home/erpdev/frappe-bench-DYPW/env/lib/python3.8/site-packages/werkzeug/middleware/shared_data.py", line 220, in __call__
07:02:14 web.1            |     return self.app(environ, start_response)
07:02:14 web.1            |   File "/home/erpdev/frappe-bench-DYPW/env/lib/python3.8/site-packages/werkzeug/local.py", line 231, in application
07:02:14 web.1            |     return ClosingIterator(app(environ, start_response), self.cleanup)
07:02:14 web.1            |   File "/home/erpdev/frappe-bench-DYPW/env/lib/python3.8/site-packages/werkzeug/wrappers/base_request.py", line 240, in application
07:02:14 web.1            |     return resp(*args[-2:])
07:02:14 web.1            | TypeError: 'NoneType' object is not callable
07:02:37 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:37] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:37 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:37] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:39 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:39] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -
07:02:39 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:39] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -
07:02:41 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:41] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:41 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:41] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:43 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:43] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -
07:02:43 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:43] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -
07:02:44 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:44] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:44 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:44] "POST /api/method/frappe.client.attach_file HTTP/1.0" 404 -
07:02:48 web.1            | 127.0.0.1 - - [13/Nov/2020 07:02:48] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -

07:02:48 web.1            | INFO:werkzeug:127.0.0.1 - - [13/Nov/2020 07:02:48] "POST /api/resource/Sales%20Invoice HTTP/1.0" 200 -
^C07:02:53 system           | SIGINT received
07:02:53 system           | sending SIGTERM to redis_cache.1 (pid 127894)
07:02:53 system           | sending SIGTERM to redis_socketio.1 (pid 127895)
07:02:53 system           | sending SIGTERM to redis_queue.1 (pid 127893)
07:02:53 system           | sending SIGTERM to web.1 (pid 127899)
07:02:53 system           | sending SIGTERM to socketio.1 (pid 127896)
07:02:53 system           | sending SIGTERM to watch.1 (pid 127909)
07:02:53 system           | sending SIGTERM to schedule.1 (pid 127913)
07:02:53 system           | sending SIGTERM to worker_short.1 (pid 127925)
07:02:53 system           | sending SIGTERM to worker_long.1 (pid 127928)
07:02:53 system           | sending SIGTERM to worker_default.1 (pid 127938)
07:03:15 system           | watch.1 stopped (rc=-15)
07:03:21 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:23 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:23 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:23 system           | sending SIGKILL to web.1 (pid 127899)
07:03:23 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:29 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:29 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:29 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:29 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:29 system           | web.1 stopped (rc=-15)
07:03:29 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:29 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:29 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:29 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:29 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:29 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:29 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:30 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:30 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:30 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:30 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:30 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:31 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:31 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:31 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:31 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:31 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:31 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:31 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:32 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:32 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:32 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:32 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:32 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:32 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:32 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:32 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:33 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:33 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:33 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:33 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:33 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:33 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:33 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:33 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:34 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:34 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:34 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:34 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:34 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:34 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:34 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:34 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:35 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:35 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:35 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:35 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:35 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:35 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:35 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:35 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:35 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:35 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:35 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:35 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:35 system           | sending SIGKILL to schedule.1 (pid 127913)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:36 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:36 system           | schedule.1 stopped (rc=-15)
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:36 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:36 system           | sending SIGKILL to worker_default.1 (pid 127938)
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:36 system           | sending SIGKILL to socketio.1 (pid 127896)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:03 redis_socketio.1 | 127900:signal-handler (1605268973) Received SIGTERM scheduling shutdown...
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | sending SIGKILL to worker_long.1 (pid 127928)
07:03:36 system           | worker_long.1 stopped (rc=-15)
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:36 system           | sending SIGKILL to worker_short.1 (pid 127925)
07:03:36 system           | worker_short.1 stopped (rc=-15)
07:03:36 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:36 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:36 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:07 redis_queue.1    | 127905:signal-handler (1605268973) Received SIGTERM scheduling shutdown...
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_queue.1    | 127905:M 13 Nov 2020 07:02:54.219 # User requested shutdown...
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:04 redis_cache.1    | 127903:signal-handler (1605268973) Received SIGTERM scheduling shutdown...
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 redis_cache.1    | 127903:M 13 Nov 2020 07:02:54.218 # User requested shutdown...
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 redis_cache.1    | 127903:M 13 Nov 2020 07:02:54.339 * Removing the pid file.
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:31 redis_cache.1    | 127903:M 13 Nov 2020 07:02:54.420 # Redis is now ready to exit, bye bye...
07:03:37 system           | sending SIGKILL to redis_cache.1 (pid 127894)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:37 system           | redis_cache.1 stopped (rc=-15)
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_socketio.1 | 127900:M 13 Nov 2020 07:02:54.072 # User requested shutdown...
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_socketio.1 | 127900:M 13 Nov 2020 07:02:54.450 * Removing the pid file.
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_socketio.1 | 127900:M 13 Nov 2020 07:02:54.450 # Redis is now ready to exit, bye bye...
07:03:37 system           | sending SIGKILL to redis_socketio.1 (pid 127895)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:37 system           | redis_socketio.1 stopped (rc=-15)
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_queue.1    | 127905:M 13 Nov 2020 07:02:54.463 * Removing the pid file.
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:29 redis_queue.1    | 127905:M 13 Nov 2020 07:02:54.463 # Redis is now ready to exit, bye bye...
07:03:37 system           | sending SIGKILL to redis_queue.1 (pid 127893)
07:03:37 system           | redis_queue.1 stopped (rc=-15)

revant_one · November 14, 2020, 12:06am

Use data import via rest api.

Wait for one data import to complete before starting another batch.

5000 individual requests from app may also slow down the system for users.

MartinHBramwell · November 14, 2020, 12:19am

Would you mind elaborating on that? I’m not sure what you mean.

revant_one · November 14, 2020, 5:12am

All steps are done programmatically:

Prepare csv template, create .csv file
Create data import, attach the previous .csv
Start import using whitelisted endpoint.
Poll for completion/failure of data import.
In case of success, update data
In case of failure, show error to user for intervention

MartinHBramwell · November 14, 2020, 11:04am

Sorry, I didn’t mean for you to go into so much detail. I understand the process and I understand csv files. I was unaware of a whitelisted end point.

So this is what I need?

export const DATA_IMPORT_API_ENDPOINT = '/api/resource/Data Import';
export const LEGACY_DATA_IMPORT_API_ENDPOINT =
  '/api/resource/Data Import Legacy';
export const FRAPPE_START_DATA_IMPORT_API_ENDPOINT =
  '/api/method/frappe.core.doctype.data_import.data_import.form_start_import';
export const FRAPPE_START_LEGACY_DATA_IMPORT_API_ENDPOINT =
  '/api/method/frappe.core.doctype.data_import_legacy.data_import_legacy.import_data';
export const FRAPPE_FILE_ATTACH_API_ENDPOINT =
  '/api/method/frappe.client.attach_file';

revant_one · November 14, 2020, 12:48pm

Yes, these are the endpoints.

saleemdev · November 23, 2020, 1:05pm

Better still, add your own custom button that redirects the data import functionality to your custom python function that instead enqueues it.

saleemdev · November 23, 2020, 1:06pm

@frappe.whitelist()
def form_start_import_custom(docname): 
	args = { "data_import": docname }
	from frappe.core.doctype.data_import.data_import import start_import
	enqueue(method=start_import, queue='short', timeout=300, **args)
	return

MartinHBramwell · November 23, 2020, 6:29pm

Thanks for the suggestion!

I am already inserting data via the REST API by iterating over dozens of pre-existing shell scripts, so also using the API to initiate bulk imports is only a minor alteration to what I am currently doing.

Sneha_Manjunath · October 17, 2023, 11:02am

Sir do you have some code snippet of how to use it, I wanted the end points and also the data format of the input data, My data is in csv file I don’t want to iterate and call the post method in every row,
The data formates I’m trying is not working and also im not getting correct end points

Because of which im unable to detect whats wrong with the data im passing, let me say i’m unaware of how the bulk data looks like

Ill be greatful to recieve some help!!

Regards,
Sneha

revant_one · October 17, 2023, 11:36am

Try it manually first. Here is the documentation https://docs.erpnext.com/docs/user/manual/en/data-import

when you do it, watch network tab of browser to check the api requests that are made.

You will need CSV that is supported by the tool. Check documentation.

Sneha_Manjunath · October 17, 2023, 11:38am

OK sir ,Thank you sir

Sneha_Manjunath · October 17, 2023, 12:09pm

Sir but what operation should i perform on my csv file, so that the data is accepted at the end point

df = pd.read_csv("Attendance.csv")
    result = df.to_json(orient="split")
    parsed = loads(result)
    dumps(parsed, indent=4)
    data_json = json.dumps(parsed)

Im performing this operation to get the data like this

{
    "columns": ["naming_series", "employee", "status", "attendance_date", "company", "working_hours", "shift", "in_time", "out_time", "late_entry", "early_exit"],
     "index": [0, 1, 2],
      "data": [
          ["HR-ATT-.YYYY.-", "HR-EMP-00001", "Present", "2023-09-07", "Prerana", 5.0, 2, "2023-09-08 09:34:23", "2023-09-08 15:34:23", 0, 0],
           ["HR-ATT-.YYYY.-", "HR-EMP-00001", "Absent", "2023-09-08", "Prerana", 6.0, 4, "2023-09-09 09:34:23", "2023-09-08 17:30:21", 0, 1],
           ["HR-ATT-.YYYY.-", "HR-EMP-00001", "Present", "2023-09-09", "Prerana", 4.6, 2, "2023-09-10 09:34:23", "2023-09-08 19:00:10", 1, 0]
           ]
    }

is this the way to append in bulk or should the approch be changed