Update taking a long time

JayRam · February 14, 2020, 8:35am

Hi,

A customer site that uses ERPNext extensively and intensively is having the following issue:

I ran bench update and I see that it’s stuck at this for many hours now:

Updating node packages…
INFO:bench.utils:yarn install
yarn install v1.5.1
[1/4] Resolving packages…
success Already up-to-date.
Done in 0.70s.
Patching sites…
Migrating site1.local
Updating DocTypes for frappe : [========================================]
Updating DocTypes for erpnext : [========================================]
Updating DocTypes for nhance : [========================================]
Migrating sspl
Executing frappe.patches.v11_0.make_all_prepared_report_attachments_private #2019-11-26 in abcd (a81126be1d0839c7)

I aborted the update, went in to patches.txt in the frappe folder and tried to comment this out, and then did a git pull from the frappe and erpnext folders and tried bench --site abcd migrate, but it wouldn’t work.

I am certain that this client has a bunch of prepared reports and I really haven’t done any housekeeping by purging the prepared reports, so is that the reason its taking this kind of time?

Any suggestions as to how to get the instance working again and I can run the specific update over the weekend?

Thanks

Jay

JayRam · February 14, 2020, 11:03am

We had 200,000 prepared reports. I deleted all prepared reports beyond 5 days and am running the update again. It still seems to be taking a long time. Since this patch is to make prepared report attachments private, will this help? As the attachments could be sitting in the Public folder, even if the prepared report does not exist anymore. Is this update making the prepared report attachments only for those that prepared reports that are not deleted? Or is the code written in such a way that it picks up every attachment and then when it discovers that the report does not exist anymore, drops that attachment and moves on? If it’s the latter, just deleting the Prepared Report may not help this patch run faster.

Anybody knows enough about this?

Thanks

Jay

clarkej · February 14, 2020, 3:36pm

Hi there Jay!

The patch seems to do a cross-reference check between Files and Prepared Report (that each points to the other), IOW 2x scans of the File table, so you probably want to let it do it’s general consistency cleanup Jay?

github.com

frappe/frappe/blob/develop/frappe/patches/v11_0/make_all_prepared_report_attachments_private.py

import frappe


def execute():
	if (
		frappe.db.count("File", filters={"attached_to_doctype": "Prepared Report", "is_private": 0})
		> 10000
	):
		frappe.db.auto_commit_on_many_writes = True

	files = frappe.get_all(
		"File",
		fields=["name", "attached_to_name"],
		filters={"attached_to_doctype": "Prepared Report", "is_private": 0},
	)
	for file_dict in files:
		# For some reason Prepared Report doc might not exist, check if it exists first
		if frappe.db.exists("Prepared Report", file_dict.attached_to_name):
			try:
				file_doc = frappe.get_doc("File", file_dict.name)

This file has been truncated. show original

Perhaps to increase the memory might help if you are running in a virtual server?