[v12] Possible Bug in Patch for Migrating from v11 to v12

Hi,
I am trying to migrate one test db from v11 to v12 and there seems to be an error for patch as below:

Executing frappe.patches.v12_0.setup_comments_from_communications in rigplv12 (_db_name_)
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/aditya/v12/apps/frappe/frappe/utils/bench_helper.py", line 97, in <module>
    main()
  File "/home/aditya/v12/apps/frappe/frappe/utils/bench_helper.py", line 18, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/aditya/v12/apps/frappe/frappe/commands/__init__.py", line 25, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/aditya/v12/apps/frappe/frappe/commands/site.py", line 234, in migrate
    migrate(context.verbose, rebuild_website=rebuild_website, skip_failing=skip_failing)
  File "/home/aditya/v12/apps/frappe/frappe/migrate.py", line 48, in migrate
    frappe.modules.patch_handler.run_all(skip_failing)
  File "/home/aditya/v12/apps/frappe/frappe/modules/patch_handler.py", line 41, in run_all
    run_patch(patch)
  File "/home/aditya/v12/apps/frappe/frappe/modules/patch_handler.py", line 30, in run_patch
    if not run_single(patchmodule = patch):
  File "/home/aditya/v12/apps/frappe/frappe/modules/patch_handler.py", line 71, in run_single
    return execute_patch(patchmodule, method, methodargs)
  File "/home/aditya/v12/apps/frappe/frappe/modules/patch_handler.py", line 91, in execute_patch
    frappe.get_attr(patchmodule.split()[0] + ".execute")()
  File "/home/aditya/v12/apps/frappe/frappe/patches/v12_0/setup_comments_from_communications.py", line 25, in execute
    new_comment.db_insert()
  File "/home/aditya/v12/apps/frappe/frappe/model/base_document.py", line 323, in db_insert
    ), list(d.values()))
  File "/home/aditya/v12/apps/frappe/frappe/database/database.py", line 125, in sql
    self.check_transaction_status(query)
  File "/home/aditya/v12/apps/frappe/frappe/database/database.py", line 260, in check_transaction_status
    frappe.throw(_("Too many writes in one request. Please send smaller requests"), frappe.ValidationError)
  File "/home/aditya/v12/apps/frappe/frappe/__init__.py", line 360, in throw
    msgprint(msg, raise_exception=exc, title=title, indicator='red')
  File "/home/aditya/v12/apps/frappe/frappe/__init__.py", line 346, in msgprint
    _raise_exception()
  File "/home/aditya/v12/apps/frappe/frappe/__init__.py", line 315, in _raise_exception
    raise raise_exception(msg)
frappe.exceptions.ValidationError: Too many writes in one request. Please send smaller requests

I have tried to resolve this issue by a simple hack in the patch by adding a simple loop to commit changes to db after every 10k entries as below:

reference file: frappe/setup_comments_from_communications.py at version-12 · frappe/frappe · GitHub

from __future__ import unicode_literals

import frappe

def execute():
    frappe.reload_doctype("Comment")
    count = 0
    for comment in frappe.get_all('Communication', fields = ['*'],
        filters = dict(communication_type = 'Comment')):

        new_comment = frappe.new_doc('Comment')
        new_comment.comment_type = comment.comment_type
        new_comment.comment_email = comment.sender
        new_comment.comment_by = comment.sender_full_name
        new_comment.subject = comment.subject
        new_comment.content = comment.content or comment.subject
        new_comment.reference_doctype = comment.reference_doctype
        new_comment.reference_name = comment.reference_name
        new_comment.link_doctype = comment.link_doctype
        new_comment.link_name = comment.link_name
        new_comment.creation = comment.creation
        new_comment.modified = comment.modified
        new_comment.owner = comment.owner
        new_comment.modified_by = comment.modified_by
        new_comment.db_insert()
        count += 1
        if count%10000 == 0:
            frappe.db.commit()
            print("Committed " + str(count))
    # clean up
    frappe.db.sql("delete from `tabCommunication` where communication_type = 'Comment'")

Simple hack above did tell me that the patch was trying to commit more than 800k comments in one go and due to which I guess the above error was coming.

1 Like

Well I think I have found another bug and I think this would be helpful only if you are migrating very OLD data from v11 to v12.

I have many taxes and many of them are now dead (changed) so I don’t know which accounts is this validation error is talking about so that I can them to either tax or chargeable or income or expense and hence I am unable to debug this error, the only option is to skip this patch for the time being, any possible solution for this patch?

Executing erpnext.patches.v12_0.move_item_tax_to_item_tax_template in rigplv12 (_3bd611f3e1fe10b1)
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/aditya/v12/apps/frappe/frappe/utils/bench_helper.py", line 97, in <module>
    main()
  File "/home/aditya/v12/apps/frappe/frappe/utils/bench_helper.py", line 18, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/aditya/v12/env/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/aditya/v12/apps/frappe/frappe/commands/__init__.py", line 25, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/aditya/v12/apps/frappe/frappe/commands/site.py", line 234, in migrate
    migrate(context.verbose, rebuild_website=rebuild_website, skip_failing=skip_failing)
  File "/home/aditya/v12/apps/frappe/frappe/migrate.py", line 48, in migrate
    frappe.modules.patch_handler.run_all(skip_failing)
  File "/home/aditya/v12/apps/frappe/frappe/modules/patch_handler.py", line 41, in run_all
    run_patch(patch)
  File "/home/aditya/v12/apps/frappe/frappe/modules/patch_handler.py", line 30, in run_patch
    if not run_single(patchmodule = patch):
  File "/home/aditya/v12/apps/frappe/frappe/modules/patch_handler.py", line 71, in run_single
    return execute_patch(patchmodule, method, methodargs)
  File "/home/aditya/v12/apps/frappe/frappe/modules/patch_handler.py", line 91, in execute_patch
    frappe.get_attr(patchmodule.split()[0] + ".execute")()
  File "/home/aditya/v12/apps/erpnext/erpnext/patches/v12_0/move_item_tax_to_item_tax_template.py", line 55, in execute
    item_tax_map, d.item_code, d.parent)
  File "/home/aditya/v12/apps/erpnext/erpnext/patches/v12_0/move_item_tax_to_item_tax_template.py", line 98, in get_item_tax_template
    item_tax_template.save()
  File "/home/aditya/v12/apps/frappe/frappe/model/document.py", line 271, in save
    return self._save(*args, **kwargs)
  File "/home/aditya/v12/apps/frappe/frappe/model/document.py", line 294, in _save
    self.insert()
  File "/home/aditya/v12/apps/frappe/frappe/model/document.py", line 228, in insert
    self.run_before_save_methods()
  File "/home/aditya/v12/apps/frappe/frappe/model/document.py", line 886, in run_before_save_methods
    self.run_method("validate")
  File "/home/aditya/v12/apps/frappe/frappe/model/document.py", line 786, in run_method
    out = Document.hook(fn)(self, *args, **kwargs)
  File "/home/aditya/v12/apps/frappe/frappe/model/document.py", line 1055, in composer
    return composed(self, method, *args, **kwargs)
  File "/home/aditya/v12/apps/frappe/frappe/model/document.py", line 1038, in runner
    add_to_return_value(self, fn(self, *args, **kwargs))
  File "/home/aditya/v12/apps/frappe/frappe/model/document.py", line 780, in <lambda>
    fn = lambda self, *args, **kwargs: getattr(self, method)(*args, **kwargs)
  File "/home/aditya/v12/apps/erpnext/erpnext/accounts/doctype/item_tax_template/item_tax_template.py", line 12, in validate
    self.validate_tax_accounts()
  File "/home/aditya/v12/apps/erpnext/erpnext/accounts/doctype/item_tax_template/item_tax_template.py", line 24, in validate_tax_accounts
    d.idx))
  File "/home/aditya/v12/apps/frappe/frappe/__init__.py", line 360, in throw
    msgprint(msg, raise_exception=exc, title=title, indicator='red')
  File "/home/aditya/v12/apps/frappe/frappe/__init__.py", line 346, in msgprint
    _raise_exception()
  File "/home/aditya/v12/apps/frappe/frappe/__init__.py", line 315, in _raise_exception
    raise raise_exception(msg)
frappe.exceptions.ValidationError: Item Tax Row 1 must have account of type Tax or Income or Expense or Chargeable

@adityaduggal, did you solve this issue? on same patch erpnext.patches.v12_0.move_item_tax_to_item_tax_template my migration from V11>>V12 just runs forever till ubuntu kills the job.

Maurizio

1 Like

Here’s another one :

@root13F, I will start updating python from 2.7 to 3.6

@jokermao I am still facing this issue and I found out that this issue is not resolved till date. I did ping @nabinhait for this but I did not get a response.

Now I checked and then I have created a ISSUE on github for the same. HERE IS THE LINK

But still I am unable to resolve the issue.