Backup and Restore, then Bench update error again

Dear team,

I am moving existing ERPNext database to new server. Here is what i did follow the instruction here

On fresh new system which is already run #bench update successfully, i execute following command:
frappe@sky:~/frappe-bench$ bench --site tss.marketsimple.com --force restore /tmp/20170215_011211_d6023943_database.sql

But getting following error,
MySQL root password: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 79, in <module> main() File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 16, in main click.Group(commands=commands)(prog_name='bench') File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 24, in _func ret = f(frappe._dict(ctx.obj), *args, **kwargs) File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/site.py", line 104, in restore force=context.force) File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/site.py", line 63, in _new_site frappe.utils.scheduler.toggle_scheduler(enable_scheduler) File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/scheduler.py", line 217, in toggle_scheduler ss = frappe.get_doc("System Settings") File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 603, in get_doc return frappe.model.document.get_doc(arg1, arg2) File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 51, in get_doc return controller(arg1, arg2) File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 84, in __init__ self.load_from_db() File "/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py", line 101, in load_from_db if not getattr(self, "_metaclass", False) and self.meta.issingle: File "/home/frappe/frappe-bench/apps/frappe/frappe/model/base_document.py", line 61, in meta self._meta = frappe.get_meta(self.doctype) File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 620, in get_meta return frappe.model.meta.get_meta(doctype, cached=cached) File "/home/frappe/frappe-bench/apps/frappe/frappe/model/meta.py", line 30, in get_meta return frappe.cache().hget("meta", doctype, lambda: Meta(doctype)) File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/redis_wrapper.py", line 171, in hget value = generator() File "/home/frappe/frappe-bench/apps/frappe/frappe/model/meta.py", line 30, in <lambda> return frappe.cache().hget("meta", doctype, lambda: Meta(doctype)) File "/home/frappe/frappe-bench/apps/frappe/frappe/model/meta.py", line 66, in __init__ self.process() File "/home/frappe/frappe-bench/apps/frappe/frappe/model/meta.py", line 226, in process self.set_custom_permissions() File "/home/frappe/frappe-bench/apps/frappe/frappe/model/meta.py", line 313, in set_custom_permissions filters=dict(parent=self.name), update=dict(doctype='Custom DocPerm')) File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 1136, in get_all return get_list(doctype, *args, **kwargs) File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 1109, in get_list return frappe.model.db_query.DatabaseQuery(doctype).execute(None, *args, **kwargs) File "/home/frappe/frappe-bench/apps/frappe/frappe/model/db_query.py", line 83, in execute result = self.build_and_run() File "/home/frappe/frappe-bench/apps/frappe/frappe/model/db_query.py", line 95, in build_and_run args = self.prepare_args() File "/home/frappe/frappe-bench/apps/frappe/frappe/model/db_query.py", line 112, in prepare_args self.set_optional_columns() File "/home/frappe/frappe-bench/apps/frappe/frappe/model/db_query.py", line 212, in set_optional_columns columns = frappe.db.get_table_columns(self.doctype) File "/home/frappe/frappe-bench/apps/frappe/frappe/database.py", line 808, in get_table_columns return self.get_db_table_columns('tab' + doctype) File "/home/frappe/frappe-bench/apps/frappe/frappe/database.py", line 804, in get_db_table_columns return [r[0] for r in self.sql("DESC%s" % table)] File "/home/frappe/frappe-bench/apps/frappe/frappe/database.py", line 149, in sql self._cursor.execute(query) File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute self.errorhandler(self, exc, value) File "/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue _mysql_exceptions.ProgrammingError: (1146, "Table '4990a72a2f199a0f.tabCustom DocPerm' doesn't exist")

I thought the old system needs also #bench update in order to get full database structure, so i executed #bench update on it, but got following error,

`frappe@pink:~/frappe-bench$ bench update
INFO:bench.utils:updating bench
Already up-to-date.
remote: Counting objects: 92, done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 92 (delta 77), reused 91 (delta 77), pack-reused 0
Unpacking objects: 100% (92/92), done.
From https://github.com/frappe/frappe
867107b…8cfa582 develop -> upstream/develop
Requirement already satisfied: Pillow in ./env/lib/python2.7/site-packages
INFO:bench.app:pulling frappe
From https://github.com/frappe/frappe

  • branch develop -> FETCH_HEAD
    Updating 867107b…8cfa582
    Fast-forward
    frappe/boot.py | 4 ±
    frappe/core/doctype/communication/communication.js | 83 ++++++++++++++++++++++++++++++++
    frappe/core/doctype/communication/communication.py | 1 +
    frappe/core/doctype/communication/communication_list.js | 20 ++++++±
    frappe/email/inbox.py | 79 +++++++++++++++++++++++++++++++
    frappe/email/page/email_inbox/email_inbox.json | 23 ---------
    frappe/email/page/email_inbox/inbox_email.html | 60 -----------------------
    frappe/email/page/email_inbox/inbox_email_actions.html | 11 -----
    frappe/email/page/email_inbox/inbox_filter.html | 8 ----
    frappe/email/page/email_inbox/inbox_footer.html | 9 ----
    frappe/email/page/email_inbox/inbox_headers.html | 27 -----------
    frappe/public/build.json | 4 ++
    frappe/public/js/frappe/list/list_sidebar.html | 8 +±-
    frappe/public/js/frappe/list/list_sidebar.js | 36 +++++++++++±-
    frappe/public/js/frappe/list/list_view.js | 12 +++±
    frappe/public/js/frappe/ui/base_list.js | 4 ±
    frappe/public/js/frappe/views/communication.js | 10 ++±
    frappe/public/js/frappe/views/inbox/inbox_view.js | 100 +++++++++++++++++++++++++++++++++++++++
    frappe/public/js/frappe/views/inbox/inbox_view_item_main_head.html | 24 ++++++++++
    frappe/public/js/frappe/views/inbox/inbox_view_item_row.html | 34 +++++++++++++
    20 files changed, 401 insertions(+), 156 deletions(-)
    create mode 100644 frappe/email/inbox.py
    delete mode 100644 frappe/email/page/email_inbox/email_inbox.json
    delete mode 100644 frappe/email/page/email_inbox/inbox_email.html
    delete mode 100644 frappe/email/page/email_inbox/inbox_email_actions.html
    delete mode 100644 frappe/email/page/email_inbox/inbox_filter.html
    delete mode 100644 frappe/email/page/email_inbox/inbox_footer.html
    delete mode 100644 frappe/email/page/email_inbox/inbox_headers.html
    create mode 100644 frappe/public/js/frappe/views/inbox/inbox_view.js
    create mode 100644 frappe/public/js/frappe/views/inbox/inbox_view_item_main_head.html
    create mode 100644 frappe/public/js/frappe/views/inbox/inbox_view_item_row.html
    INFO:bench.app:pulling erpnext
    From https://github.com/frappe/erpnext
  • branch develop -> FETCH_HEAD
    Already up-to-date.
    Requirement already up-to-date: pip in ./env/lib/python2.7/site-packages
    Migrating trading.marketsimple.com
    Executing frappe.patches.v7_2.update_communications in trading.marketsimple.com (c7cd8ec800acdd3d)
    Traceback (most recent call last):
    File “/usr/lib/python2.7/runpy.py”, line 174, in _run_module_as_main
    main”, fname, loader, pkg_name)
    File “/usr/lib/python2.7/runpy.py”, line 72, in _run_code
    exec code in run_globals
    File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 79, in
    main()
    File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 16, in main
    click.Group(commands=commands)(prog_name=‘bench’)
    File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 716, in call
    return self.main(*args, **kwargs)
    File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 696, in main
    rv = self.invoke(ctx)
    File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
    File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
    File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
    File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 534, in invoke
    return callback(*args, **kwargs)
    File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
    File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/init.py”, line 24, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
    File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/site.py”, line 210, in migrate
    migrate(context.verbose, rebuild_website=rebuild_website)
    File “/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py”, line 31, in migrate
    frappe.modules.patch_handler.run_all()
    File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py”, line 29, in run_all
    if not run_single(patchmodule = patch):
    File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py”, line 63, in run_single
    return execute_patch(patchmodule, method, methodargs)
    File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py”, line 83, in execute_patch
    frappe.get_attr(patchmodule.split()[0] + “.execute”)()
    File “/home/frappe/frappe-bench/apps/frappe/frappe/patches/v7_2/update_communications.py”, line 11, in execute
    frappe.reload_doc(“core”, “doctype”, “communication”)
    File “/home/frappe/frappe-bench/apps/frappe/frappe/init.py”, line 660, in reload_doc
    return frappe.modules.reload_doc(module, dt, dn, force=force, reset_permissions=reset_permissions)
    File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/utils.py”, line 152, in reload_doc
    return import_files(module, dt, dn, force=force, reset_permissions=reset_permissions)
    File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/import_file.py”, line 19, in import_files
    reset_permissions=reset_permissions)
    File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/import_file.py”, line 24, in import_file
    ret = import_file_by_path(path, force, pre_process=pre_process, reset_permissions=reset_permissions)
    File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/import_file.py”, line 58, in import_file_by_path
    ignore_version=ignore_version, reset_permissions=reset_permissions)
    File “/home/frappe/frappe-bench/apps/frappe/frappe/modules/import_file.py”, line 129, in import_doc
    doc.insert()
    File “/home/frappe/frappe-bench/apps/frappe/frappe/model/document.py”, line 205, in insert
    self.db_insert()
    File “/home/frappe/frappe-bench/apps/frappe/frappe/model/base_document.py”, line 296, in db_insert
    ), d.values())
    File “/home/frappe/frappe-bench/apps/frappe/frappe/database.py”, line 138, in sql
    self._cursor.execute(query, values)
    File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/MySQLdb/cursors.py”, line 205, in execute
    self.errorhandler(self, exc, value)
    File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/MySQLdb/connections.py”, line 36, in defaulterrorhandler
    raise errorclass, errorvalue
    _mysql_exceptions.OperationalError: (1048, “Column ‘in_dialog’ cannot be null”)`

Some conventions:
New system: sky
Old system: pink

Could you please advise me how to fix this issue so that i could migrate ERPNext database to new system properly?

Thank you very much.

You cannot restore a version 7 backup in version 8 instance.

Instead what you can try following steps:

  1. Switch to master branch for both frappe and erpnext repo.
  2. Restore your backup
  3. Run bench migrate
  4. Switch to develop branch and pull from github
  5. Run bench update

Please let me know if it works.

1 Like

@thanks @nabinhait

For step 1, how could i switch to master branch?
same question for step 4.

Thanks again.

Ignore my previous reply.

I have pushed a fix, now you should be able to run bench update on existing bench. And after updating bench, try to restore again.

2 Likes

perfect!
All works smoothly.

Thanks a lot @nabinhait