Upgrade v11 to v12 results in multiple errors

Hi All

first of all, thank you for this wonderful piece of software!

I’m using the VM version of erpnext v11. I wanted to upgrade to v12. Before doing so, I migrated to python 3.5 with bench migrate-env python3.5 which worked fine. I had to install a newer version of openssl 1.1.1c (compiled from source) and then uninstall the python cryptography from the virtualenv. I reinstalled it with pip install pyOpenSSL cryptography --no-cache-dir otherwise it was still using the 1.0.1j version. After that, everything was running fine.

Now to the upgrade process. My steps were as follow:

bench switch-to-branch version-12 frappe erpnext --upgrade
bench update --patch
bench setup requirements
bench --site [your-site-name] migrate
bench restart

The first command ran fine but the problem appears at the second one. (the later 3 commands weren’t used)

console log
frappe@erpnext:~/frappe-bench$ bench update --patch
Backing up sites...
Patching sites...
Migrating erpnext.vm
Executing erpnext.patches.v12_0.set_priority_for_support in erpnext.vm (d56cb677eaab3383)
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 97, in <module>
    main()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 18, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/frappe/frappe-bench/env/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 25, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/commands/site.py", line 233, in migrate
    migrate(context.verbose, rebuild_website=rebuild_website, skip_failing=skip_failing)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/migrate.py", line 48, in migrate
    frappe.modules.patch_handler.run_all(skip_failing)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 41, in run_all
    run_patch(patch)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 30, in run_patch
    if not run_single(patchmodule = patch):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 71, in run_single
    return execute_patch(patchmodule, method, methodargs)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/modules/patch_handler.py", line 91, in execute_patch
    frappe.get_attr(patchmodule.split()[0] + ".execute")()
  File "/home/frappe/frappe-bench/apps/erpnext/erpnext/patches/v12_0/set_priority_for_support.py", line 7, in execute
    set_issue_priority()
  File "/home/frappe/frappe-bench/apps/erpnext/erpnext/patches/v12_0/set_priority_for_support.py", line 14, in set_issue_priority
    for priority in frappe.get_meta("Issue").get_field("priority").options.split("\n"):
AttributeError: 'NoneType' object has no attribute 'options'

After I was commenting out the patches in the set_priority_for_support.py so that it passed the update. I reran bench update --patch and antoher error occured:

console log
frappe@erpnext:~/frappe-bench$ bench update --patch
Backing up sites...
Patching sites...
Migrating erpnext.vm
Updating DocTypes for frappe        : [========================================]
Updating DocTypes for erpnext       : [========================================]
Updating customizations for Address
Generating Website Theme Files...
Compiling Python Files...
*** Error compiling '../apps/erpnext_demo/erpnext_demo/make_demo.py'...
  File "../apps/erpnext_demo/erpnext_demo/make_demo.py", line 42
    print "Creating Fresh Database..."
                                     ^
SyntaxError: Missing parentheses in call to 'print'

INFO:bench.utils:sudo supervisorctl restart frappe-bench-workers: frappe-bench-web:
frappe-bench-frappe-schedule: stopped
frappe-bench-frappe-default-worker-0: stopped
frappe-bench-frappe-long-worker-0: stopped
frappe-bench-frappe-schedule: ERROR (abnormal termination)
frappe-bench-frappe-default-worker-0: ERROR (abnormal termination)
frappe-bench-frappe-long-worker-0: ERROR (abnormal termination)
frappe-bench-frappe-short-worker-0: ERROR (abnormal termination)
frappe-bench-frappe-schedule: started
frappe-bench-node-socketio: ERROR (abnormal termination)
frappe-bench-frappe-web: started
________________________________________________________________________________
Bench: Deployment tool for Frappe and ERPNext (https://erpnext.org).
Open source depends on your contributions, so please contribute bug reports, patches, fixes or cash and be a part of the community
()

Now I’m lost on how to proceed further since I don’t want to break the update more than necessary.
The website currently shows the following picture:


and the network log shows most of the css and js files are not found (404), which is clear since the server is halfway down.

after login it shows the following error:
Session Start Failed 'ascii' codec can't decode byte 0xe2 in position 167281: ordinal not in range(128) which probably is also related to the missing parts of the server.

I’d be very happy if you could shed some light on how to proceed with the upgrade steps.

Best regards
Simon

Welcome to ERPNext Simon!

To access the complete traceback for clues, refer to the logs say here /home/frappe/frappe-bench/logs

An unrecognized filename may be the problem, check for example Error: 'ascii' codec can't decode byte 0xc3 in position 19: ordinal not in range(128)

Hi clarkej

Thank you for pointing me in some direction.

My findings so far:

The VM for v11 I downloaded somehow contained the erpnext_demo app which is deprecated since v7. The install script for erpnext_demo wanted to execute erpnext_demo/erpnext_demo/make_demo.py in which python3 doesn’t support print without parentheses, which indicates the error message in the second console log. 2 ways to fix it: 1. add the parentheses to the make_demoy.py or 2. remove the app from the VM with:

bench --site erpnext.vm uninstall-app erpnext_demo
bench remove-app erpnext_demo

Then when calling bench update --patch everything runs fine.

Then I had to run in ~/

sudo chown -R $USER:$GROUP ~/.npm
sudo chown -R $USER:$GROUP ~/.config

Then run (probably an issue from switching to python 3.5)

cd ~/frappe-bench
sudo npm install -g yarn
bench update --requirements
bench setup socketio
bench build
bench update

to get the css and js files back to work.

Now only the ascii-problem is left. I assume one of the interns created an app in a wrong directory.
I’ll keep you updated.

Best regards
Simon

Hi all

After running bench update (Bench Start Error in Development - Error: No such commands watch serve worker schedule, frappe commands!?) all the worker apps went back to working state.

So the only thing left is the ASCII problem.

According to the log it’s 0xd0 what’s upsetting python from working. According to the ASCII table, this is the character ð . The web.log shows

web.log
Traceback (most recent call last):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/www/desk.py", line 21, in get_context
    boot = frappe.sessions.get()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/sessions.py", line 132, in get
    bootinfo = get_bootinfo()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/boot.py", line 54, in get_bootinfo
    load_translations(bootinfo)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/boot.py", line 188, in load_translations
    messages = frappe.get_lang_dict("boot")
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 90, in get_lang_dict
    return get_dict(fortype, name)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/translate.py", line 114, in get_dict
    messages = get_messages_from_include_files()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/translate.py", line 470, in get_messages_from_include_files
    messages.extend(get_messages_from_file(os.path.join(frappe.local.sites_path, file)))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/translate.py", line 498, in get_messages_from_file
    message) for message in  extract_messages_from_code(sourcefile.read(), path.endswith(".py"))]
  File "/home/frappe/frappe-bench/env/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1801299: ordinal not in range(128)

So I did a search for that character in all *.py files since I don’t know what python files are included for translation: grep -nr --include \*.py 'ð'

grep result
env/lib/python3.5/site-packages/IPython/core/latex_symbols.py:105:    "\\eth" : "ð",
env/lib/python3.5/site-packages/IPython/core/latex_symbols.py:106:    "\\dh" : "ð",
env/lib/python3.5/site-packages/IPython/utils/tests/test_pycolorize.py:42:    print("wěird ünicoðe")
env/lib/python3.5/site-packages/stdnum/is_/vsk.py:21:"""VSK number (Virðisaukaskattsnúmer, Icelandic VAT number).
env/lib/python3.5/site-packages/bs4/dammit.py:762:        0xf0 : b'\xc3\xb0',     # ð
env/lib/python3.5/site-packages/pip/_vendor/idna/uts46data.py:226:    (0xD0, 'M', u'ð'),
env/lib/python3.5/site-packages/pip/_vendor/idna/uts46data.py:1669:    (0x1D9E, 'M', u'ð'),
env/lib/python3.5/site-packages/idna/uts46data.py:226:    (0xD0, 'M', u'ð'),
env/lib/python3.5/site-packages/idna/uts46data.py:1669:    (0x1D9E, 'M', u'ð'),
env/lib/python3.5/site-packages/docutils/utils/math/tex2unichar.py:61:    'eth': '\xf0', # ð LATIN SMALL LETTER ETH
env/lib/python3.5/site-packages/docutils/utils/math/math2html.py:332:      '\\delta':'δ', '\\dh':'ð', '\\digamma':'ϝ', '\\epsilon':'ϵ',
env/lib/python3.5/site-packages/docutils/utils/math/math2html.py:333:      '\\eta':'η', '\\eth':'ð', '\\gamma':'γ', '\\i':'ı',

I’ll further investigate which files are included in translate.py.

I’ll keep you updated.

Best regards
Simon