Try ERPNext Try Frappe Cloud Buy Support Partners Foundation

Update fails - TypeError: can't pickle module objects

Hey guys, anyone has any idea where this is coming from ? in a previous post the idea was this is related to redis but it seems redis is running correctly on my server.

Trying to update the system on the latest version-12 developing branch on an up to date ubuntu 18

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/utils/bench_helper.py", line 97, in <module>
    main()
  File "/opt/erpnext/frappe12/apps/frappe/frappe/utils/bench_helper.py", line 18, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/opt/erpnext/frappe12/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/erpnext/frappe12/env/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/erpnext/frappe12/env/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/erpnext/frappe12/env/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/erpnext/frappe12/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/erpnext/frappe12/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/erpnext/frappe12/env/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/commands/__init__.py", line 25, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/commands/site.py", line 233, in migrate
    migrate(context.verbose, rebuild_website=rebuild_website, skip_failing=skip_failing)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/migrate.py", line 48, in migrate
    frappe.modules.patch_handler.run_all(skip_failing)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/modules/patch_handler.py", line 41, in run_all
    run_patch(patch)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/modules/patch_handler.py", line 30, in run_patch
    if not run_single(patchmodule = patch):
  File "/opt/erpnext/frappe12/apps/frappe/frappe/modules/patch_handler.py", line 71, in run_single
    return execute_patch(patchmodule, method, methodargs)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/modules/patch_handler.py", line 77, in execute_patch
    block_user(True)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/modules/patch_handler.py", line 127, in block_user
    frappe.db.set_global('__session_status', block and 'stop' or None)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/database/database.py", line 698, in set_global
    self.set_default(key, val, user)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/database/database.py", line 712, in set_default
    frappe.defaults.set_default(key, val, parent, parenttype)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/defaults.py", line 134, in set_default
    add_default(key, value, parent)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/defaults.py", line 147, in add_default
    d.insert(ignore_permissions=True)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/model/document.py", line 222, in insert
    self.run_method("before_insert")
  File "/opt/erpnext/frappe12/apps/frappe/frappe/model/document.py", line 787, in run_method
    out = Document.hook(fn)(self, *args, **kwargs)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/model/document.py", line 1052, in composer
    doc_events = frappe.get_doc_hooks()
  File "/opt/erpnext/frappe12/apps/frappe/frappe/__init__.py", line 899, in get_doc_hooks
    hooks = get_hooks('doc_events', {})
  File "/opt/erpnext/frappe12/apps/frappe/frappe/__init__.py", line 946, in get_hooks
    hooks = _dict(cache().get_value("app_hooks", load_app_hooks))
  File "/opt/erpnext/frappe12/apps/frappe/frappe/utils/redis_wrapper.py", line 80, in get_value
    self.set_value(original_key, val, user=user)
  File "/opt/erpnext/frappe12/apps/frappe/frappe/utils/redis_wrapper.py", line 48, in set_value
    self.set(key, pickle.dumps(val))
TypeError: can't pickle module objects

Did you get any solution on this?

Any solution for this? i am also facing problem with this error

if custom code in custom app’s hooks.py includes imported module, there is such error, change the system conf to activate develop mode, then system will skip pickling hooks.py.

1 Like

Yes, this is the issue that I experienced.

In developer mode, the hooks are not cached which is why it works fine normally.

With developer mode off, hooks are cached. So when you import a module it’s trying to cache the dict() containing a module key which isn’t possible and that causes the pickle error.

Another workaround besides just turning on developer mode, is you can rewrite your imports to be like this: “import xyz as _xyz”

This works because the caching function skips any keys that begin with underscores.

3 Likes

seems more cute workaround.

Update: confirm it is working
:+1:

Hi
Can you please guide where to make this change? Which file and what is its location

@Manan_Shah i think in hooks.py in your custom app

Thanks for the reply
I see these two lines

from __future__ import unicode_literals

from . import __version__ as app_version

Do I need to change anything in these two lines?

What is the

referring to?

Hi
We need to change this line

to

from . import xyz as _xyz and run bench update, this solved the problem

I am getting this same error when trying to run bench migrate.
developer_mode is set as 1 in the site_config.json, yet this issue occurs.
Any help?

The only way I can get it to work is by changing the following in frappe/frappe/__init__.py,

no_cache = conf.developer_mode or False

if app_name:
	hooks = _dict(load_app_hooks(app_name))
else:
	if no_cache:
		hooks = _dict(load_app_hooks())
	else:
		hooks = _dict(cache().get_value("app_hooks", load_app_hooks))

It works if I explicitly set if True instead of if no_cache
Though it should work as no_cache is being set from conf.developer_mode, shouldn’t it? :thinking:

4 Likes

I removed custom app in apps.txt in sites but still get this error, do you have solution for this?

  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 1034, in get_hooks
    hooks = _dict(cache().get_value("app_hooks", load_app_hooks))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/redis_wrapper.py", line 80, in get_value
    self.set_value(original_key, val, user=user)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/redis_wrapper.py", line 48, in set_value
    self.set(key, pickle.dumps(val))
TypeError: can't pickle module objects

Thank you, @benknowles and @szufisher

:bowing_man:

I am getting this error despite developer mode. Any thoughts on that? Thank you.

Edit: Thanks @shubhamgupta123, editing __init__.py did the trick!

Edit 2: Nope, does not work. As soon as I edit __init__.py, I have local changes in frappe app and cannot proceed with the update. Reverting the changes removes the necessary change in the file. git pull doesn’t work either - it reports to be current. So I am running in circles, unable to update.

@rmehta @revant_one

Please go through this thread.

@shubhamgupta123 This will interest you


Whenever patches are run, it comes to frappe.get_hooks method n times

n = 1 (for website_clear_cache hook) + num_of_apps_installed (for before_migrate hook) + 1 (for doc_events hook)

When it comes for the hook doc_events, for some reason conf.developer_mode is set to 0 which causes no_cache to be False

app_name hook default conf.developer_mode no_cache
None website_clear_cache None True True
frappe before_migrate None True True
custom_app before_migrate None True True
None doc_events None 0 False

It is the last time it comes to this method, it fails resulting in the pickle error.

@rmehta @revant_one please check if you could do something about this issue!

1 Like

try catch then bypass cache and the error as needed?