Why there are no option "run-tests" for test execution? (Bench version 4.0.1)

I can’t run tests. I just have installed bench. Should I setup some additional module for testing?

katerina@katerina-VirtualBox:~/ERPNext$ bench
Usage: bench [OPTIONS] COMMAND [ARGS]…

Bench manager for Frappe

Options:
–version
–help Show this message and exit.

Commands:
backup backup site
backup-all-sites backup all sites
config change bench configuration
disable-production Disables production environment for the…
download-translations Download latest translations
get-app clone an app from the internet and set it up…
init Create a new bench
new-app start a new app
new-site Create a new site in the bench
release Release app (internal to the Frappe team)
remove-app completely remove app from bench
renew-lets-encrypt Renew Let’s Encrypt certificate
restart Restart supervisor processes
retry-upgrade
set-default-site Set default site for bench
set-mariadb-host Set MariaDB host for bench
set-nginx-port Set nginx port for site
set-ssl-certificate Set ssl certificate path for site
set-ssl-key Set ssl certificate private key path for site
set-url-root Set url root for site
setup Setup bench
shell
src Prints bench source folder path, which can be…
start Start Frappe development processes
switch-to-branch Switch all apps to specified branch, or…
switch-to-develop Switch frappe and erpnext to develop branch
switch-to-master Switch frappe and erpnext to master branch
switch-to-v4 Switch frappe and erpnext to v4 branch
switch-to-v5 Switch frappe and erpnext to v4 branch
update Update bench
katerina@katerina-VirtualBox:~/ERPNext$ bench run-tests
Usage: bench [OPTIONS] COMMAND [ARGS]…

Error: No such command “run-tests”.

Run the command from the frappe-bench folder.

@rushabh_mehta, all: just to further clarify, the above-mentioned error is observed on our internal development server where we currently have the configuration as follows

  • ERPNext: v6.27.22
  • Frappe Framework: v6.27.21

Actually, we observe two problems at the same time:

  1. lack of run-tests command in the list of supported bench commands in bench command output (we assumed to see it, based on the considerations in https://frappe.github.io/frappe/user/en/guides/basics/writing-tests.html)
  2. error when trying to run the command, bench run-tests (per the instructions in https://frappe.github.io/frappe/user/en/guides/basics/writing-tests.html)

The specific STR for the first problem is provided by Katerina above.

The specific STR for the second problem we face is as follows

  • we connect to our server, using ssh
  • we navigate to the folder ~/frappe-bench/apps/erpnext there
  • we type the command in the prompt as follows: bench run-tests

Expected behavior: we could expect to see a result of test search/execution or a graceful message about no tests found in the application folder.

The actual behavior is quite different - we see the wall of text in the error stack trace (see P.S. below). This stack trace does not help to understand the root cause of our problem, unfortunately. Could you please help to resolve this issue?

Our current goal is to try frappe unit tests on ERP Next and then see the way to adapt it to our every-day development practices.

Thank you.

George

P.S. below is the stack trace. Any thoughts?


ubuntu@ip-yyy-yyy-yyy-yyy:~/frappe-bench/apps/erpnext$ bench run-tests
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 79, in <module>
    main()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 16, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 24, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/commands/utils.py", line 302, in run_tests
    force=context.force, profile=profile)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 39, in main
    ret = run_all_tests(app, verbose, profile)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 74, in run_all_tests
    _add_test(app, path, filename, verbose, test_suite=test_suite)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 164, in _add_test
    make_test_records(doctype, verbose)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 180, in make_test_records
    make_test_records(options, verbose, force)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 180, in make_test_records
    make_test_records(options, verbose, force)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 180, in make_test_records
    make_test_records(options, verbose, force)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 181, in make_test_records
    make_test_records_for_doctype(options, verbose, force)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 226, in make_test_records_for_doctype
    frappe.local.test_objects[doctype] += make_test_objects(doctype, test_module.test_records, verbose)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 273, in make_test_objects
    d.insert()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 212, in insert
    self.run_before_save_methods()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 687, in run_before_save_methods
    self.run_method("validate")
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 638, in run_method
    return Document.hook(fn)(self, *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 816, in composer
    return composed(self, method, *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 799, in runner
    add_to_return_value(self, fn(self, *args, **kwargs))
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 632, in <lambda>
    fn = lambda self, *args, **kwargs: getattr(self, method)(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/erpnext/erpnext/stock/doctype/item/item.py", line 65, in validate
    super(Item, self).validate()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/website/website_generator.py", line 35, in validate
    self.route = self.make_route()
  File "/home/ubuntu/frappe-bench/apps/erpnext/erpnext/stock/doctype/item/item.py", line 144, in make_route
    return frappe.db.get_value('Item Group', self.item_group, 'route') + '/' + self.scrub(self.item_name)
TypeError: unsupported operand type(s) for +: 'NoneType' and 'unicode'

@vjFaLk: when we run the command from frappe-bench folder, we observe the same sort of issues as I described in my post (see the STR for problem #2 there)

Can you post the log for bench run-tests from the frappe-bench folder?

@Ben_Cornwell_Mott: here it goes

ubuntu@ip-yyy-yyy-yyy-yyy:~/frappe-bench$ bench run-tests
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 79, in <module>
    main()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 16, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 24, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/commands/utils.py", line 302, in run_tests
    force=context.force, profile=profile)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 39, in main
    ret = run_all_tests(app, verbose, profile)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 74, in run_all_tests
    _add_test(app, path, filename, verbose, test_suite=test_suite)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 164, in _add_test
    make_test_records(doctype, verbose)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 180, in make_test_records
    make_test_records(options, verbose, force)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 180, in make_test_records
    make_test_records(options, verbose, force)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 180, in make_test_records
    make_test_records(options, verbose, force)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 181, in make_test_records
    make_test_records_for_doctype(options, verbose, force)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 226, in make_test_records_for_doctype
    frappe.local.test_objects[doctype] += make_test_objects(doctype, test_module.test_records, verbose)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 273, in make_test_objects
    d.insert()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 212, in insert
    self.run_before_save_methods()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 687, in run_before_save_methods
    self.run_method("validate")
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 638, in run_method
    return Document.hook(fn)(self, *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 816, in composer
    return composed(self, method, *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 799, in runner
    add_to_return_value(self, fn(self, *args, **kwargs))
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/model/document.py", line 632, in <lambda>
    fn = lambda self, *args, **kwargs: getattr(self, method)(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/erpnext/erpnext/stock/doctype/item/item.py", line 65, in validate
    super(Item, self).validate()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/website/website_generator.py", line 35, in validate
    self.route = self.make_route()
  File "/home/ubuntu/frappe-bench/apps/erpnext/erpnext/stock/doctype/item/item.py", line 144, in make_route
    return frappe.db.get_value('Item Group', self.item_group, 'route') + '/' + self.scrub(self.item_name)
TypeError: unsupported operand type(s) for +: 'NoneType' and 'unicode'
1 Like

It looks like the test is failing (seems to be Item or Item Group related). Have you written or modified a test?

You can try isolating whether this is a specific test or a system issue by running tests on specific doctypes / modules:

bench run-tests --doctype “Item Group”
bench run-tests --module “Stock”

etc…

1 Like

@Ben_Cornwell_Mott: Hey Ben, thank you for the suggestions.

We have not actually modified any tests provided by Frappe out of the box. Nor did we deploy any new test developed by ourselves yet. However, our dev server contains a number of custom Frappe apps that we developed for our client. Is there any possible impact of those custom apps on the behavior of unit test framework/bench runtime?

A few more findings on our end are as follows:

running bench run-tests --doctype “Item Group”

it reproduces the error as previously specified (see the stack traces in my earlier messages)

running bench run-tests --doctype “Stock”

It leads to an error, too. However, the call stack trace is different (see below):

ubuntu@ip-yyy-yyy-yyy-yyy:~/frappe-bench$ bench run-tests --doctype "Stock"
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 79, in <module>
    main()
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 16, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 24, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/commands/utils.py", line 302, in run_tests
    force=context.force, profile=profile)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 35, in main
    ret = run_tests_for_doctype(doctype, verbose, tests, force, profile)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/test_runner.py", line 93, in run_tests_for_doctype
    test_module = get_module_name(doctype, module, "test_")
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/modules/__init__.py", line 70, in get_module_name
    app = scrub(app or get_module_app(module)),
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/modules/__init__.py", line 77, in get_module_app
    return frappe.local.module_app[scrub(module)]
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/modules/__init__.py", line 17, in scrub
    return frappe.scrub(txt)
  File "/home/ubuntu/frappe-bench/apps/frappe/frappe/__init__.py", line 647, in scrub
    return txt.replace(' ','_').replace('-', '_').lower()
AttributeError: 'NoneType' object has no attribute 'replace'

running bench run-tests on a local VM under Virtual Box

for purity of the experiment, I set up the clean VM under my local Virtual Box instance, using the Production Image published at https://erpnext.com/download. I have not installed any custom apps there yet.

In this configuration, bench run-tests worked ok, with the result output telling, “Ran 415 tests in 548.198s”

You seemed to mistype the stock module test:

bench run-tests --doctype "Stock" is what you wrote when it should be bench run-tests --module"Stock"

That said, I can’t seem to get the module tests to work properly for me either:

frappe@erpnext:~/frappe-bench$ bench run-tests --module Stock
Traceback (most recent call last):
File “/usr/lib/python2.7/runpy.py”, line 162, in _run_module_as_main
main”, fname, loader, pkg_name)
File “/usr/lib/python2.7/runpy.py”, line 72, in _run_code
exec code in run_globals
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 79, in
main()
File “/home/frappe/frappe-bench/apps/frappe/frappe/utils/bench_helper.py”, line 16, in main
click.Group(commands=commands)(prog_name=‘bench’)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 716, in call
return self.main(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 696, in main
rv = self.invoke(ctx)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/core.py”, line 534, in invoke
return callback(*args, **kwargs)
File “/home/frappe/frappe-bench/env/local/lib/python2.7/site-packages/click/decorators.py”, line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/init.py”, line 24, in _func
ret = f(frappe._dict(ctx.obj), *args, **kwargs)
File “/home/frappe/frappe-bench/apps/frappe/frappe/commands/utils.py”, line 302, in run_tests
force=context.force, profile=profile)
File “/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py”, line 37, in main
ret = run_tests_for_module(module, verbose, tests, profile)
File “/home/frappe/frappe-bench/apps/frappe/frappe/test_runner.py”, line 102, in run_tests_for_module
module = importlib.import_module(module)
File “/usr/lib/python2.7/importlib/init.py”, line 37, in import_module
import(name)
ImportError: No module named Stock

1 Like

@Ben_Cornwell_Mott: Ben, thanks for spotting the typo in my command, my bad …

I have just re-run the correct command per what you suggested (that is, bench run-tests --module Stock) . I got the same error trap as you displayed in your last message.

What do you think is the best way to tackle this issue, moving forward?

Thanks.

I’m not sure I know the answer to that question. Obviously it’s bad if a test is failing. You are able to run individual tests (see bench run-tests --help to see how to do that), so you should be able to test your client software. Given that you are running on v6, I’m not sure how much support there is for fixing the root of the issue.

I’ve posted a github issue with run-tests for v7, as I’m getting an issue with Employee creation no matter what test I run:
https://github.com/frappe/erpnext/issues/5998

I’ve also submitted this issue for the module tests failing:
https://github.com/frappe/erpnext/issues/5999

1 Like

@rmehta: thank for the prompt comments to the both issues that @Ben_Cornwell_Mott raised in github, per the thread above.

May I ask you to update the documentation to Frappe unit test framework in order to explain --Module key option more comprehensively, per the details in [ERROR] Run tests fails on module · Issue #5999 · frappe/erpnext · GitHub ?

Doc was updated - Update documentations for test runner options by safo-bora · Pull Request #1978 · frappe/frappe · GitHub