Python 3 Migration Discussion

Bench can be installed on python 3.5 try it

DO NOT RUN ON PRODUCTION SYSTEM

sudo pip3.5 install -e bench-repo

Generally recommended to run 2to3 script to migrate to 3. There are some drawbacks for urllib import, etc.
http://python3porting.com/noconv.html#import-errors

Relative small size of source code of bench and written tests made easy to run 2to3 and then manually correct imports.

Bench commands works. (not the frappe.commands)

Frappe and ERPNext cannot be installed on/using this bench.

Code needs change.

Ran 2to3 script on frappe here is a gist https://gist.github.com/revant/52bd8e8c5e745bc030a54d169ff7ee54

Github Issue:

4 Likes

@revant_one thanks for this…

Does not seem so major …

I am more worried about migration. Can we just create a new virtualenv in bench and then migrate ?

What I’m manually doing to test is,

sudo pip3.5 install -e ~/.bench
cd ~/frappe-bench-dev
rm -fr ./env
bench setup env

There were error with MySQL-python while installing frappe on pure v3, replaced mysqlclient (just to proceed)

Bench still needs some changes for pure v3 use.

revant@revant-laptop:~/frappe-bench-dev$ bench get-app https://github.com/frappe/frappe
INFO:bench.app:getting app frappe
Traceback (most recent call last):
  File "/usr/local/bin/bench", line 11, in <module>
    load_entry_point('bench', 'console_scripts', 'bench')()
  File "/home/revant/.bench/bench/cli.py", line 40, in cli
    bench_command()
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/revant/.bench/bench/commands/make.py", line 30, in get_app
    get_app(git_url, branch=branch)
  File "/home/revant/.bench/bench/app.py", line 58, in get_app
    shallow_clone = '--depth 1' if check_git_for_shallow_clone() else ''
  File "/home/revant/.bench/bench/utils.py", line 321, in check_git_for_shallow_clone
    git_version = get_git_version()
  File "/home/revant/.bench/bench/utils.py", line 308, in get_git_version
    version = '.'.join(version.split('.')[0:2])
TypeError: a bytes-like object is required, not 'str'

The above error only exists on pure v3 bench. v2 bench works as expected.

python-future is the missing compatibility layer between Python 2 and
Python 3. It allows you to use a single, clean Python 3.x-compatible
codebase to support both Python 2 and Python 3 with minimal overhead.

1 Like

It seems __future__ is used in frappe.
https://github.com/frappe/frappe/blob/develop/frappe/init.py#L7

Bench doesn’t seem to use __future__
Right now bench init is not working on 3.5. there seems to be error with str / bytes.

@iHello Can you help compatibility correction of bench using future?

Hello @revant_one

Can you post a ink to the branch you are working on?

Right now everything is merged into upstream / master.
No branch on fork

Just install the bench using following command

sudo pip3.5 install -e bench-repo

instead of

sudo pip2.7 install -e bench-repo

ok cool.

Hello @revant_one.

I have not been able to reproduce the problem as you have spelt out albeit i have successfully installed bench on python 3.5 but bench init fails as I expect. The reason why I expect it to fail is that bench init is tightly coupled with frappe and frappe is also not yet python 3.5 compatible.

Here is how I setup my system.

  1. I created a virtual environment with python 3.5
  2. Installed bench with sudo pip3.5 install -e bench-repo. No problems encountered
  3. I changed setup_env function in bench/utils.py by removing the -p option. This will ensure that the virtualenv will be created with the virtual environment I created.
  4. I ran bench init frappe-bench

This is where I started facing problems:

  1. MysqlDB is not python 3 compliant and most likely never will be. I changed the requirements.txt to swap python-mysql with pymysql instead and then tweaked init in bench/utils.py to not raise any exception as i try the bench init command again.

From this point on, bench init starts stalling because of python 2 style coming from frappe. Since bench is so tightly coupled with frappe, I think we also need to make sure that frappe is also python 3 compliant. I actually have a PR looking for some love in frappe - https://github.com/frappe/frappe/pull/3216 and in my free time I am going to fix other things like unicode(), xrange, DropboxClient, iteritems, dictionary comprehensions, basestrings, etc. These are the issues I have discovered at the moment.

In the mean time, did you encounter all these issues?

1 Like

Tried again today. Faced same thing you faced.

revant@revant-laptop:~$ sudo pip3.5 install -e ~/.bench
[sudo] password for revant: 
The directory '/home/revant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/revant/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Obtaining file:///home/revant/.bench
Requirement already satisfied: Click in /usr/local/lib/python3.5/dist-packages (from bench==4.1.0)
Requirement already satisfied: jinja2 in /usr/lib/python3/dist-packages (from bench==4.1.0)
Requirement already satisfied: virtualenv in /usr/lib/python3/dist-packages (from bench==4.1.0)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from bench==4.1.0)
Requirement already satisfied: honcho in /usr/local/lib/python3.5/dist-packages (from bench==4.1.0)
Requirement already satisfied: psutil in /usr/lib/python3/dist-packages (from bench==4.1.0)
Requirement already satisfied: python-crontab in /usr/local/lib/python3.5/dist-packages (from bench==4.1.0)
Requirement already satisfied: semantic_version in /usr/local/lib/python3.5/dist-packages (from bench==4.1.0)
Requirement already satisfied: GitPython==0.3.2.rc1 in /usr/local/lib/python3.5/dist-packages (from bench==4.1.0)
Requirement already satisfied: MarkupSafe in /usr/lib/python3/dist-packages (from jinja2->bench==4.1.0)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.5/dist-packages (from python-crontab->bench==4.1.0)
Requirement already satisfied: gitdb>=0.5.1 in /usr/local/lib/python3.5/dist-packages (from GitPython==0.3.2.rc1->bench==4.1.0)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil->python-crontab->bench==4.1.0)
Requirement already satisfied: smmap>=0.8.5 in /usr/local/lib/python3.5/dist-packages (from gitdb>=0.5.1->GitPython==0.3.2.rc1->bench==4.1.0)
Installing collected packages: bench
  Found existing installation: bench 4.1.0
    Can't uninstall 'bench'. No files were found to uninstall.
  Running setup.py develop for bench
Successfully installed bench
revant@revant-laptop:~$ bench init test-bench
INFO:bench.utils:virtualenv -q env -p /usr/bin/python3
INFO:bench.utils:./env/bin/pip -q install --upgrade pip
INFO:bench.utils:./env/bin/pip -q install wheel
INFO:bench.utils:./env/bin/pip -q install -e git+https://github.com/frappe/python-pdfkit.git#egg=pdfkit
INFO:bench.app:getting app frappe
INFO:bench.utils:git clone https://github.com/frappe/frappe.git   --origin upstream
Cloning into 'frappe'...
remote: Counting objects: 126346, done.
remote: Compressing objects: 100% (161/161), done.
remote: Total 126346 (delta 70), reused 0 (delta 0), pack-reused 126185
Receiving objects: 100% (126346/126346), 95.64 MiB | 214.00 KiB/s, done.
Resolving deltas: 100% (94917/94917), done.
Checking connectivity... done.
installing frappe
INFO:bench.app:installing frappe
INFO:bench.utils:test-bench/env/bin/pip install -q  -e test-bench/apps/frappe --no-cache-dir
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-lly_nux8/mysql-python/
Traceback (most recent call last):
  File "/usr/local/bin/bench", line 11, in <module>
    load_entry_point('bench', 'console_scripts', 'bench')()
  File "/home/revant/.bench/bench/cli.py", line 40, in cli
    bench_command()
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/revant/.bench/bench/commands/make.py", line 19, in init
    verbose=verbose, clone_from=clone_from)
  File "/home/revant/.bench/bench/utils.py", line 56, in init
    get_app(frappe_path, branch=frappe_branch, bench_path=path, build_asset_files=False, verbose=verbose)
  File "/home/revant/.bench/bench/app.py", line 76, in get_app
    install_app(app=app_name, bench_path=bench_path, verbose=verbose)
  File "/home/revant/.bench/bench/app.py", line 107, in install_app
    find_links=find_links))
  File "/home/revant/.bench/bench/utils.py", line 130, in exec_cmd
    raise CommandFailedError(cmd)
bench.utils.CommandFailedError: test-bench/env/bin/pip install -q  -e test-bench/apps/frappe --no-cache-dir
revant@revant-laptop:~$ 

I setup environment with command following instead it sets up v3 bench

bench setup env

Good thing it’s now obvious that frappe needs to be python 3 compliant before we can move forward.

I also tried using bench setup env but after removing the initial env folder, bench was complaining about not finding usr/bin/pip. It was turning to too much of a hassle for me to debug it so I just edited the code to remove the -p option.

Anyway, I’m going to continue fixing the frappe code to make it python 3 compatible.

Meanwhile, python-mysql has to go…

1 Like

Any news about this? We are evaluating erpnext and we are very interested in the support of python3.

Python 3 support is on our road map to v9. At the moment, you can successfully bench init with python 3. One thing holding us back is third party dependencies that are still locked in python 2. Once we can get those out of the way, it’ll be smooth sailing. But the target is to have python 3 compatibility before the end of 2017.

If you are a developer, you can also help us by pushing some python 3 compatibility code. Contributions are welcome.

3 Likes

Yes, we have two programmers here and the intention is to release all the code we make if we decide to use erpnext (obviously all that is not company knowhow;)).

But the priority here is that the erp is 100% functional on January 1, 2018, by the time this is achieved there will be no problem to collaborate in what we can to improve the system :slight_smile:

Here is a complicated implementation, we have a group of companies with several physical stores, own online stores, vertical ebay stores and we are a complicated business as is the second hand.

At the end of this week we will decide if we bet Erpenext, if so we will comment to see if we can help in parallel while we set up the system :slight_smile:

2 Likes

A change that we made at work to support celery inside docker containers to django sites was to replace supervisord (py2 only at the moment) with circus (equivalent maintained by mozilla) https://circus.readthedocs.io/en/latest/. Tested with pyhon 3.5 and 3.6.

1 Like

We are also presently considering circus. Thanks for the heads up.

And we hope you like ERPNEXT enough to join :slight_smile:

1 Like

Is there any list of hotspots, or a master bug that links to all instances of Py3 bugs, so that I can see where I can contribute? I’m pretty solid at Python, and have done a few 2->3 conversions, but I would rather step into such a large codebase with a good idea of what’s been triaged so I can help most efficiently. I’d really prefer to work in 3 for real client work instead of 2.7+future, which is lacking in so many ways.

Just tried right now, all on develop branches
sudo pip3 install -e bench-repo successful
bench init --frappe-branch develop frappe-bench-py3 error with bench while backup
bench new-site test_frappe_py3 successful
bench start successful
0.0.0.0:8001/login is successful
there were errors during login.
you can try on your own, manually install bench.

These are two current issues for python 2 to 3 that I see open. Others can also point out steps.