Try ERPNext Buy Support Partners Foundation

[Important] Bench version 2 - see this before you run bench update

Hi everyone,

We have merged bench v2 into master.

Things can break.
So, if you want to continue without disruption, you can change the bench branch to v1.x.


Troubleshooting

  1. If the update breaks at: bench.utils.CommandFailedError: sudo supervisorctl restart frappe-bench-processes:, update again. A fix has been pushed.

  2. If the system keeps asking for sudo password for a user, you will need to go to your frappe-bench folder and run sudo bench setup sudoers USER. Replace USER with frappe / ubuntu or whichever user you used to install erpnext.

  3. If you see an error like below, but if you see that the processes stopped and restarted, you can ignore this line.

    error: <class 'xmlrpclib.Fault'>, <Fault 10: 'BAD_NAME: frappe-bench-processes'>: file: /usr/lib/python2.7/xmlrpclib.py line: 794
    
  4. If you still want to get rid of the above message, you will have to regenerate all your configuration files by going to the frappe-bench folder and running these commands:

    mv config config-bak
    mkdir -p config/pids
    sudo supervisorctl stop all
    sudo service nginx stop
    bench setup config
    bench setup redis
    bench setup supervisor
    bench setup nginx
    bench setup procfile
    sudo bench setup sudoers frappe # or the user that you used to install erpnext
    sudo nginx -t  # check if this command shows any error 
    sudo service nginx reload    
    sudo supervisorctl reread
    sudo supervisorctl update
    

    If after doing this, you get the following error in sudo supervisorctl status:

        frappe-bench-redis:frappe-bench-redis-queue FATAL      Exited too quickly (process log may have details)
    

    then run:

        sudo update-rc.d redis-server disable
        sudo service redis-server stop
        sudo supervisorctl start all
    
  5. If you get the error pkg_resources.DistributionNotFound: bench==0.92, try:

    # go back one level outsite frappe-bench folder
    cd ..
    sudo pip uninstall bench
    sudo pip install -e bench-repo

Thank you @Kody_Baker, @revant_one, @Ben_Cornwell_Mott for your persistence :smile:


What’s new?

  1. frappe-bench/config.json deprecated and merged into frappe-bench/sites/common_site_config.json. This will be done automatically on bench update

  2. Some keys in site_config.json / common_site_config.json have changed, which will be changed automatically on bench update.

    • celery_broker --> redis_queue with default as redis://localhost:6379
    • async_redis_server --> redis_socketio with default as redis://localhost:12311
    • cache_redis_server --> redis_cache with default as redis://localhost:11311
  3. File names in frappe-bench/config have changed. If you get any error about config, run:

     bench setup redis
     bench setup procfile
    
     # if production setup
     bench setup supervisor
     bench setup nginx
    
  4. Multi-bench setup

    • Assumes all benches are installed in parallel, i.e. some-folder/frappe-bench-1, some-folder/frappe-bench-2, etc.
    • On bench init, loops through all benches’ configuration and sets unique ports for webserver, socketio and various redis-server
    • bench setup production creates symlinks of nginx and supervisor with the bench’s name
  5. Other improvements

    • A new installer using ansible (WIP) - for now creates development setup
      (production setup yet to be implemented)
    • Test cases for bench and refactored code
    • Nginx https config auto-redirects http:// to https://

What do you think? I am sure some of you must have questions.

14 Likes

@anand, thank you for the announcement! I’ll give it a shot tomorrow.

I have a question about #2. I can’t find any information about redis usage in Frappe, could you please clarify why 3 instances of redis are needed instead of one? Or at least point me at some article.

  1. redis-queue - celery queuing
  2. redis-cache - caching
  3. redis-socketio - pub/sub for realtime using socketio

We are using 3 instances because each redis instance can utilise separate cores of the processor and thus give better throughput. You can use a single redis but partition it in different databases, like redis://localhost:6379/0, redis://localhost:6379/1 and redis://localhost:6379/2, (or use the same redis, which I think will work without issues too).

Frankly, I am not a redis expert. So if you think this is a bad design, we could surely re-evaluate :smile:

I’m not a redis expert either, but I think it was a good decision in that case.

However there is a trend of using more machines with average hardware instead of one big machine with lots of CPUs and RAM. ERPNext is already pretty big system for one machine and I think some manual is required to cover complex deployment cases.

@alex_melkoff monolithic architecture has been working for us. One of our servers has like 4000 sites. It keeps things simple.

@anand, wow, that’s a lot of sites… But what about highload? How many simultaneous users can such a system handle?

@alex_melkoff we don’t have stats on that yet. But in the coming months, we will be capping it to 1000 users a server just for safety. Still that is a big number and turns out to be cheaper and easier to manage. And it is so easy to troubleshoot too :smile:

I assume as long as you just do bench update nothing is going to happen apart from updating ERPNext and frappe (using the existing bench), right?

In order to pull the new bench v2, you’d have to specifically perform a git pull inside of the folder where the bench is installed. Is that so?

@vrms bench update also updates bench code. Since I can’t possible test it for all scenarios, things could break. So if you don’t want to take even a bit of risk, you should change bench’s branch to v1.x. Or try it out in your local/test setup first.

@anand just to clarify and out of interest

  1. which of the branches in your repo is the v2? The new-install branch?
  2. when precisely are you going to merge? Will there be an extra announcement, so anybody knows, when to be extra cautious with a bench update?

Merged

2 Likes

Things broke for our installation. Want this posted to github rather than this thread?

Updates everything, but then at the end of setup after writing the ./assets/, it hits this error:

Wrote ./assets/css/module.min.css - 2k
error: <class 'xmlrpclib.Fault'>, <Fault 10: 'BAD_NAME: frappe-bench-processes'>: file: /usr/lib/python2.7/xmlrpclib.py line: 794
Traceback (most recent call last):
  File "/usr/local/bin/bench", line 9, in <module>
    load_entry_point('bench==0.1', 'console_scripts', 'bench')()
  File "/home/ubuntu/bench-repo/bench/cli.py", line 40, in cli
    bench_command()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/bench-repo/bench/commands/update.py", line 60, in update
    _update(pull, patch, build, bench, auto, restart_supervisor, requirements, no_backup, upgrade, force=force)
  File "/home/ubuntu/bench-repo/bench/commands/update.py", line 96, in _update
    restart_supervisor_processes(bench=bench_path)
  File "/home/ubuntu/bench-repo/bench/utils.py", line 267, in restart_supervisor_processes
    exec_cmd(cmd, cwd=bench)
  File "/home/ubuntu/bench-repo/bench/utils.py", line 99, in exec_cmd
    raise CommandFailedError(cmd)
bench.utils.CommandFailedError: sudo supervisorctl restart frappe-bench-processes:

For reference: supervisorctl status

frappe:frappe-web                RUNNING    pid 18572, uptime 1 day, 0:55:17
frappe:frappe-worker             RUNNING    pid 18571, uptime 1 day, 0:55:17
frappe:frappe-workerbeat         RUNNING    pid 18580, uptime 1 day, 0:55:16
node-socketio                    RUNNING    pid 1634, uptime 6 days, 0:08:53
redis-async-broker               RUNNING    pid 1633, uptime 6 days, 0:08:53
redis-cache                      RUNNING    pid 1648, uptime 6 days, 0:08:53

Ok let me see if I can make this backward compatible. Give me 5 mins.

@Kody_Baker can you try running update again? I’ve pushed a fix.

In your case, you might need to run cd /home/ubuntu/frappe-bench && setup sudoers ubuntu if it keeps asking you the sudo password for ubuntu user

Looks like we’re 99% fixed.
No more traceback, but the error persists.

Wrote ./assets/css/module.min.css - 2k
error: <class 'xmlrpclib.Fault'>, <Fault 10: 'BAD_NAME: frappe-bench-processes'>: file: /usr/lib/python2.7/xmlrpclib.py line: 794
frappe-web: stopped
frappe-workerbeat: stopped
frappe-worker: stopped
frappe-web: started
frappe-workerbeat: started
frappe-worker: started

You can ignore that error till you move to the new configuration. For new configuration you will have to regenerate all config files + nginx config.

How do I move to new config?
bench update migrate?

@Kody_Baker see point 4 in troubleshooting in the first post

1 Like

Worked. New error now at the end:

...Wrote ./assets/css/module.min.css - 2k
frappe-bench-frappe-async-worker: stopped
frappe-bench-frappe-longjob-worker: stopped
frappe-bench-frappe-workerbeat: stopped
frappe-bench-frappe-web: stopped
frappe-bench-frappe-worker: stopped
frappe-bench-frappe-async-worker: started
frappe-bench-frappe-longjob-worker: started
frappe-bench-frappe-workerbeat: started
frappe-bench-frappe-web: started
frappe-bench-frappe-worker: started
frappe-bench-node-socketio: ERROR (abnormal termination)