Frappe 7 will replace Celery with Python RQ

Hi everyone,

This might come as a shock to some developers, but we are moving away from Celery and will be replacing it with Python RQ.

We have had some bad experiences using Celery for background tasks. It is too complicated, takes a lot of time to debug and often feels like a black box.

Python RQ is really really simple. Because of its minimal feature set, we gain a lot more control and visibility of background jobs. It also has a great monitoring utility called rq-dashboard.

We will release Frappe 7 in about 2-3 weeks.

If your custom app has scheduled jobs like all, hourly, daily, etc. mentioned in hooks, then you won’t have to change anything. :smile:

However, if you have background jobs decorated using @celery_task that you called using method.delay(), you might have to change the call to something like frappe.utils.background_jobs.enqueue(method, **kwargs). Also, the enqueue method will handle initialization of site, database connection, rollback/commit and cleanup, so that you only have to worry about your background job.

Along with this, we are also cleaning up the bench installer (will be using ansible which is much much easier to maintain) and adding multi-bench capabilities so that you can host different sets of sites in different benches. I think those who really need this feature know the immense benefits of this. :wink: So donations are welcome :blush:

We will keep you posted on our progress.

Best,
Anand Doshi.

PS. I would love to hear your thoughts on this.


Edit 1: We will also be dropping support for the Async Tasks feature. It wasn’t documented and didn’t work reliably. You can still call functions asynchronously by enqueuing them, but there will be no async magic available.

24 Likes

a question coming without any background in writing code, just pure interest …

will that change require existing custom apps (which might be depending on celery) to be re-written?

If so, would that break a system which still depends on Celery (due to any custom App) when updating to Frappe 7?

@vrms I think I mentioned that

1 Like

Hello Anand,

Great News…!

Most of users will like the multi-bench feature. We are eager to update bench to v7.

Thanks and Regards,
Gangadhar Kadam

@anand thanks for clarifying & sorry for apparently not having grasped your initial post in all required detail

Reading some threads over the net favored celery over RQ for advanced apps:

https://www.quora.com/What-is-the-difference-between-RQ-redis-queue-and-Celery

and

Is there a document mentioning technical merit/rationale for adoption of RQ by ERPNext except celery being too complicated and difficult to debug? Just curious.

@anand It is already 4 weeks (29 days), just to remind you on a lighter note. :wink:

Try to deploy once Frappe 7 will be released.

I thinks its already implemented in the development branch. Please see [develop branch] Celery deprecated. Long live Python RQ

RQ had all features we needed. Using a simpler system is better.

1 Like

Hello,

i have heard, that with frappe 7 and bench 3 there will be a possibility to deploy multiple benches on one server.

i have a question regarding multi-bench setup on a single server.

Is there a documentation somewhere on how to drive multi-bench on one server?

The usecase is to run the production site on one bench and run a development and testing site on another bench.

All 3 benches shall run in production mode on the remoteserver, but on each bench shall be running an ERPNext instance and other apps with a different Git-branch.

This is a very important usecase. We need explicitly separated benches, because it is possible to kill a bench with all its sites when making a mistake on one site.
Also it is not possible to install the erpnext app from different branches at one time in one bench.

This is not possible, because the apps are benchwide by design.

@vjFaLk can you write the docs for this?

1 Like

It is a very good decision which your team has taken.