Event Streaming

1

Hello all

I am testing out the event streaming functionality

I have created a common user in both sites

I have created an event producer in the remote site, once saved I noticed that this automatically created an event consumer in the Main site

I have approved the doctypes to be streamed in the event consumer document of the main site, and I noticed the status also changed in the remote site.

But the documents to be streamed (in this case items) are not showing up in the remote site.

There is nothing in the sync log report/list

Is there anything else I need to do?

Olamide

2

The documents created after the setup will be synced and not the ones created till then. So, create an Item after this setup is done. If you have created an Item after the setup, can you please check if the Event Update Log is getting created?

3

Thanks @rucha_mahabal

The event update log is been created at the producer site. Nothing created at the consumer site

I do not know if this matters, but I get the error below if I open the Event update log

image
image1384×771 28 KB

4

Is there no Event Sync Log created on the Consumer Site? Also, make sure that your common user has appropriate permissions for Item creation or whichever doctype you are subscribing to on both the sites.

5

I see the error in the Event Sync Log

what do you think the error below means ?

frappe.exceptions.DuplicateEntryError: (‘Warehouse’, ‘Stores - RH’, IntegrityError(1062, “Duplicate entry ‘Stores - RH’ for key ‘PRIMARY’”))

Regards

6

Now the items are syncing

For the people following behind me, the two companies in the different sites must have the same name.

The system will create the Company, but that item sync will fail.

Thanks @rucha_mahabal

1 Like
7

The dependency syncing should work. The dependencies are synced first and then the item is created. Can you share the entire traceback from the Event Sync Log so that I can work towards fixing the exact cause?

8

Find below

Traceback (most recent call last):
File “/opt/bench/bench13/apps/frappe/frappe/event_streaming/doctype/event_producer/event_producer.py”, line 203, in sync
set_insert(update, producer_site, event_producer.name)
File “/opt/bench/bench13/apps/frappe/frappe/event_streaming/doctype/event_producer/event_producer.py”, line 227, in set_insert
sync_dependencies(doc, producer_site)
File “/opt/bench/bench13/apps/frappe/frappe/event_streaming/doctype/event_producer/event_producer.py”, line 389, in sync_dependencies
dependency.insert()
File “/opt/bench/bench13/apps/frappe/frappe/model/document.py”, line 264, in insert
self.run_post_save_methods()
File “/opt/bench/bench13/apps/frappe/frappe/model/document.py”, line 946, in run_post_save_methods
self.run_method(“on_update”)
File “/opt/bench/bench13/apps/frappe/frappe/model/document.py”, line 813, in run_method
out = Document.hook(fn)(self, *args, **kwargs)
File “/opt/bench/bench13/apps/frappe/frappe/model/document.py”, line 1099, in composer
return composed(self, method, *args, **kwargs)
File “/opt/bench/bench13/apps/frappe/frappe/model/document.py”, line 1082, in runner
add_to_return_value(self, fn(self, *args, **kwargs))
File “/opt/bench/bench13/apps/frappe/frappe/model/document.py”, line 807, in
fn = lambda self, *args, **kwargs: getattr(self, method)(*args, **kwargs)
File “/opt/bench/bench13/apps/erpnext/erpnext/setup/doctype/company/company.py”, line 109, in on_update
self.create_default_accounts()
File “/opt/bench/bench13/apps/erpnext/erpnext/setup/doctype/company/company.py”, line 160, in create_default_accounts
create_charts(self.name, self.chart_of_accounts, self.existing_company)
File “/opt/bench/bench13/apps/erpnext/erpnext/accounts/doctype/account/chart_of_accounts/chart_of_accounts.py”, line 61, in create_charts
_import_accounts(chart, None, None, root_account=True)
File “/opt/bench/bench13/apps/erpnext/erpnext/accounts/doctype/account/chart_of_accounts/chart_of_accounts.py”, line 52, in _import_accounts
account.insert()
File “/opt/bench/bench13/apps/frappe/frappe/model/document.py”, line 225, in insert
self._validate_links()
File “/opt/bench/bench13/apps/frappe/frappe/model/document.py”, line 782, in _validate_links
frappe.LinkValidationError)
File “/opt/bench/bench13/apps/frappe/frappe/init.py”, line 385, in throw
msgprint(msg, raise_exception=exc, title=title, indicator=‘red’, is_minimizable=is_minimizable)
File “/opt/bench/bench13/apps/frappe/frappe/init.py”, line 364, in msgprint
_raise_exception()
File “/opt/bench/bench13/apps/frappe/frappe/init.py”, line 315, in _raise_exception
raise raise_exception(msg)
frappe.exceptions.LinkValidationError: Could not find Company: Xlevel

9

Hi @rucha_mahabal

Is there any way we can push doctypes created before the event consumption list was created.

Use case scenario, we set up a new branch and they require all items to be able to make sales. It would be simpler to push items to them rather than having to upload item list.

What do you think? Way out?

10

I figured it out

Once you do a transfer to the branch or a purchase receipt the items will be created automatically at the remote branch

11

Ok @rucha_mahabal

Things are looking good so far.

I however have a request to make.

is there a way to filter the documents to be streamed?

Use case scenario, in a retail environment I may not want to stream all Material transfers to all stores, I may just want to stream transfers meant for store A to Store A and B to B.

How possible is this ?

12

@olamide_shodunke

So currently syncing all the previous data on subscription is not available. However, in my opinion, it is better if this first stage can be taken up manually using tools like Data Import. I can still add it to my todo list and find out a way to sync all the previous master data on subscription approval based on some configuration (not all doctypes you subscribe to need a data dump). Apart from that, the on-demand dependency syncing is already in place. So if the consumer node has subscribed to Sales Invoices while syncing the Sales Invoices if items, cost centers, etc are not created on the consumer node, they will be created first and then the Sales Invoice will be synced.

Glad to hear that!

This is a really good use case. Condition-based syncing is really important. I’ll add this to my to-do list. Thanks for the feedback!

4 Likes
13

Hello all

I have a question regarding Exactly how Event streaming functions:
From what i can see from the code, If change happens on the Producer, the producer sends a post request to ‘event_producer.event_producer.new_event_notification’ of the consumer with the url of the producer. The Consumer then pulls all changes since last from the producer. If you want to send anything back, you will need to create a Producer on what was ‘the consumer’ and consume it on what was ‘the producer’ and the same notify and pull would happen from the other side?

  1. Why not send the event over websocket rather than notify the client to pull?

  2. In cases where the clients connection is firewalled (which is the case most of the time as you would avoid external access to you local site from a security perspective) or has a Dynamic IP. So connection would only be possible from Client to Server and not from Server back to client. You would need to use a VPN which would in turn add extra overhead on the network side again. Web socket again would solve this issue as Client can connect to the Server and server can send updates back as well without issues regarding firewalls and dynamic IPs or the need for VPN`s. Client would also be able to send Back any Events it needs over the same socket?

Amazing work so far though. I have clients currently begging for this functionality.

Also i just want to +1 the idea of “syncing all the previous data on subscription” and +1 on Condition-based syncing. This would be an amazing feature especially in cases where where the Local site on the Remote Area crashed and needs to be rebuild ASAP. It would be amazing if you where able to point a blank server at the main server and it sync`s all data needed for that branch automatically.

4 Likes
14

@fkardame You want us to get it on here?

15

Not really as there are w different objectives.
So what event streaming is trying to achieve is fine but not suitable for every deployment with multiple branches.
As it will need local erp instances at every location which will increase the cost of hardware and its maintenance.

16

Depends on the “cost” of this hardware

We have ERPNext running locally in some locations, some with as much as 10 concurrent users. The hardware the “server” is running on is basically the same configuration as the PCs the staff use for their day to day tasks.

I agree that this may not be the case in all situations, but in my experience the usual ERPNext installation does not require that much hardware power.

So this might not be that much of a deal breaker as you think

17

Possible options

  • Erpnext running on virtual machine with Windows host.
  • Linux desktop environment with erpnext running on it.
  • Erpnext running on wsl.

All the solution will need a piece of hardware no matter a simple machine or a server machine and it will need regular maintenance.
The setup time for individual configure of each location will increase automatically.
Its just an overhead to maintain stability, it will require higher support man hours.
My major concern

  • Additional expense for hardware and maintenance.
  • Extra man hours will be needed to maintain this type of deployment.

This deployment is ideal for large enterprise with high volume of documents but not for an sme with high number of remote locations with low volume of documents.

Will try to test the existing feature in the coming week and share my feedback.
Btw with this feature we will try to compete with Microsoft dynamic, sap and oracle, so it is a plus point.

18

Hello @rucha_mahabal

I keep getting the following error message and it prevents sales invoice from showing up at the consumer site

Traceback (most recent call last):
File “/opt/bench/develop/apps/frappe/frappe/event_streaming/doctype/event_producer/event_producer.py”, line 205, in sync
set_update(update, producer_site)
File “/opt/bench/develop/apps/frappe/frappe/event_streaming/doctype/event_producer/event_producer.py”, line 253, in set_update
local_doc.save()
File “/opt/bench/develop/apps/frappe/frappe/model/document.py”, line 281, in save
return self._save(*args, **kwargs)
File “/opt/bench/develop/apps/frappe/frappe/model/document.py”, line 323, in _save
self.validate_update_after_submit()
File “/opt/bench/develop/apps/frappe/frappe/model/document.py”, line 739, in validate_update_after_submit
self._validate_update_after_submit()
File “/opt/bench/develop/apps/frappe/frappe/model/base_document.py”, line 669, in _validate_update_after_submit
frappe.UpdateAfterSubmitError)
File “/opt/bench/develop/apps/frappe/frappe/init.py”, line 385, in throw
msgprint(msg, raise_exception=exc, title=title, indicator=‘red’, is_minimizable=is_minimizable)
File “/opt/bench/develop/apps/frappe/frappe/init.py”, line 364, in msgprint
_raise_exception()
File “/opt/bench/develop/apps/frappe/frappe/init.py”, line 315, in _raise_exception
raise raise_exception(msg)
frappe.exceptions.UpdateAfterSubmitError: Not allowed to change Date after submission

19

Was this sales invoice cancelled and amended on the producer site?

20

No it was not. It was a fresh transaction.

But now that you mention it, In my tests I also found out that if two customers were merged into one. Transactions from the resulting customer always give an error.

But this sales invoice was a fresh transaction on a unique customer