Is it possible to do automated runs using Data Migration Tool. Secondly is it possible to control data to get in a pull type sync using data migration tool ? e.g daily I want to download certain items generated only on that day from a remote source.
Thanks,
Palash
Hi @palashjhabak,
Yes it is possible to do automated runs by adding a task in Frappe’s hooks.
You can take a look at https://github.com/DOKOS-IO/mautic if you want a concrete example of how this can be done.
As for controlling data, sure you can do everything you want by adding custom methods in your different data mappings. Again, you can check the above code to see examples of what can be done with the data migration tool.
Good luck!
Awesome. Let me have a check and get back here if I have more doubts. Thanks for the quick response @chdecultot really appreciate it.
Palash
Scheduler part I am clear from your mautic app. Will mention a scheduler in hooks.py which will programmatically create a new run in every period
Data controlling code I will write in connector ( get method perhaps ? to sync from remote to local ) and in the field mappings can I map direct fields in local doctype to field inside object of object in json coming from remote source or will I have to write pre process to achieve this ?
Palash
In my experience, the best practice would be to only use the connector to send/receive data using a vanilla API or an API library.
If you need to pre or post process data coming from the API or before sending it to the API, you can use the pre- post-process methods in the init.py file of each mapping.
This way, you have a clear separation between data processing and data sending/receiving.
If you don’t need any data processing (same data structure between the remote software and ERPNext) then just use a simple mapping without pre-/post-processing functions.
Agree with your suggestions but if my remote API accepts filters e.g date filters (to get data of any particular dates) where do I give these filters to the API? And where should I store the last synced date? so that in the next run I give next applicable date as the filter.
I am not sure if I have clearly started my problem statement.
According to my understanding pre-process is called just after the data has come from the remote API (in the case of pull) before passing it to the data migration tool. So filters like date
Yes sure, if you need to pass a filter to the API you should pass it in the connector.
Any filter registered in the mapping will be automatically passed to the connector.
As an example, I have already integrated a “search” filter in the Mautic connector: search = filters.get('search')
You just need to add this key and a value in your mapping and it will be automatically passed to the connector.
You are also correct, the pre-process method is called after fetching the remote data (for a GET statement). You can use it for any data processing before the creation/update of the local data.
Post-process is called after the local data creation/update.
For a PUSH operation, I would say store your last synced date in your settings doctype. I guess it is the easiest.
You could also get it by analyzing previous migration runs. Up to you!
Thanks for the detailed clarification. I think I now have fair bit of idea of how to go about it.
Thanks,
Palash