Been looking for generic tools to help write data-migrations/crawlers to have runners to mirror data from a GraphQL or REST end-point to local storage to ease integrations with say ERPNext.
My use cases could be to periodically extract data from 3PL warehouses or shipping carriers to a local file or db storage to make custom modules weaving in that data fast and effifient (than fetching on demand from various api’s, especially since FedEx and UPS tend to limit history via their API’s)
Another use-case would be to mirror parts/components/items from PLM-like systems into ERPNext. Making a tailored integration would of course work, but it’s nice to split it up in steps, storing the raw responses from 3rd party API’s, then apply the transformation and filtering before injecting the data in a custom DocType.
Thoughts?
I’ve briefly looked at Singer. It looks promising. But there’s a learning curve, and I haven’t had time to decipher its terminology and structure.
I used to work on Microsoft’s ERP systems. I miss SQL Integration Services, and haven’t found anything equivalent for Linux/MySQL.
So far, whenever I’ve needed ETL for ERPNext, I’ve just been writing Python from scratch. Slowly building a personal library of useful scripts, migration kits, etc.
Would prefer using a robust ETL tool, that meets my requirements, so that I’m not reinventing the wheel. But I haven’t found anything yet.
My reasoning behind looking at standalone “runners” for integration tasks is mainly to ensure we can split up the task in smaller pieces, i.e. fetching ALL data first to a local cache, before creating entries in erpnext. You don’t want to end up in a situation where a slow REST endpoint stops responding and you have ended up with incomplete data in the destination database…