New App for one-page shopping cart experience

gvyshnya · July 8, 2016, 5:09pm

Hello,

We are impressed and grateful for the work done by Frappe and the rest of the community dedicated to building ERP Next into a truly remarkable and enabling platform for organizations worldwide.

We are building a new, custom ERPN/Frappe application to deliver a one-page shopping cart/checkout experience to Web users with ERPN-backed ecommerce sites. In addition we have been building an associated Payment Gateway Setup application to make deploying payment gateways a bit easier.

At DigiThinkIT (Headquarterd in Orlando, Florida, USA) we believe “a rising tide raises all ships” and we want to make solid and valuable contributions to the Community with the intention that some or all of this work may be used to enhance this great platform for all who use it.

After a lot of research, we believe we have identified the following limitations to the default Shopping Cart application which we intend on addressing with a new application:

it does not allow more than one payment gateway per system instance
it does not support complex product configurations where a user has to configure its almost unique product item’s look and feel based on the set of pre-defined manufacturing options/features (this would typically map the unique configured product to a Quote and Sales order with multiple line items on ERP Next backend side etc.)
it has less than efficient multi-page and multi-step UI/UX for check-out flow (ideally, the one-page checkout experience is captivating to the end users)
it does not allow estimated shipping costs to the quote at the shopping/cart based on real time apis from various shippers
it does not facilitate the scenario where a user places a preliminary (or “Saved Cart”) order to the system, without immediate online payment performed (in which case the system will ideally map such a preliminary order to a Quote Document in ERP Next rather than creates a new Sales Order)
when a guest user without an ERPN account shops on the site and is prompted to create an account in the system (to make the first purchase), the system does not return a user to the cart after login (this is a little confusing to less than savvy users)
it doesn’t allow for multiple payment options (PayPal, CC, Amazon Pmts, Apple Pay, eCheck)

Our intention is to build a few new apps that address most of the limitations above.

The purpose of this post is to share our general architecture/implementation plan, report our progress, discoveries and challenges as we also get feedback from the community on:

Feasibility of the proposed approach/possible limitations and drawbacks
Additional features that may be useful
Any other relevant feedback on the solution proposed

Lastly, we are hiring and always looking for talented developers who are passionate about using their talent to empower others with great tools. If you are interested in finding a home with a dynamic, driven team that likes to solve complex challenges in software, feel free to contact us.

The proposed solution architecture is highlighted in the diagram below:
https://drive.google.com/file/d/0B5lf5LUI2xktQVotRmkxNnpKMWM/

In an ideal world, we envision extending ERPN following the current modular architecture with an extensible system for plugging new payment gateways to the proposed infrastructure. We are going to try it by developing the new payment gateway for Aurothize.net for Credit Cards and eCheck and direct ACH, PayPal, and others and stacking it there, together with the existing core app for Paypal-ERPN integration.

We appreciate any feedback/critique/suggestions on our proposed solution. We will work to provide periodic updates regularly in this thread for this project and will provide GitHub access soon.

Respectfully,

George Vyshnya,
on behalf of ERP Next team at DTI (www.digithinkit.com)

Laurens · July 8, 2016, 5:13pm

Wow brilliant! What a great idea. Will your team also focus on a nice webshop layout to enhance user experience? I feel this also lacks somehow in the default cart / site.

Eric_Delisle · July 8, 2016, 5:50pm

@Laurens we will be working on the layouts as well. We have already built a very complex 3D designer for configuring custom products and have this currently tied into the existing ERPN items system through BOMs and workflows for manufacturing.

As we accomplish more we will share it with the community.

Laurens · July 8, 2016, 5:53pm

Nice. Will this be a separate app to install or will it be integrated with the existing erpnext app? Basically replacing the default website? As I understand it, the basic functionality is the same (but better and with improvements and extra features) so I guess merging it with the existing erpnext app would be a great idea.

Eric_Delisle · July 8, 2016, 5:58pm

Right now we are only creating a custom app that will work with existing framework as closely as possible. We will certainly offer it to the core once it is built, tested, etc., but this decision will be up to Frappe itself.

rmehta · July 11, 2016, 9:10am

@gvyshnya @Eric_Delisle

Thanks for the document

I think you should also consider fixing as many issues in the current cart as possible. This will save you form un-necessary rework and also ensure your contributions stay updated and maintained.

Will be happy to give feedback to your designs as they come.

Eric_Delisle · July 11, 2016, 1:44pm

@rmehta We would be glad to help address known issues, outstanding feature requests and roadmap items.

We are currently building our backlog for our upcoming formal sprints on the shopping cart, session management and payment gateway work.

I did a quick search on the Frappe github and only saw one listed shopping cart issue. Do you have an internal list of known issues to address?

Who is the core team member who has focused on this area? Please introduce them to this thread and let’s start gathering the known issues so we can be sure they are in our planning.

We have succeeded in creating several key tests and will provide more detail about them this week.

rmehta · July 11, 2016, 5:20pm

@Eric_Delisle I meant the “issues” you have listed in your original post.

It is just more maintainable if there is one version of everything you do.

Felipe_Orellana · July 13, 2016, 3:50pm

Hi! First of all congrats on frappe/erpnext guys. The project is awesome and well designed in my eyes.

Some of the issues are engineering related to frappe. Like generating unique guest account per browser session. I think creating this “one page” cart first can help illustrate some of the these issues.

Also, we are going to be using this cart for a client as well as on our site and didn’t want to fork erpnext/frappe to get these things working quickly as we are time constrained. We hope this cart app can be used to pin point places in the framework third party developers may need access to or hooks to implement interesting apps with erpnext.

For example, the way we’ve implemented guest users at the moment was through monkey patching(I know this isn’t ideal, but was necessary to hook in early in the page requests process), as we wanted to generate a temporary user upon a session starting so a user would not be forced to create an account. The existing on_session_creation and boot_session hooks seems to come in late or after login often times which would not let us create our guest account early enough. Bellow is a snipet of the code that creates the guest account right now

def patch_method(obj, method, override):
    """Monkey Patch helper. Will override an object's method with a custome one"""
    orig = getattr(obj, method)
    if hasattr(orig, "monkeypatched") and orig.monkeypatched == override:
        return

    override.patched_method = orig

    def __fn(*args, **kwargs):
        return override(*args, **kwargs)
    __fn.monkeypatched = override

    setattr(obj, method, __fn)

# MONKEY PATCHING HACK in hooks.py

def x_render_page(path):
    try:
        filepath, ext = os.path.splitext(path)
        # we only care about pages
        if ext not in [".js", ".css"]:
            user = frappe.get_user()
            if user.name == "Guest":
                email = random_string(20) + "@guest.local"
                user = frappe.get_doc({
                    "doctype": "User",
                    "email": email,
                    "first_name": "Guest User",
                    "enabled": 1,
                    "new_password": random_string(10),
                    "user_type": "Website User"
                })
                user.flags.ignore_permissions = True
                user.insert()
                frappe.local.login_manager.login_as(user.name)
                frappe.set_user(user.name)

        result = x_render_page.patched_method(path)
    except Exception as ex:
        log(traceback.format_exc())
        raise ex
    return result

    # monkey patch frappe render_page so we may generate guest users
    patch_method(frappe.website.render, "render_page", x_render_page)

Perhaps a signals or event system for some of the internals of frappe would remove the need for monkey patching methods or perhaps we just hooked in the wrong place to begin with. But for now this does the work for us until another method is preferable.

Btw, the same method works for overriding page controllers which we could not do before:

# on hooks.py
from erpnext.templates.pages import cart as erpnext_cart
from customapp.templates.pages import cart as custom_cart
patch_method(erpnext_cart, "get_context", custom_cart.get_context)

Also not ideal, but gets around the current limitation of overriding erpnext page controllers

I know we would not have to resort to this type of work arounds if we forked frappe and submitted our patches to add our fixes but perhaps it illustrates improvements that could be done on the framework and lets us “break” things to see where these improvements may be applied(and identified), such that later we can properly engineer hooks without our current time constraint on our side.

Btw, I managed to destroy my test site’s redis cache by passing the wrong value to frappe.set_user at one point due to the pickling that’s happening with data internally, but that is probably a topic for another thread

kolate_sambhaji · July 14, 2016, 6:14am

Shopping Cart Issues/Feature Request:

Ability to add more than one product image with zoom function in website
Integration with Magento or any open source E-Commerce
Integration with any open source shipping api
Multi Currency
Supplier Portal for selling Items
Theming plug-in like we add Theme in wordpress
Email/SMS notifications on placing order
Tracking status of shiiping and devlivery
some more features from this issue.

My personal opinion about shopping cart development is some funding campaign like quick book integrations or pdf header/footer development will work. We can see lots of potential customer for shopping cart.

Regards,
Sambhaji Kolate,
New Indictrans Technologies Pvt Ltd.

gvyshnya · July 14, 2016, 10:40am

@kolate_sambhaji : thanks a lot for your suggestions as well as pointing to the issue with Shopping Cart re-factoring.

Email notifications and integration with FedEx Shippment API/Delivery tracking are on our radar for the current project. The rest of the features you mentioned are really good contributions to build later as well.

Eric_Delisle · July 14, 2016, 7:58pm

@anand @rmehta

Our tests using the method described here by @Felipe_Orellana seem to be our best option to build these new features rapidly using our “Monkey Patch” method.

We are working to make sure whatever approach we take we DON’T fork ERPN so the Apps we contribute will work now and with future updates.

Does the method described seem like a reasonable approach or are we missing something in the architecture that would be a better approach?

Also, does your roadmap currently include an event system as suggested by Felipe?

rmehta · July 15, 2016, 5:36am

@Felipe_Orellana if you need additional hooks, please feel free to send patches on GitHub

@Eric_Delisle In the long run, my observation is that if you are not making sure your designs are embedded in the app, you always run the risk of things breaking in upgrades.

If what you design is somehow a part of the standard app, it gets maintained by everyone - means better feedback, testing, and every change is verified by the core group.

One of the reasons why ERPNext feels well-designed and modern is that we are always updating the system to be upto standard and this may mean large changes. For example for Version 8, we are already looking at moving to Python3.

Many times we rename core DocTypes, simplify, optimize models / modules so that the system is cleaner and easier to maintain and I don’t think that is going to change in the near future.

So any contribution that is in the erpnext repo will be maintained.

On another note, we have seen the pain users face in ecosystems like Odoo, where apps are not fully supported by the core product. This leads to fragmentation of the ecosystem and the same work is done by multiple partners. In ERPNext we want to build a core product that is complete. So getting more features in the product is the way forward.

Eric_Delisle · July 15, 2016, 11:12am

We agree completely with the process you follow. As I stated from the start we will work to contribute to the community and to the core itself to help not only our clients but everyone.

This is also why we are preparing to attend the conference in October and are committed to supporting the Foundation when your government (like ours) gets around to doing their job without a bribe.

My question about methods and approach were more about:

To address a workflow challenge of 2 conflicts which should naturally resolve over time but cause project management challenges in the short term.

The minor conflict is:

A) Our needs to work diligently and quickly along with our capability to enhance ERPN, but sometimes in ways not yet supported like in session management for UX in the shopping cart process. This is driven by our commitment to our client.

B) Our respect for the time constraints of the core team’s workload and focus your current roadmap, instead of being taken away by our immediate needs (even if we pay for your time). This is driven by our commitment to Frappe and the community.

Our simple TEMPORARY solution was to create our own hooks and event interception through Monkey Patching tools we created.

Our reasoning was this would give us a way to build core functionality faster on our side, test our ideas, iterate, and once we have a solid architecture with few bugs, offer solid Pull Requests based on our work.

AND… We meet our need of demonstrating full, production ready, functionality to our client.

PLUS… If our patch is acceptable but takes your team a long time to merge for testing reasons, priorities, or time constraints (or whatever), we all have the ability to take the appropriate amount of time to add to the core wisely, not hastily.

The RISK of this approach is relatively small to us as our Monkey Patching is designed to have minimal impact on the core and if it changes, our patching will survive upgrades in most cases we believe.

In summary, I hope this discussion helps the community consider ways to work deeply while respecting that we are all working to meet the needs of many different constituents. Priorities often are influenced by factors which conflict with other priorities and finding the best way forward usually only requires good communication, good intention and forethought.

@Felipe_Orellana please share our repo with our Monkey tools so Frappe can see how we are hooking in and perhaps they can see and choose/prioritize some events or hooks they can add to the core in the near future.

Thanks

Felipe_Orellana · July 15, 2016, 3:01pm

@rmehta, @Eric_Delisle

Hi, so the monkey patching business is totally temporary and only there to get our stuff running quickly. I will go over your current hooks system and see about submitting a patch. Though I don’t see a lot of documentation about the current hook system other than looking at current implementation in frappe and erpnext apps. Would you be able to point out where I can find this @rmehta and best practices?

This is where I am keeping random quickly made tools we are using: GitHub - DigiThinkIT/frappe_utils

Btw, on that repo there is an events.py file with an implementation of a very basic observer pattern signals/slots based event class(with syntax sugar) I’ve used in the past. Would something like that be of interest to the team to add for lower/early level hook points?

gvyshnya · July 19, 2016, 11:26am

@rmehta: could you please let us know suggestions/feedback of experts of the core Frappe team, based on the code review of the repository with our quick tools? Thank you

rmehta · July 19, 2016, 5:09pm

@gvyshnya really depends on the use case.

IMO you should use hooks, if the hooks you are looking for do not exist, then send a PR.

monkey patch will be terrible to maintain as time goes.

Eric_Delisle · July 19, 2016, 5:28pm

@Felipe_Orellana provided a suggestion in his post here above along with a suggestion for an event class to handle the hook issue.

He did not want to suggest a PR without some comments from you on this method first. @rmehta could you respond to his suggestion above?

If you like this direction, he will be happy to submit a PR in this direction. If not, maybe point out where he can see more on the current hook system so he can offer a good PR based on it.

rmehta · July 20, 2016, 9:44am

@Eric_Delisle @Felipe_Orellana I still need some light on the use case.

So far there have been numerous apps built on frappe/erpnext (erpnext itself is an extension of frappe) so there are already enough hooks.

Felipe_Orellana · July 20, 2016, 4:02pm

Hey @rmehta

After looking over at how the frappe hooks system is written I realize how to implement and add additional hooks. I will submit a patch for the hook I had in mind.

I was originally confused as how it worked as methods are passed as strings(and later dynamically called) and not actual method references like other system I’ve used.

Since method and value hooks are implemented and store the same way does(or will) the frappe framework have a way of documenting hooks and their method signatures?