[Solved] What's the right way to version control my customizations?

MartinHBramwell · May 15, 2020, 8:17pm

Update: What’s described in this post does work, but it is the wrong thing to do. See reply below.

I have two virtual machines running identical freshly installed ERPNext V12 copies.
Call them “dev” and “stage”.

On dev

I created a site "mySite" and I created an app "my_app" installed in the site.

Within the directory tree of "my_app", I created a directory called customizations …

├── LICENSE
├── README.md
├── __pycache__
│   └── setup.cpython-36.pyc
├── customizations
│   ├── core
│   │   └── custom
│   └── README.md

In the frappe application directory I created a symbolic link called "custom" from the "core" subdirectory to my "customizations" subdirectory:

├── erpnext
└── frappe
│   ├── frappe
│   │   ├── __init__.py
│   │   ├── core
│   │   │   ├── README.md
│   │   │   ├── __init__.py
│   │   │   ├── __pycache__
│   │   │   ├── custom -> /home/erpdev/frappe-bench/apps/my_app/customizations/core/custom
│   │   │   ├── doctype

Without doing any customizations at all, I used the button [ Export Customizations ].
So my app directory looked like this:

├── LICENSE
├── README.md
├── __pycache__
│   └── setup.cpython-36.pyc
├── customizations
│   ├── core
│   │   └── custom
│   │       ├── sales_invoice.json
│   │       └── sales_invoice_item.json
│   └── README.md

Next, I copied the custom subdirectory and called it patches

├── customizations
│   ├── core
│   │   └── custom
│   │       ├── sales_invoice.json
│   │       └── sales_invoice_item.json
│   ├── patches
│   │   ├── core
│   │   │   └── custom
│   │   │       ├── sales_invoice.json
│   │   │       └── sales_invoice_item.json
│   └── README.md

I the Sales Invoice page I selected [ customize ] and added a new section and a few fields to the Sales Invoice default doctype and ran [ Export Customizations ] again.

Next, I used the Linux command diff to create patch files and then deleted all the others:

├── customizations
│   ├── core
│   │   └── custom
│   ├── patches
│   │   ├── core
│   │   │   └── custom
│   │   │       ├── sales_invoice.patch
│   │   │       └── sales_invoice_item.patch
│   └── README.md

cd customizations/patches/core/custom;
diff -u sales_invoice.json ../../../core/custom/sales_invoice.json > sales_invoice.patch

I committed and pushed my app to GitHub

On stage

In the apps directory I cloned my GitHub repo.
I ran: bench --site mySite install-app my_app
I created a "custom" symlink in the same place in the frappe directory hierarchy as before
I ran [Export Customizations] as before.
I used the Linux command patch to apply the patches to the unaltered exported files
I ran :

bench restart
bench --site mySite clear-cache
bench --site mySite migrate

I now have the “dev” customizations working on “stage”.

Questions

So I understand that even though it works, all of that is what I am not supposed to do!!!

Why will I regret doing it this way?
Where can I find similar step by step instructions about how to do it the right way??

MartinHBramwell · May 18, 2020, 3:08pm

Right, well I finally figured this out.

There’s a vitally important yet scantly documented command :

bench --site mySite export-fixtures

I’ll explain and demonstrate how it actually works.

I’ll use a virtual machine with snapshotting so I can do some work, then revert back to a snapshot, and continue. So I’ll :

Create a site
Take a snapshot
Create an app, add it to version control.
Install app in site.
Customize the site
Export the customizations
Add the exported customizations to app using export-fixtures
Commit the app changes to version control
Revert vm to the earlier snapshot
Restore the app changes from version control
migrate the bench to install all the changes

Create a site

With the bench already running in a separate terminal window…

export NEW_SITE=example_site;
export NEW_APP=example_app;
export REPO="git@github.com:yourself.yourorg/fixtures_example.git";

bench new-site --mariadb-root-password ${MYPWD} --admin-password ${ADMPWD}  ${NEW_SITE}

Take a Snapshot

This depends on which hypervisor you use. Alternatively you could work with two separate machine.

Create an app and add it to version control.

Go to your GitHub account and create a new repository, like

https://github.com:yourself.yourorg/fixtures_example

Then …

bench new-app ${NEW_APP};      # answer the prompts for app meta-data

# step into apps directory
pushd apps;

# pull the initialized GitHub repository into a temporary directory
git clone ${REPO} temp_dir;

# move the repo's files into our app's directory.
mv temp_dir/* ${NEW_APP};
mv temp_dir/.g* ${NEW_APP};

# delete the temporary directory.
rm -fr temp_dir;

# step into example_app directory
pushd ${NEW_APP};

# add and commit all files to local repository
git add -A
git commit -am "Adds 'example_app' initial files";

# push local repo to GitHub
git push

# step back out to bench directory
popd;
popd;

Install app in site.

bench --site ${NEW_SITE} install-app ${NEW_APP}

Not sure why. Not sure if it indicates a problem. You probably need to start bench again at this point

bench start

Flush the toilet…

bench --site ${NEW_SITE} clear-cache
bench --site ${NEW_SITE} migrate

Customize the site

A simplest possible change: the invoice header has no section title, I add one, “Customer Details”.

Sales Invoice before

Customize Form before

Customize Form after

Sales Invoice after

The next step is to 'export customizations`, but first notice:

erpdev@fossa:~/frappe-bench$ ls -la apps/frappe/frappe/core/custom
ls: cannot access 'apps/frappe/frappe/core/custom': No such file or directory
erpdev@fossa:~/frappe-bench$

Export the customizations

After pressing press the button [ Export Customizations ], choosing core module and sending, you’ll see:

erpdev@fossa:~/frappe-bench$ ll apps/frappe/frappe/core/custom
total 20
drwxr-xr-x 2 erpdev erpdev 4096 May 17 11:38 ./
drwxrwxr-x 8 erpdev erpdev 4096 May 17 11:38 ../
-rw-r--r-- 1 erpdev erpdev 7873 May 17 11:38 sales_invoice.json
-rw-r--r-- 1 erpdev erpdev  720 May 17 11:38 sales_invoice_item.json
erpdev@fossa:~/frappe-bench$

The file sales_invoice.json looks like this:

erpdev@fossa:~/frappe-bench$ pushd apps/frappe/frappe/core/custom
erpdev@fossa:~/frappe-bench/apps/frappe/frappe/core/custom$ head -n 35 sales_invoice.json
{
 "custom_fields": [],
 "custom_perms": [],
 "doctype": "Sales Invoice",
 "property_setters": [
  {
   "_assign": null,
   "_comments": null,
     :          :         :
   "doc_type": "Sales Invoice",
     :          :         :
   "field_name": "customer_section",
     :          :         :
   "property": "label",
   "property_type": "Data",
   "value": "Customer Details"
  },
  {
   "_assign": null,
   "_comments": null,
     :          :         :

Things to note:

We can see that the change is not, in fact, a custom_field, but a property_setter.
This all happens in frappe app directory. Neither example_site nor example_app are affected yet.

Running git status shows that, so far, there have been no changes in our app :

erpdev@fossa:~/frappe-bench/apps/example_app$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
erpdev@fossa:~/frappe-bench/apps/example_app$

Add the exported customizations to app using `export-fixtures`

The next two necessary tasks are:

indicate in our application configuration which types of customizations we want to incorporate.
tell frappe to provide them.

So:
#1. Configuration:

Edit hooks.py …

erpdev@fossa:~/frappe-bench$ nano ./apps/example_app/example_app/hooks.py

… from this …

     :          :         :
app_license = "MIT"

# Includes in <head>
# ------------------
     :          :         :

… to this …

     :          :         :
app_license = "MIT"

fixtures = ["Property Setter"]

# Includes in <head>
# ------------------
     :          :         :

Things to note:

The docs only mention "Custom Field" as a possible “fixture”. Here we use "Property Setter".
According to @aakvatech in this post and other things I’ve found, the fixtures array can include the following, but I have no idea if the list is complete or correct:

"DocType”
“Custom Field”
“Custom Form”
“Custom Script”
“Property Setter”
“Print Format”

There are filtering possibilities on each of those types, but I found no documentation about how one does it. This thread covers a lot of it, but I got an"unknown DocType" error when I tried to filter the (large!) list of property setters.
Everything so far has happened in the frappe app directory. Neither example_site nor example_app have been affected yet.

#2. Command:

The following lines show the export-fixtures command and it’s effect on our git managed directory.

erpdev@fossa:~/frappe-bench$ bench --site ${NEW_SITE} export-fixtures
Exporting Property Setter app example_app filters None
erpdev@fossa:~/frappe-bench$ pushd apps/${NEW_APP}
erpdev@fossa:~/frappe-bench$
erpdev@fossa:~/frappe-bench/apps/example_app$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   example_app/hooks.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)
  example_app/fixtures/

no changes added to commit (use "git add" and/or "git commit -a")
erpdev@fossa:~/frappe-bench/apps/example_app$

Commit the app changes to version control

So now we need to commit the example_app/hooks.py, but we also need to add and then commit the newly generated example_app/directory.

erpdev@fossa:~/frappe-bench/apps/example_app$ git add example_app/fixtures
erpdev@fossa:~/frappe-bench/apps/example_app$ git commit -am "Adds Sales Invoice Customizations"
[master 3030347] Adds Sales Invoice Customizations
 1 file changed, 2 insertions(+)
erpdev@fossa:~/frappe-bench/apps/example_app$
erpdev@fossa:~/frappe-bench/apps/example_app$ git push
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
     :          :         : 
     :          :         : 
To github.com:yourself.yourorg/fixtures_example.git
   b7a0b01..3030347  master -> master
erpdev@fossa:~/frappe-bench/apps/example_app$

Revert vm to the earlier snapshot

With the virtual machine reverted, or moving over to an alternative machine, it’s time to see if the app has survived detachment and reinstallation

Restore the app changes from version control

erpdev@fossa:~/frappe-bench$ export NEW_SITE=example_site;
erpdev@fossa:~/frappe-bench$ export NEW_APP=example_app;
erpdev@fossa:~/frappe-bench$ export REPO="git@github.com:yourself.yourorg/fixtures_example.git";
erpdev@fossa:~/frappe-bench$ 
erpdev@fossa:~/frappe-bench$ bench get-app  ${REPO}
INFO:bench.app:Getting app fixtures_example
$ git clone git@github.com:yourself.yourorg/fixtures_example.git --branch try2 --depth 1 --origin upstream
Cloning into 'fixtures_example'...
remote: Enumerating objects: 21, done.
     :          :         : 
     :          :         : 
✔ Built js/libs.min.js
Done in 1.05s.
erpdev@fossa:~/frappe-bench$ 
erpdev@fossa:~/frappe-bench$ bench --site ${NEW_SITE} install-app ${NEW_APP}

Installing example_app...
erpdev@fossa:~/frappe-bench$

Not sure why. Not sure if it indicates a problem. You probably need to start bench again at this point

bench start

`migrate` the bench to install all the changes

erpdev@fossa:~/frappe-bench$ bench --site ${NEW_SITE} clear-cache
erpdev@fossa:~/frappe-bench$ 
erpdev@fossa:~/frappe-bench$ bench --site ${NEW_SITE} migrate
Migrating example_site
Updating DocTypes for frappe        : [========================================]
Updating DocTypes for erpnext       : [========================================]
Updating customizations for Address
Generating Website Theme Files...
Compiling Python Files...
erpdev@fossa:~/frappe-bench$

Done

We are back to where we were, with “Customer Details” correctly added as the title of the invoice header section …

Shinzuco · April 3, 2021, 10:12pm

Nice, functionally is it different from

Customising the form
Add “custom fields” to hooks.py and bench export-fixtures?

MartinHBramwell · April 4, 2021, 12:12am

Use those wherever possible to avoid having to regenerate patches when the altered core files change.

Shinzuco · April 5, 2021, 10:58am

Makes sense, but then what is the advantage of going through the property setter method?

MartinHBramwell · April 5, 2021, 11:56am

Sorry, I think I misunderstood your question. I’ve been working on other things and so it’s a while since I messed with customizations, and assumed you were asking about altering ERPNext & Frappe *.py and *.js files.

In my second post I state:

Things to note:

The docs only mention "Custom Field" as one of several possible “fixtures”. Here we use "Property Setter" .

According to aakvatech in this … … the fixtures array can include the following,

"DocType”

“Custom Field”

“Custom Form”

“Custom Script”

“Property Setter”

“Print Format”