[discussion] Immutable Ledger

ERPNext Accounting
41

By “More flexible costing option” I meant from a programmers point of view. Many a times businesses have to change their process to match what the programmers have created and accounting software in most cases are designed initially with the accounting principles in mind but when trying to apply these in real world cases they dont always match the way business would like to actually process transactions. I cannot say for all business but in our case that was the situation. We spend over 2 years trying to just get our current system to work the way we wanted and even now its not perfect but close enough.

So as a programmer first I think using a “actual or lot” costing model will lead to better “trust” of the system.

With the “actual cost” model we are able to help our finance team to improve monitoring the system for odd transactions, our sales team with acurate commission numbers, our management team with actual profit and loss reports without any additional programming and queries to get the data they need adjusted or corrected to get to that end. Its just more simpler in my opinion. Please dont mis-understand me, im not saying ERPNext does not do this or other systems dont.

Ive been trying for 2 years to try to get our business to adopt ERPNext and my struggles were always with arguments around the costing model. I guess I can start a bounty to add this costing into ERPNext but first I have to get our FD to see the light, so I have been trying to save up my own funds to eventually do this - or learn to contribute the code which is what im trying to do as well for sometime (Costing is complication and I dont want to break something).

Like @rmehta was saying, ERPNext has this issue currently and like I learnt with our current system - its only after many months or years after someone does some digging that you find things are off track and then it makes it harder to correct. We started from scratch re-implementing (a very expensive excersise) our system 3 years back and now we have ‘pulse’ reports giving us a view of anything that goes off the track - all of which I want to add to ERPNext once my team of dev are on track with how.

2 Likes
42

The concept of open and closed periods is standard in the ERPs I have used since the 1980s. If ERPNext doesn’t have this feature then it would be a handicap in some countries and/or certain organizations

2 Likes
43

ERPNext does support closing of books and locking transactions for closed periods.

Thanks

Jay

2 Likes
44

I think it is okay on the new proposal. But can’t we have a toggle that says Post Valuation on Transaction date rates (Slower and takes system resources) or Post Valuation based on Today’s rates (Faster, but your Valuation rates may not be exact, though the overall impact is likely to be minimal). Certain service providers (like Frappe Cloud) may not want to offer the Valuation Rates based on Transaction Date option at all to minimize impact to other multitenanted clients.

Let me check with a few of my clients of the proposed approach is acceptable.

So please don’t even begin coding this till the community okays this approach.

This too big a change for us to implement without a broad consensus.

Thanks

Jay

2 Likes
45

I think we have come long way from the argument of system performance. Correct me if I am wrong, but this is not about back dated entries hogging the CPU time. Editing posting date is a bad accounting practice and frowned upon by some users/auditors. In addition, this is difficult to implement and the current implementation has a defect which we may not be able to fix. On the other hand, current implementation that allows changing the posting date is very useful for many of SME customers. Providing both options and allowing customers to chose either one of them is more work for developers + may not be acceptable for those looking for in-built immutability of records.
In any case, I think the onus is on SME customers who think the new approach is going to make their life very difficult. So please collate those usecases and put it in the spreadsheet I shared earlier. If there are enough usecases affecting SME customers, certainly we should revisit the new approach.

5 Likes
46

I have been reading this whole discussion and trying to assimilate the changes that are being proposed.

First off all the reason that has been put forward is that it is computationally expensive. Ok it may be but what is the analysis. As per my understanding of the examples that have been put forward it is O(n) but then it is a simple example just to explain.

Whereas I understand, appreciate and welcome the fact that only forward entries should be allowed, the reason being provided is not understandable. Moreover there is a high probability of breaking a lot of things in accounting which makes we worried.

As I am more from technical background I wanted to understand from @nabinhait’s example the store does not reflect the real position even after putting in the correct entry. Is that acceptable. Doesn’t change in the value of the store with two dates, posting dates and trans date create problem from an audit point of view.

Further we have to understand a lot of sme’s and shops have employees that multitask. That means they can be handling both stores and accounting. This is where the chances of mistakes are maximum. The system should be such that is there is a genuine error it can be corrected without an objection from the audit. I would suggest that we should study what other packages are doing and benefits and problems of those approach before deciding on an approach.

Another problem is that we donot have proforma invoice in ERP. Hence it is not possible to send the proforma to the customer to check and make any changes before making the final invoice. Currently the option suggested is to make a quotation with new series and send it to customer as proforma. If we want to make the ledgers immutable, then we will need a better mechanism of generating proforma invoice in order to reduce the chances of error in the final invoice.

-Amit

2 Likes
47

I’m worried by some of the proposed changes. We seem to be forcing one point of view of how we think the system should be operated on the user.

When it comes to accounting documents, ERP systems I’ve been exposed to track the following 3 dates:

Posting Date: The date on which the transaction has an impact on the General Ledger
Document Date: The memorandum date on which a document was physically received, generated or sent. Has no impact on the General Ledger
Entry Date: The date on which the document was entered into the sytem (This date should not be editable)

My understanding up till now is that Posting Date in ERPNext is the same as Posting Date I’ve described above. I believe it should remain so.

Document Date and Entry Date are not really represented uniformly in ERPNext.

For me the proposed Transaction Date should be equivalent to the Document Date as defined above.

Entry date should be the date of the booking in the system. It should not be editable and should represent the time-stamp of when the transaction was entered.

As I’ve said in the past, there are many legitimate reasons why retrospective payments can be made in Finance Systems. It’s not our place to enforce constraints on what can and cannot be done by the user. Instead, we should satisfy ourselves with being able to provide an accurate record of when and in what sequence any transaction happened. Therefore, if an accountant finds it necessary to open a closed period do make some adjustments, he should be free to do so. But it should be clear that the time-stamp/Entry Date will show that he made this change several months after the period was closed. That’s an auditors job to pick out.

Using the definitions above, one would expect the following treatment in the scenarios below:

Reversal in the current period of document posted in the currently open period:

  1. The original document is left in place (i.e. not deleted)
  2. A reversal document with the opposite accounting impact is posted
  3. The new reversal document retains the same posting date as the original document. If we decide to add a document date field, this document will also retain the same document date as the original document. However, the transaction date will be the exact date/time when the reversal was posted. Also, the General Ledger balances will be updated with the impact of the new document
  4. The original document is updated with a link to the reversal document and the reversal document is updated with a link to the original document (perhaps we should add two new fields to journals called reversed_by and reversal_document_for).

Reversal in the current period of document posted in a closed period

  1. The original document is left in place (i.e. not deleted)
  2. A reversal document with the opposite accounting impact is posted
  3. The new reversal document will have to be updated with a posting date in the currently open period. If we decide to add a document date field, this document will also retain the same document date as the original document. However, the transaction date will be the exact date/time when the reversal was posted. Also, the General Ledger balances will be updated with the impact of the new document
  4. The original document is updated with a link to the reversal document and the reversal document is updated with a link to the original document (perhaps we should add two new fields to journals called reversed_by and reversal_document_for).

Opening of a closed period to reverse a document posted in that period

  1. The original document is left in place (i.e. not deleted)
  2. A reversal document with the opposite accounting impact is posted
  3. The new reversal document retains the same posting date as the original document. If we decide to add a document date field, this document will also retain the same document date as the original document. However, the transaction date will be the exact date/time when the reversal was posted. Also, the General Ledger balances will be updated with the impact of the new document. The impact of this on future balances can be managed by maintaining a Balance Carry Forward value along with a Period Balance Value for each account in each month. It means that all that needs to be done in the case of the retrospective posting/reversal is to update the period balance in the month where the change occurred and the balance carry forward for the following month.
  4. The original document is updated with a link to the reversal document and the reversal document is updated with a link to the original document (perhaps we should add two new fields to journals called reversed_by and reversal_document_for).

For Retrospective Stock Ledger Entries

  1. Both Moving Average Price and FIFO valuation are affected by the timing/sequence of the events.
  2. The Transaction Date should represent the sequence of the events within a particular period.
  3. We need to be able to open an close a period from the Stock perspective the same way that we can open and close from the Finance Perspective. In the bigger ERPs you are usually allowed to open period P-1 for Stock postings
  4. If we adopt the point above and open a past period from the Stock perspective and then make a change such as cancelling a stock document or entering a forgotten stock posting such a posting should be accepted.
  5. The finance impact of the posting in the step above should be accepted, the FIFO stack or Moving Average Price should be recalculated. However, accounting documents linked to all stock-relevant movements following the retrospective posting should not be cancelled/reposted.
  6. The approach above may lead to some inconsistencies between the Stock and Accounting ledger. This should be able to be explained by looking at the transactions sorted by Entry Date, Transaction Date and Posting Date in that order.

Edit
We might also want to consider the possibility that every event that has an effect on the General Ledger (e.g. Stock Movement, Purchase/Sales Invoice etc) should generate a Journal Entry. I.e. these source documents should not update the general ledger directly but only through a Journal Entry (which will then be linked to the source document).

4 Likes
48

This shouldn’t be an issue of what users find flexible but about what accounting standards and accounting rules (US) say. We should use this as the contract which the implementation should satisfy.

Traditionally, the general ledger is mostly a no-delete, append-only document. That is why accountants have terms like reversal, contra entry, etc. That is why I also support that General Ledger should not allow deletion of entries. Let’s also not forget about accounting periods which brings about the concepts of Prior Period Errors, Change in Accounting Policies, Change in Accounting Estimates (IAS 8) and then Errors After the Reporting Period (IAS 10).

Under IAS 8 and IAS 10, there are circumstances where you have to make corrections retrospectively. An example is when you forget to input say a Purchase Invoice that was received in say January and then remembered in February or you made a wrong depreciation entry. That means it should be possible to back date entries in the GL including when it’s in a closed period. Obviously, after making a retrospective change, closing balance for the said period and the next opening balance would change.

Actually, this is the correct thing to do. We don’t have any choice but to find a design that makes this as efficient as possible.

7 Likes
49

I have to say that I don’t understand why anyone is arguing against an immutable ledger at the database level. Anyone who runs their business without a full audit history is looking a disaster and/or severe penalties, no matter how much they trust themselves or their bookkeepers. One fat-finger edit can destroy your accounting, and necessitate restoring backups to verify against, if you even have them.

The implementation detail of updating reports to add up the edit entries should never be a reason to outright reject something that actually protects every business owner.

4 Likes
50

Hi @tundebabzy

Trust you’re doing great. Seems like you’re arguing both sides? In the first part of your comment it appears you’re all for immutability and towards the end it seems you’re against it. Could you please clarify?

Thanks

51

I’m in for ‘immutability’ as in not being able to delete any record from the GL. Like I said earlier, the GL is supposed to be an append-only document so records should only be added at the bottom including records that reverse previous entries. I know that I said users should be able to back date entries into earlier periods and that sounds like I supporting a system that is not append-only. If you think of the GL as a combination of 12 smaller GLs, the append-only requirement will not be hard to imagine.

So to be clear, I’m in for:

  • No delete from the GL. This should not be a subject of argument as @Joshua_Bowman and others have rightly pointed out
  • Append-only to the GL
  • Back dated entries as needed even though it might be expensive and slow. Accounting standards require it. There will be changes and that is why IAS requires disclosure of such.
6 Likes
52

This is really clear and I like it.

2 Likes
53

Hi @tundebabzy

The above point is where I really require clarification. Is the GL still immutable if you allow back dated entries? What would then be the point of the whole exercise? Under what conditions would you bypass the immutability of the Ledger?

Just in case you’re wondering, I am definitely in support of immutability because I know it is the recommended standard practice in accounting and honestly, it significantly improves the reliability of the system not to talk of the bug (as disclosed by @nabinhait) which leaves our Stock Ledger perpetually inaccurate when using FIFO

Having said all this, I’m however a bit concerned, just like several others, about the impact this would have on the end-users. If I correctly understand the earlier explanations from the ERPNext team, we would still have the reports showing ‘amended’ transactions (similar to what we have now) when we run them using the ‘Transaction Date’. Using the Posting Date however would show a report of entries according to their order of creation

At this point, I think the naming convention requires some clarifications too as it might be causing some confusion. Since Posting Date is generally understood as the date when the transactions impact the Ledgers, I think we should keep it this way instead of using the term Transaction Date. For the date when the entry is made, we can consider keeping this simply as ‘Entry Date’

So for every transaction, we have:

Posting Date: Date when the Ledgers are impacted (Can be amended as required)
Entry Date: Date when the entry is Created (Can not be changed)

Therefore, immutable ledgers (from a user’s perspective) simply means that the Ledgers will now be generated using Entry Date rather than Posting Date in order to avoid frequent recalculations and other compliance issues. Also, Cancelling a document will work just like doing a return. There will however be the ability for us to still view reports according to Posting Date (as we have it currently)

If all the foregoing is correct, then there’s not much impact on the end-user as long as the logic for validating transactions is also based on Posting Date and not Entry Date

@nabinhait could you please confirm the above statement regarding logic for validation of transactions?

Thanks

EDIT: The more I look at this, the clearer it becomes (I think). Going from the very definition of Posting Date above, I now understand better why the ERPNext team chose their initial naming convention. If the Posting Date is actually agreed as the date when the transaction impacts the ledger then it means Posting Date = Entry Date because that’s when it actually impacts the ledgers under the new proposed system

The Transaction Date would then refer to the date when the actual transaction took place. If I’m getting this correctly then the initial use of Posting Date and Transaction Date by the ERPNext team is actually the correct way to go under the new proposed system!

2 Likes
54

Yes, an immutable ledger just means that entries that have already been made into the General Ledger doctype cannot be amended - the only way to make a change is to append a new line

The only date that should be relevant to the operation of the general ledger is the posting date.

Exactly. Let the Transaction Date be the memorandum date of the source document. Better still, let’s call it Document Date for more clarity.

1 Like
55

Hi all,

Based on my current understanding of the proposed new system (as reflected in my edited post above), here are the critical points to note and my submission on the matter

There are 2 important aspects of the changes being proposed:

  1. Transaction Documents can no longer be outrightly Cancelled/Deleted after Submission. Cancelling a transaction document will now be handled similar to making a return

  2. The order of Posting in the ledgers will now be according to the date when the entry is created. This means you can only back date the Transaction Date but NOT the Posting Date

For Point 1 above, I think most people agree it helps for the reliability of the system and compliance to standards

For Point 2, I think this will not have much impact on end-users and will help improve the general accuracy of the system provided the 2 conditions below are satisfied:

a) We are able to generate reports based on Transaction Date (This will reflect the impact of corrective entries in their actual and expected order similar to what we have currently)

b) The logic for validation of transaction documents is also based on Transaction Date

The part about reports has already been confirmed and we await confirmation on the validation logic from @nabinhait or any other member of the ERPNext team

Thanks

3 Likes
56

@wale
You are on point.

Now that we have more or less finalized the design contract, we now need input on the design implementation

57

You should be able to back date both the posting date (in cases where you open an already closed accounting period for example) and transaction date. What can’t be backdated is the entry date which is a record of when the posting was performed (this is actually the creation date of the doctype). All three dates are required at a minimum.

1 Like
58

Dears,
I guess the deletion ability of Journal Entries is a very good feature. Since most SMEs do not have in-house accountants, this ability allows them to do entries without consulting an accountant and keep their records up to date. Whether a closed period is editable or not and if the company tampers with past data is an ethical issue. We cannot stop them from turning accounting tricks by any system.
If we take away this feature, the SMEs would revert to Excel or else to keep their transactions and ErpNext would just simply be another accounting driven system. This generally kills ERPs being used on the spot for transactions and calls for separate solutions being applied.
With the current flow of accounting entries, if you don’t allow negative stocks, all entries need to be made with their actual dates so that stock is valid at all times then the faulty documents are deleted to reflect the correct state of the system.
Let’s say that the SME uses ErpNext as their only system to run the company and does not employ an in-house accountant. This can be the case for many reason. Either their process is repetitive or the value chain produces simple outputs. For all the postings after the accountant’s evaluation if there are records to be corrected for currency changes, posting accounts approach even account names there will be significant changes between the records that the accountant is keeping and the ones at the ErpNext for the SME. This will lead to unmatching ledgers between accountant’s system and the SME. This is a higher risk. This will render SME’s ErpNext system invalid compared to the accountant’s.
Accounting correctness is an ethics issue not a software issue. We should be very careful in applying this new approach.

2 Likes
59

Hmmm…

As highlighted in my last post, a major part of this new proposed system is hinged on the premise that the Posting Date MUST be in chronological order (i.e. Entry Date). Since this is the case, allowing Posting Date to be back dated under any circumstance will defeat the purpose of avoiding forward recalculations. I therefore wonder how the issue of reopening closed accounting periods will be handled. Will there be some Tool/Feature that allows a document to have a Posting Date in a different accounting period from the Transaction Date?

This would then mean that a report for an accounting period, say 2017, could potentially contain transactions with Posting Date in 2018 (made by users with appropriate permissions) but the Transaction Date must be 2017. It also means that all such transactions will not be considered while drawing reports for 2018. Basically, the main validation for what transactions fall into an accounting period will be the Transaction Date and not the Posting Date

I think it’s also important to note here that in the ERPNext context, there will be a difference between Posting/Entry Date and Creation Date

Creation Date: The date when the document is Created/Saved on the system
Posting/Entry Date: The date when the document is Submitted and ledgers are actually impacted

Kind regards,

1 Like
60

Could we borrow a leaf from the banking sector and call the “Posting Date” Value Date?