Do you recognize this error? (it's getting creepy!)

[MartinHBramwell]

During bench start I get the following error:
Any hints would be greatly appreciated

Is there a bench command, that reruns yarn perhaps, that would get me past it?

19:42:20 web.1            | 127.0.0.1 - - [17/Nov/2020 19:42:20] "GET / HTTP/1.0" 404 -
19:42:31 watch.1          | Rebuilding erpnext.css
19:42:34 watch.1          | Rebuilding marketplace.css
19:42:34 watch.1          | Rebuilding erpnext-web.min.js
19:42:35 watch.1          | Rebuilding erpnext-web.css
19:42:35 watch.1          | Rebuilding marketplace.min.js
19:43:08 watch.1          | 
19:43:08 watch.1          | <--- Last few GCs --->
19:43:08 watch.1          | 
19:43:08 watch.1          | [915281:0x3ae1d10]   139722 ms: Mark-sweep 987.1 (996.0) -> 986.5 (996.0) MB, 1656.4 / 0.0 ms  (average mu = 0.208, current mu = 0.141) allocation failure scavenge might not succeed
19:43:08 watch.1          | [915281:0x3ae1d10]   141759 ms: Mark-sweep 987.1 (996.0) -> 986.9 (996.0) MB, 2027.8 / 0.0 ms  (average mu = 0.105, current mu = 0.005) allocation failure scavenge might not succeed
19:43:08 watch.1          | 
19:43:08 watch.1          | 
19:43:08 watch.1          | <--- JS stacktrace --->
19:43:08 watch.1          | 
19:43:08 watch.1          | ==== JS stack trace =========================================
19:43:08 watch.1          | 
19:43:08 watch.1          |     0: ExitFrame [pc: 0x1409219]
19:43:08 watch.1          |     1: StubFrame [pc: 0x1473478]
19:43:08 watch.1          | Security context: 0x3fed3eec08d1 <JSObject>
19:43:08 watch.1          |     2: new FunctionDeclaration [0x8ff149e2ba9] [/home/erpdev/frappe-bench-DELS/apps/frappe/node_modules/rollup/dist/rollup.js:1] [bytecode=0x19bd77a646c1 offset=0](this=0x34cc6f16e419 <FunctionDeclaration map = 0xd1b9458c19>)
19:43:08 watch.1          |     3: arguments adaptor frame: 3->0
19:43:08 watch.1          |     4: ConstructFrame [pc: 0x138a2b7]
19:43:08 watch.1          |     5: /* anonymous */ [0x3c100...
19:43:08 watch.1          | 
19:43:08 watch.1          | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
19:43:08 watch.1          |  1: 0xa17c40 node::Abort() [/usr/bin/node]
19:43:08 watch.1          |  2: 0xa1804c node::OnFatalError(char const*, char const*) [/usr/bin/node]
19:43:08 watch.1          |  3: 0xb95a7e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
19:43:08 watch.1          |  4: 0xb95df9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
19:43:08 watch.1          |  5: 0xd53075  [/usr/bin/node]
19:43:08 watch.1          |  6: 0xd53706 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/bin/node]
19:43:08 watch.1          |  7: 0xd5ffc5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/bin/node]
19:43:08 watch.1          |  8: 0xd60e75 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/bin/node]
19:43:08 watch.1          |  9: 0xd6392c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
19:43:08 watch.1          | 10: 0xd2a34b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/bin/node]
19:43:08 watch.1          | 11: 0x106c91e v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/bin/node]
19:43:08 watch.1          | 12: 0x1409219  [/usr/bin/node]
19:43:09 watch.1          | Aborted (core dumped)
19:43:09 watch.1          | error Command failed with exit code 134.
19:43:09 watch.1          | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
19:43:10 system           | watch.1 stopped (rc=0)

On a different installation I see that the next to rebuild after marketplace.min.js is erpnext.min.js. How could I fix whatever is defective in there?

19:52:06 watch.1          | Rebuilding erpnext-web.min.js
19:52:06 watch.1          | Rebuilding erpnext-web.css
19:52:06 watch.1          | Rebuilding marketplace.min.js
19:52:15 watch.1          | Rebuilding erpnext.min.js
19:52:19 watch.1          | Rebuilding item-dashboard.min.js

[smino]

@MartinHBramwell

JavaScript heap out of memory

Where I would start. Whats the hosting environment?

[MartinHBramwell]

Self-hosted.
Two separate bench installations in a single user account. One production. One development.
Production is working correctly.
Failure is somewhere in development directory hierarchy.
I installed some customizations to Sales Invoice from an earlier version.
Bench won’t start.

How can I purge all JS and Python object files and rebuild Frappe & ERPnext applications from source?

[MartinHBramwell]

I just tried bench rebuild, but it made no difference.

[smino]

bench update --requirements or bench setup --requirements

If this is being hosted on a virtualization platform, how much memory is allocated to the ERPNext VM’s?

[MartinHBramwell]

I am looking at …

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Hey guys! I was able to find a solution for this problem. Thanks to @smino for guiding me through additional steps I had not considered when trouble shooting my problem. Through online research and @smino 's guide, I found this link: https://stackoverflow.com/questions/53230823/fatal-error-ineffective-mark-compacts-near-heap-limit-allocation-failed-javas On there, they explain that you can add an option to a Node process to expand the memory available by adding the following line to the build…

But I do not know where I am supposed to put:

–max-old-space-size=8192

@smino

If this is being hosted on a virtualization platform, how much memory is allocated to the ERPNext VM’s?

My VPS looks like this:

----------------------------------------------------------------------
 CPU Model             : AMD EPYC 7401P 24-Core Processor
 CPU Cores             : 1
 CPU Frequency         : 1999.996 MHz
 CPU Cache             : 512 KB
 Total Disk            : 39.0 GB (13.0 GB Used)
 Total Mem             : 1987 MB (180 MB Used)
 Total Swap            : 1022 MB (264 MB Used)
 System uptime         : 0 days, 1 hour 25 min
 Load average          : 0.94, 1.45, 1.09
 OS                    : Ubuntu 20.04.1 LTS
 Arch                  : x86_64 (64 Bit)
 Kernel                : 5.4.0-53-generic
 TCP CC                : cubic
 Virtualization        : KVM
 Organization          : AS34971 Prometeus di Daniela Agro
 Location              : Amsterdam / NL
 Region                : North Holland
----------------------------------------------------------------------
 I/O Speed(1st run)    : 465 MB/s
 I/O Speed(2nd run)    : 376 MB/s
 I/O Speed(3rd run)    : 449 MB/s
 Average I/O speed     : 430.0 MB/s
----------------------------------------------------------------------
 Node Name        Upload Speed      Download Speed      Latency     
 Speedtest.net    6848.72 Mbps      6581.47 Mbps        3.00 ms     
 Shanghai   CT    81.71 Mbps        2012.83 Mbps        237.45 ms   
 Shanghai   CU    394.20 Mbps       788.22 Mbps         312.16 ms   
 Guangzhou  CT    353.42 Mbps       1287.65 Mbps        216.98 ms   
 Shenzhen   CM    213.50 Mbps       1910.78 Mbps        285.01 ms   
 Singapore  SG    372.53 Mbps       1731.56 Mbps        237.62 ms   

I’ve been using this VPS successfully for weeks. The problem just suddenly appeared today, while I was attempting to install a customization.

[MartinHBramwell]

So I’m back in action after editing ${HOME}/${MY_BENCH}/apps/frappe/package.json like this:

                    :                       :                       :                       :                       :   
                    :                       :                       :                       :                       :   
  "scripts": {
    "build": "node --max_old_space_size=1536  rollup/build.js",
    "production": "FRAPPE_ENV=production node --max_old_space_size=1536 rollup/build.js",
    "watch": "node --max_old_space_size=1536 rollup/watch.js",
    "snyk-protect": "snyk protect",
    "prepare": "yarn run snyk-protect"
  },
                    :                       :                       :                       :                       :   
                    :                       :                       :                       :                       :   

[revant_one]

if you don’t wish to change the package.json, set Environment variable for NODE_OPTIONS=--max-old-space-size=1536

refer Command-line API | Node.js v19.3.0 Documentation

[sonyvec]

Hi, Martin!

That is what I ended up doing with my package.json file. I tried other options, but after several tests, keeping the file with the --max_old_space_size=xxxx added was the best solution. I also added a SWAP file to my Ubuntu installation and added 4GB, this also helped with smoother operation overall.

I’ll edit my solution on the other issue that way it’ll help other users who may be having the same issue and I apologize if it made you have to find the file yourself, I know it can be tricky to find this kinds of configurations and settings through the apps.

[MartinHBramwell]

Aha! Yes, that is better.

[MartinHBramwell]

Wouldn’t it be cool if there was a ERPNext Administrator’s Wiki with a supported performance analysis toolkit that members could use to report their optimization efforts organized by hosting environment: base metal, VPS, Amazon, etc and then in subcategories: operating system, # of CPUs, amount of memory, # of users, etc.