Frappe V8.0.20 & later: socketio error

@clarkej

yes tried that, bench clear-cache & bench clear-website-cache.

Earlier checked in chrome after clearing cache, now even checked in edge browser & internet explorer similar error:

SCRIPT7002: XMLHttpRequest: Network Error 0x2ee7, Could not complete the operation due to error 00002ee7.

SCRIPT7002 points to:

Is there any way I can upgrade to 8.0.19 from 7.2.13 instead of current version of frappe?

‘Is there any way I can upgrade to 8.0.19 from 7.2.13 instead of current version of frappe?’

Yes but why not exhaust better options first?

So you are on Windows! Here are 4 things to try

Hey @clarkej Thanks a lot for all the help

I tried: Google Chrome’s DNS Prefetching: Settings > Show Advanced Settings > Privacy and disable the option Predict network actions to improve page load performance.

But there’s no change in error, I don’t think this is browser related issue o/w I wouldn’t been able to replicate in other browser (eg edge/IE).

Even though the issue has not been reported, but I have gut feeling there is some fix required in frappe v 8.0.20 to fix this issue. Multiple instances giving same error doesn’t happen that often unless there a bug.

Well your browsers all point to a network or dns issue and run on Windows, yes?

So these makes sense -
restart browser
ifconfig /flushdns
restart Windows

Test access on Mac or Linux desktop to eliminate Windows

Exploring the commits since v8.0.20 or to return to .19 might narrow down the problem commit for an idea of the problem source.

So far still no root cause smoking gun you just have symptoms

Tried restarting PC event tried on 2 other PCs, getting same error.

Still not able to identify root cause, but tried update on one more instance from frappe 8.0.18 to 8.0.23, getting same error in browser console

Everything seems to be working fine apart from this error, so its easy to be missed out unless one happens to check the browser console, this might be the reason its not been reported.

This can also be specific to Ubuntu server, and may not be getting replicated on CentOS. I would request core team to check if they are able to replicate the error.

Can anyone help, still facing same issue with multiple Ubuntu instances.

Issue occurs after updating from frappe 8.0.19 or older to current.

Seems like a config error, Try setting up your config files again

bench setup production

and restart your nginx and supervisor

@rmehta Thanks for reply

Yes it looks like config error, tried bench setup production & restarted nginx & supervisor. Rebooted server as well.

Even manually compared config files of other instances with new, there is no difference in there.

Surprising thing is, I have tried updating 5 different instances all has os ubuntu 14/16, current version frappe version between 7.2.xx to 8.0.19 & in all after update getting same error. But no one else has reported it.

Can you check if it can be replicated at your end on Ubuntu.

Error log has repeated errors as below:

2017/04/23 09:22:54 [error] 1482#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 203.88.159.32, server: site1.local, request: “GET /socket.io/?EIO=3&transport=polling&t=1492939370452-54 HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=1492939370452-54”, host: “demo.finbyz.in”, referrer: “http://demo.finbyz.in/desk

Edit: Output of bench setup production

supervisor.conf already exists and this will overwrite it. Do you want to continue? [y/N]: y
Port configuration list:

Site site1.local assigned port: 80
nginx.conf already exists and this will overwrite it. Do you want to continue? [y/N]: y
INFO:bench.utils:sudo /usr/bin/supervisorctl reread
No config updates to processes
INFO:bench.utils:sudo /usr/bin/supervisorctl update
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
INFO:bench.utils:sudo service nginx reload

  • Reloading nginx configuration nginx

Your ‘connection refused’ may be an nginx config issue?

In my nginx.conf redis listens on port 9000 for local requests -

upstream frappe-bench-socketio-server {
server 127.0.0.1:9000 fail_timeout=0;
}

frappe@erpnext:~/frappe-bench/logs$ nc -v localhost 9000
nc: connect to localhost port 9000 (tcp) failed: Connection refused
Connection to localhost 9000 port [tcp/*] succeeded!

Here are some config problem and fix clues

my nginx.conf is same as yours.

nc -v localhost 9000 output is
Connection to localhost 9000 port [tcp/*] succeeded!

I tried fresh install of ERPNEXT on ubuntu, it doesn’t have any errors.

I copied full config folder from new install to instance where I was getting error, run bench migrate & even rebooted server, still no success. Same error is still there

Say run this check:

frappe@erpnext:~/frappe-bench/config$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

In /var/log/nginx/error.log compare a working instance with one that does not.

Confirm the nginx startup has no errors. Here is an example of a client connection:

(In my case the web browser client runs on 192.168.0.104 that also hosts a kvm guest 192.168.0.105 that runs erpnext.vm - )

2017/04/26 11:38:28 [debug] 1368#0: *6508 http init upstream, client timer: 0
2017/04/26 11:38:28 [debug] 1368#0: *6508 epoll add event: fd:3 op:3 ev:80000005
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "X-Forwarded-For: "
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script var: “192.168.0.104”
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "X-Forwarded-Proto: "
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script var: “http”
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "X-Frappe-Site-Name: erpnext.vm
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "Host: "
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script var: “192.168.0.205”
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "X-Use-X-Accel-Redirect: True
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: “Connection: close
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: “”
2017/04/26 11:38:28 [debug] 1368#0: 6508 http script copy: “”
2017/04/26 11:38:28 [debug] 1368#0: 6508 http proxy header: “User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0”
2017/04/26 11:38:28 [debug] 1368#0: 6508 http proxy header: "Accept: image/png,image/;q=0.8,
/
;q=0.5”
2017/04/26 11:38:28 [debug] 1368#0: *6508 http proxy header: “Accept-Language: en-US,en;q=0.5”
2017/04/26 11:38:28 [debug] 1368#0: *6508 http proxy header: “Accept-Encoding: gzip, deflate”
2017/04/26 11:38:28 [debug] 1368#0: 6508 http proxy header: “Referer: http://192.168.0.205/assets/css/mnt.min.css
2017/04/26 11:38:28 [debug] 1368#0: 6508 http proxy header: “Cookie: io=i0aSFyTU2cSAvlgSAAAY; user_image=; user_id=Guest; system_user=yes; full_name=Guest”
2017/04/26 11:38:28 [debug] 1368#0: 6508 http proxy header:
"GET /files/background-image.png HTTP/1.0
X-Forwarded-For: 192.168.0.104
X-Forwarded-Proto: http
X-Frappe-Site-Name: erpnext.vm
Host: 192.168.0.205
X-Use-X-Accel-Redirect: True
Connection: close
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: image/png,image/
;q=0.8,
/
;q=0.5
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://192.168.0.205/assets/css/mnt.min.css
Cookie: io=i0aSFyTU2cSAvlgSAAAY; user_image=; user_id=Guest; system_user=yes; full_name=Guest

"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http cleanup add: 087AE97C
2017/04/26 11:38:28 [debug] 1368#0: *6508 get rr peer, try: 1
2017/04/26 11:38:28 [debug] 1368#0: *6508 socket 9
2017/04/26 11:38:28 [debug] 1368#0: *6508 epoll add connection: fd:9 ev:80000005
2017/04/26 11:38:28 [debug] 1368#0: *6508 connect to 127.0.0.1:8000, fd:9 #6509

sudo nginx -t output
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

There are repeated errors in /var/log/nginx/error.log file

2017/04/24 09:01:27 [error] 1490#0: *178 connect() failed (111: Connection refused) while connecting to upstream, client: 64.233.173.31, server: site1.local, request: “GET /socket.io/?EIO=3&transport=polling&t=1493024486070-1&sid=sf2Kaolb0DX07NEmAAAL HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=1493024486070-1&sid=sf2Kaolb0DX07NEmAAAL”, host: “erp.finbyz.in”, referrer: “http://erp.finbyz.in/desk

2017/04/24 09:01:27 [error] 1490#0: *181 connect() failed (111: Connection refused) while connecting to upstream, client: 49.34.114.173, server: site1.local, request: “GET /socket.io/?EIO=3&transport=websocket&sid=sf2Kaolb0DX07NEmAAAL HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=websocket&sid=sf2Kaolb0DX07NEmAAAL”, host: “erp.finbyz.in”

2017/04/24 09:01:27 [error] 1490#0: *183 connect() failed (111: Connection refused) while connecting to upstream, client: 49.34.114.173, server: site1.local, request: “GET /socket.io/?EIO=3&transport=polling&t=1493024486070-1&sid=sf2Kaolb0DX07NEmAAAL HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=1493024486070-1&sid=sf2Kaolb0DX07NEmAAAL”, host: “erp.finbyz.in”, referrer: “http://erp.finbyz.in/desk

2017/04/24 09:01:28 [error] 1490#0: *183 connect() failed (111: Connection refused) while connecting to upstream, client: 49.34.114.173, server: site1.local, request: “POST /socket.io/?EIO=3&transport=polling&t=1493024487298-2&sid=sf2Kaolb0DX07NEmAAAL HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=1493024487298-2&sid=sf2Kaolb0DX07NEmAAAL”, host: “erp.finbyz.in”, referrer: “http://erp.finbyz.in/desk

2017/04/24 17:37:32 [error] 1560#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 203.88.150.67, server: site1.local, request: “GET /socket.io/?EIO=3&transport=polling&t=1493055451449-92 HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=1493055451449-92”, host: “erp.finbyz.in”, referrer: “http://erp.finbyz.in/desk

Yes now before these errors inspect the log for clues to why these errors

Any joy here Mukesh?

You want to check earlier in the log for signs that might explain what caused the error events later in the log - log areas to inspect are server startup, web client connections and client http requests.

To raise the log level output from say INFO to NOTICE in /etc/nginx/nginx.conf edit this line:

error_log /var/log/nginx/error.log info;

then restart the service with this command:
sudo service nginx reload

http://nginx.org/en/docs/debugging_log.html

To troubleshoot and compare a healthy server log with a problem server log can be a big help…

@clarkej Thanks a lot for all the help.

Actually got tired of solving this puzzle. Eventually did fresh install and restored data from old instance.

Actually time and effort gone in solving the error went much beyond time required to setup and restore data on new server.

Thanks again for the help, really appreciate it. :smiley:

2 Likes

Recently @gvrvnk faced this issue,
Just reinstalled frappe and it worked!

Ran from frappe-bench and somehow it solved the problem

./env/bin/pip install -e apps/frappe
2 Likes

Hey revant, thanks for suggestion. Good to know someone else also faced this issue, lolz :stuck_out_tongue_winking_eye:

Tried reinstalling frappe, as suggested but didn’t work for me. But solved it by fresh install and restoring backup, so cheers!

I actually am experiencing the same issue as well. Doing this Frappe V8.0.20 & later: socketio error - #21 by revant_one doesnt help.

Still investigating but i get the feeling this error is appearing because of this PR, it is the only place that seems to specify the port 6787 New Build System using NodeJS by netchampfaris · Pull Request #3117 · frappe/frappe · GitHub

Am trying to hack at it to see what i can get, looking at the error it seems that whatever the script is doing, it is ignoring the URL and just merely substituting in the protocol as the URL. furthermore the server itself does not have port 6787 open so I am also kind of curious what is it actually that file_watcher_port is pointing to.

@netchampfaris, realized that it is your PR, any ideas why it behaves this way?

1 Like

Fixed via pull request #3232