Frappe V8.0.20 & later: socketio error

I tried updating various instances each time after update/ upgrade from 8.0.18/7.2.13 getting below error in console log. This has started happening after v8.0.20, even tried on latest release v8.0.23.

server os: Ubuntu 15 & Ubuntu 16

http:6787/socket.io/?EIO=3&transport=polling&t=1493040979461-36 Failed to load resource: net::ERR_NAME_NOT_RESOLVED
http:6787/socket.io/?EIO=3&transport=polling&t=1493040988459-37 Failed to load resource: net::ERR_NAME_NOT_RESOLVED
http:6787/socket.io/?EIO=3&transport=polling&t=1493040997461-38 Failed to load resource: net::ERR_NAME_NOT_RESOLVED

Please suggest solution if I can correct is without resorting to back-up, Thanks

update: bench setup socketio got below log: Console ERROR is still same

INFO:bench.utils:npm install socket.io redis express superagent cookie
/home/ubuntu/frappe-bench
├── babel-core@6.24.1
├── babel-preset-babili@0.0.12
├── babel-preset-es2015@6.24.1
├── babel-preset-es2016@6.24.1
├── babel-preset-es2017@6.24.1
├── chokidar@1.6.1
├── cookie@0.3.1
├── express@4.15.2
├── redis@2.7.1
├── socket.io@1.7.3
└── superagent@3.5.2

npm WARN optional Skipping failed optional dependency /chokidar/fsevents:
npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.1.1
npm WARN enoent ENOENT: no such file or directory, open '/home/ubuntu/frappe-bench/package.json'
npm WARN frappe-bench No description
npm WARN frappe-bench No repository field.
npm WARN frappe-bench No README data
npm WARN frappe-bench No license field.
1 Like

This suggests the npm WARN is not an issue -

As for this:
http:6787/socket.io/?EIO=3&transport=polling&t=1493040997461-38 Failed to load resource: net::ERR_NAME_NOT_RESOLVED

Maybe restart services or cycle the server itself -

frappe@erpnext:~/bench-repo$ sudo supervisorctl stop all
frappe@erpnext:~/bench-repo$ sudo supervisorctl start all

These logs may give you an idea of the problem:

frappe@erpnext:~/frappe-bench$ ls -al logs | grep redis
-rw-r–r-- 1 frappe root 0 Nov 24 15:18 redis-cache.error.log
-rw-r–r-- 1 frappe root 52620 Apr 5 18:48 redis-cache.log
-rw-r–r-- 1 frappe root 0 Nov 24 15:18 redis-queue.error.log
-rw-r–r-- 1 frappe root 56447 Apr 5 18:48 redis-queue.log
-rw-r–r-- 1 frappe root 0 Nov 24 15:18 redis-socketio.error.log
-rw-r–r-- 1 frappe root 56447 Apr 5 18:48 redis-socketio.log

frappe@erpnext:~/frappe-bench$ less /var/log/nginx/error.log

Here’s another approach Error when loading site

Thanks @clarkej

tried restarting services, even rebooted instance but still browser console is giving same error repeatedly.
less /var/log/nginx/error.log is blank

ubuntu@ip-172-31-0-47:~/frappe-bench$ ls -al logs | grep redis
-rw-r--r-- 1 ubuntu root          0 Jun 17  2016 redis-cache.error.log
-rw-r--r-- 1 ubuntu root      77020 Apr 24 15:33 redis-cache.log
-rw-r--r-- 1 ubuntu root          0 Jun 17  2016 redis-queue.error.log
-rw-r--r-- 1 ubuntu root      77020 Apr 24 15:33 redis-queue.log
-rw-r--r-- 1 ubuntu root          0 Jun 17  2016 redis-socketio.error.log
-rw-r--r-- 1 ubuntu root      77020 Apr 24 15:33 redis-socketio.log

Error in browser console

desk.min.js:12 localStorage cleared
desk.min.js:9 Cleared App Cache.
desk.min.js:12 localStorage cleared
http:6787/socket.io/?EIO=3&transport=polling&t=1493048638468-0 Failed to load resource: net::ERR_NAME_NOT_RESOLVED
http:6787/socket.io/?EIO=3&transport=polling&t=1493048641425-4 Failed to load resource: net::ERR_NAME_NOT_RESOLVED
GET http://http:6787/socket.io/?EIO=3&transport=polling&t=1493048646054-5 net::ERR_NAME_NOT_RESOLVED
GET http://http:6787/socket.io/?EIO=3&transport=polling&t=1493048646054-5 net::ERR_NAME_NOT_RESOLVED

Tried on 3 different Ubuntu instances getting same error after install of frappe v 8.0.20, not able to understand what’s causing it

1 Like

Ok the error shows in the browser console - I assumed console log referred to a terminal console!

You have tried these?

bench clear-cache
bench clear-website-cache

Try Chrome & Firefox - be sure to clear their caches

No reports from others of this error - I must learn how & where to review github commits!

@clarkej

yes tried that, bench clear-cache & bench clear-website-cache.

Earlier checked in chrome after clearing cache, now even checked in edge browser & internet explorer similar error:

SCRIPT7002: XMLHttpRequest: Network Error 0x2ee7, Could not complete the operation due to error 00002ee7.

SCRIPT7002 points to:
https://docs.microsoft.com/en-us/scripting/javascript/reference/javascript-run-time-errors

Is there any way I can upgrade to 8.0.19 from 7.2.13 instead of current version of frappe?

‘Is there any way I can upgrade to 8.0.19 from 7.2.13 instead of current version of frappe?’

Yes but why not exhaust better options first?

So you are on Windows! Here are 4 things to try

Hey @clarkej Thanks a lot for all the help

I tried: Google Chrome’s DNS Prefetching: Settings > Show Advanced Settings > Privacy and disable the option Predict network actions to improve page load performance.

But there’s no change in error, I don’t think this is browser related issue o/w I wouldn’t been able to replicate in other browser (eg edge/IE).

Even though the issue has not been reported, but I have gut feeling there is some fix required in frappe v 8.0.20 to fix this issue. Multiple instances giving same error doesn’t happen that often unless there a bug.

Well your browsers all point to a network or dns issue and run on Windows, yes?

So these makes sense -
restart browser
ifconfig /flushdns
restart Windows

Test access on Mac or Linux desktop to eliminate Windows

Exploring the commits since v8.0.20 or to return to .19 might narrow down the problem commit for an idea of the problem source.

So far still no root cause smoking gun you just have symptoms

Tried restarting PC event tried on 2 other PCs, getting same error.

Still not able to identify root cause, but tried update on one more instance from frappe 8.0.18 to 8.0.23, getting same error in browser console

Everything seems to be working fine apart from this error, so its easy to be missed out unless one happens to check the browser console, this might be the reason its not been reported.

This can also be specific to Ubuntu server, and may not be getting replicated on CentOS. I would request core team to check if they are able to replicate the error.

Can anyone help, still facing same issue with multiple Ubuntu instances.

Issue occurs after updating from frappe 8.0.19 or older to current.

Seems like a config error, Try setting up your config files again

bench setup production

and restart your nginx and supervisor

@rmehta Thanks for reply

Yes it looks like config error, tried bench setup production & restarted nginx & supervisor. Rebooted server as well.

Even manually compared config files of other instances with new, there is no difference in there.

Surprising thing is, I have tried updating 5 different instances all has os ubuntu 14/16, current version frappe version between 7.2.xx to 8.0.19 & in all after update getting same error. But no one else has reported it.

Can you check if it can be replicated at your end on Ubuntu.

Error log has repeated errors as below:

2017/04/23 09:22:54 [error] 1482#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 203.88.159.32, server: site1.local, request: “GET /socket.io/?EIO=3&transport=polling&t=1492939370452-54 HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=1492939370452-54”, host: “demo.finbyz.in”, referrer: “http://demo.finbyz.in/desk

Edit: Output of bench setup production

supervisor.conf already exists and this will overwrite it. Do you want to continue? [y/N]: y
Port configuration list:

Site site1.local assigned port: 80
nginx.conf already exists and this will overwrite it. Do you want to continue? [y/N]: y
INFO:bench.utils:sudo /usr/bin/supervisorctl reread
No config updates to processes
INFO:bench.utils:sudo /usr/bin/supervisorctl update
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
INFO:bench.utils:sudo service nginx reload

  • Reloading nginx configuration nginx

Your ‘connection refused’ may be an nginx config issue?

In my nginx.conf redis listens on port 9000 for local requests -

upstream frappe-bench-socketio-server {
server 127.0.0.1:9000 fail_timeout=0;
}

frappe@erpnext:~/frappe-bench/logs$ nc -v localhost 9000
nc: connect to localhost port 9000 (tcp) failed: Connection refused
Connection to localhost 9000 port [tcp/*] succeeded!

Here are some config problem and fix clues

my nginx.conf is same as yours.

nc -v localhost 9000 output is
Connection to localhost 9000 port [tcp/*] succeeded!

I tried fresh install of ERPNEXT on ubuntu, it doesn’t have any errors.

I copied full config folder from new install to instance where I was getting error, run bench migrate & even rebooted server, still no success. Same error is still there

Say run this check:

frappe@erpnext:~/frappe-bench/config$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

In /var/log/nginx/error.log compare a working instance with one that does not.

Confirm the nginx startup has no errors. Here is an example of a client connection:

(In my case the web browser client runs on 192.168.0.104 that also hosts a kvm guest 192.168.0.105 that runs erpnext.vm - )

2017/04/26 11:38:28 [debug] 1368#0: *6508 http init upstream, client timer: 0
2017/04/26 11:38:28 [debug] 1368#0: *6508 epoll add event: fd:3 op:3 ev:80000005
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "X-Forwarded-For: "
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script var: “192.168.0.104”
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "X-Forwarded-Proto: "
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script var: “http”
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "X-Frappe-Site-Name: erpnext.vm
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "Host: "
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script var: “192.168.0.205”
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: "X-Use-X-Accel-Redirect: True
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: “Connection: close
"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http script copy: “”
2017/04/26 11:38:28 [debug] 1368#0: 6508 http script copy: “”
2017/04/26 11:38:28 [debug] 1368#0: 6508 http proxy header: “User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0”
2017/04/26 11:38:28 [debug] 1368#0: 6508 http proxy header: "Accept: image/png,image/;q=0.8,
/
;q=0.5”
2017/04/26 11:38:28 [debug] 1368#0: *6508 http proxy header: “Accept-Language: en-US,en;q=0.5”
2017/04/26 11:38:28 [debug] 1368#0: *6508 http proxy header: “Accept-Encoding: gzip, deflate”
2017/04/26 11:38:28 [debug] 1368#0: 6508 http proxy header: “Referer: http://192.168.0.205/assets/css/mnt.min.css
2017/04/26 11:38:28 [debug] 1368#0: 6508 http proxy header: “Cookie: io=i0aSFyTU2cSAvlgSAAAY; user_image=; user_id=Guest; system_user=yes; full_name=Guest”
2017/04/26 11:38:28 [debug] 1368#0: 6508 http proxy header:
"GET /files/background-image.png HTTP/1.0
X-Forwarded-For: 192.168.0.104
X-Forwarded-Proto: http
X-Frappe-Site-Name: erpnext.vm
Host: 192.168.0.205
X-Use-X-Accel-Redirect: True
Connection: close
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: image/png,image/
;q=0.8,
/
;q=0.5
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://192.168.0.205/assets/css/mnt.min.css
Cookie: io=i0aSFyTU2cSAvlgSAAAY; user_image=; user_id=Guest; system_user=yes; full_name=Guest

"
2017/04/26 11:38:28 [debug] 1368#0: *6508 http cleanup add: 087AE97C
2017/04/26 11:38:28 [debug] 1368#0: *6508 get rr peer, try: 1
2017/04/26 11:38:28 [debug] 1368#0: *6508 socket 9
2017/04/26 11:38:28 [debug] 1368#0: *6508 epoll add connection: fd:9 ev:80000005
2017/04/26 11:38:28 [debug] 1368#0: *6508 connect to 127.0.0.1:8000, fd:9 #6509

sudo nginx -t output
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

There are repeated errors in /var/log/nginx/error.log file

2017/04/24 09:01:27 [error] 1490#0: *178 connect() failed (111: Connection refused) while connecting to upstream, client: 64.233.173.31, server: site1.local, request: “GET /socket.io/?EIO=3&transport=polling&t=1493024486070-1&sid=sf2Kaolb0DX07NEmAAAL HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=1493024486070-1&sid=sf2Kaolb0DX07NEmAAAL”, host: “erp.finbyz.in”, referrer: “http://erp.finbyz.in/desk

2017/04/24 09:01:27 [error] 1490#0: *181 connect() failed (111: Connection refused) while connecting to upstream, client: 49.34.114.173, server: site1.local, request: “GET /socket.io/?EIO=3&transport=websocket&sid=sf2Kaolb0DX07NEmAAAL HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=websocket&sid=sf2Kaolb0DX07NEmAAAL”, host: “erp.finbyz.in”

2017/04/24 09:01:27 [error] 1490#0: *183 connect() failed (111: Connection refused) while connecting to upstream, client: 49.34.114.173, server: site1.local, request: “GET /socket.io/?EIO=3&transport=polling&t=1493024486070-1&sid=sf2Kaolb0DX07NEmAAAL HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=1493024486070-1&sid=sf2Kaolb0DX07NEmAAAL”, host: “erp.finbyz.in”, referrer: “http://erp.finbyz.in/desk

2017/04/24 09:01:28 [error] 1490#0: *183 connect() failed (111: Connection refused) while connecting to upstream, client: 49.34.114.173, server: site1.local, request: “POST /socket.io/?EIO=3&transport=polling&t=1493024487298-2&sid=sf2Kaolb0DX07NEmAAAL HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=1493024487298-2&sid=sf2Kaolb0DX07NEmAAAL”, host: “erp.finbyz.in”, referrer: “http://erp.finbyz.in/desk

2017/04/24 17:37:32 [error] 1560#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 203.88.150.67, server: site1.local, request: “GET /socket.io/?EIO=3&transport=polling&t=1493055451449-92 HTTP/1.1”, upstream: “http://127.0.0.1:9000/socket.io/?EIO=3&transport=polling&t=1493055451449-92”, host: “erp.finbyz.in”, referrer: “http://erp.finbyz.in/desk

Yes now before these errors inspect the log for clues to why these errors

Any joy here Mukesh?

You want to check earlier in the log for signs that might explain what caused the error events later in the log - log areas to inspect are server startup, web client connections and client http requests.

To raise the log level output from say INFO to NOTICE in /etc/nginx/nginx.conf edit this line:

error_log /var/log/nginx/error.log info;

then restart the service with this command:
sudo service nginx reload

http://nginx.org/en/docs/debugging_log.html

To troubleshoot and compare a healthy server log with a problem server log can be a big help…

@clarkej Thanks a lot for all the help.

Actually got tired of solving this puzzle. Eventually did fresh install and restored data from old instance.

Actually time and effort gone in solving the error went much beyond time required to setup and restore data on new server.

Thanks again for the help, really appreciate it. :smiley:

2 Likes