403 Forbidden Nginx

Hello,

I am facing an error 403 when trying to access erpnext today.
Nginx error log says:
2021/02/01 21:35:24 [error] 1297#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.230, server: erp.jjd, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:8000/”, host: “erp.jjd”
2021/02/01 21:35:24 [error] 1297#0: *1 open() “/home/erp/.local/lib/python3.6/site-packages/bench/config/templates/502.html” failed (13: Permission denied), client: 192.168.1.230, server: erp.jjd, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:8000/”, host: “erp.jjd”

supervisorctl status:
frappe-bench-redis:frappe-bench-redis-cache RUNNING pid 1366, uptime 0:00:43
frappe-bench-redis:frappe-bench-redis-queue RUNNING pid 1367, uptime 0:00:43
frappe-bench-redis:frappe-bench-redis-socketio RUNNING pid 1368, uptime 0:00:43
frappe-bench-web:frappe-bench-frappe-web STARTING
frappe-bench-web:frappe-bench-node-socketio RUNNING pid 2102, uptime 0:00:02
frappe-bench-workers:frappe-bench-frappe-default-worker-0 RUNNING pid 2119, uptime 0:00:02
frappe-bench-workers:frappe-bench-frappe-long-worker-0 STARTING
frappe-bench-workers:frappe-bench-frappe-schedule STARTING
frappe-bench-workers:frappe-bench-frappe-short-worker-0 RUNNING pid 2130, uptime 0:00:01

netstat -plant:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:12000 0.0.0.0:* LISTEN 1368/redis-server 1
tcp 0 0 127.0.0.1:13000 0.0.0.0:* LISTEN 1366/redis-server 1
tcp 0 0 0.0.0.0:2921 0.0.0.0:* LISTEN 898/sshd
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 991/systemd-resolve
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1293/nginx: master
tcp 0 0 127.0.0.1:11000 0.0.0.0:* LISTEN 1367/redis-server 1
tcp 0 64 10.0.0.116:2921 192.168.1.230:51270 ESTABLISHED 1999/sshd: root [pr
tcp 0 0 10.0.0.116:2921 192.168.1.230:51266 ESTABLISHED 1360/sshd: root [pr
tcp6 0 0 :::2921 :::* LISTEN 898/sshd
tcp6 0 0 :::3306 :::* LISTEN 972/mysqld
tcp6 0 0 :::5355 :::* LISTEN 991/systemd-resolve
tcp6 0 0 :::80 :::* LISTEN 1293/nginx: master

/etc/nginx/conf.d/frappe-bench.conf:

upstream frappe-bench-frappe {
server 127.0.0.1:8000 fail_timeout=0;
}

upstream frappe-bench-socketio-server {
server 127.0.0.1:9000 fail_timeout=0;
}

setup maps

server blocks

server {

listen 80;


server_name
	erp.jjd
	merp.jjdstore.kr
	;

root /home/erp/frappe-bench/sites;





add_header X-Frame-Options "SAMEORIGIN";
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";

location /assets {
	try_files $uri =404;
}

location ~ ^/protected/(.*) {
	internal;
	try_files /erp.jjd/$1 =404;
}

location /socket.io {
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "upgrade";
	proxy_set_header X-Frappe-Site-Name erp.jjd;
	proxy_set_header Origin $scheme://$http_host;
	proxy_set_header Host $host;

	proxy_pass http://frappe-bench-socketio-server;
}

location / {

	rewrite ^(.+)/$ $1 permanent;
	rewrite ^(.+)/index\.html$ $1 permanent;
	rewrite ^(.+)\.html$ $1 permanent;

	location ~ ^/files/.*.(htm|html|svg|xml) {
		add_header Content-disposition "attachment";
		try_files /erp.jjd/public/$uri @webserver;
	}

	try_files /erp.jjd/public/$uri @webserver;
}

location @webserver {
	proxy_set_header X-Forwarded-For $remote_addr;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_set_header X-Frappe-Site-Name erp.jjd;
	proxy_set_header Host $host;
	proxy_set_header X-Use-X-Accel-Redirect True;
	proxy_read_timeout 120;
	proxy_redirect off;

	proxy_pass  http://frappe-bench-frappe;
}

# error pages
error_page 502 /502.html;
location /502.html {
	root /home/erp/.local/lib/python3.6/site-packages/bench/config/templates;
	internal;
}

# optimizations
sendfile on;
keepalive_timeout 15;
client_max_body_size 50m;
client_body_buffer_size 16K;
client_header_buffer_size 1k;

# enable gzip compresion
# based on https://mattstauffer.co/blog/enabling-gzip-on-nginx-servers-including-laravel-forge
gzip on;
gzip_http_version 1.1;
gzip_comp_level 5;
gzip_min_length 256;
gzip_proxied any;
gzip_vary on;
gzip_types
	application/atom+xml
	application/javascript
	application/json
	application/rss+xml
	application/vnd.ms-fontobject
	application/x-font-ttf
	application/font-woff
	application/x-web-app-manifest+json
	application/xhtml+xml
	application/xml
	font/opentype
	image/svg+xml
	image/x-icon
	text/css
	text/plain
	text/x-component
	;
	# text/html is always compressed by HttpGzipModule

}

Any help would be much appreciated!

JS

Is this the first attempt to connect to a new installation or was everything working ok, then not?

What does bench setup nginx return?

It was working OK until today. - used over 4months.

cd ~/frappe-bench;
[erp@erp frappe-bench]$ bench setup nginx
nginx.conf already exists and this will overwrite it. Do you want to continue? [y/N]: y
Port configuration list:

Site erp.jjd assigned port: 80

Still 403 error after rebooting.

sudo bench setup nginx
sudo service nginx restart

Also, how are you accessing the server?

Are you trying to connect on the same computer that ERPNext is running on or from a remote computer?

Still 403 error. From remote computer.

From a remote computer. ERPNEXT is on Centos 8 VM.

The nginx log looks like says a client , 192.168.1.230 , is trying to connect to port 8000 Are the port forwards correct or has there been a firewall change? On the CentOs host try installing lynx and going to localhost:80 and 8000 if headless, or in a browser if possible. 403 usually indicates a permission issue of some sort.

It’s on the same internal network so firewalld is disabled on ERPNext server. 10.0.0.116 is the IP of ERPNext server.

Lynx to erp.jjd:80 (403 error), erp.jjd:8000 ( Unable to connect to remote host.), 10.0.0.116:80 (Nginx default page), 10.0.0.116:8000 (Unable to connect to remote host.)

Looks like Ngnix port 8000 is dead or something but I am not sure how to bring it up.

Did you try lynx to localhost:80 or 8000 ?

What is the Adapter, Attached to settings in (i presume) virtualbox , nat or bridged?

Yes, tried to localhost:80 and localhost:8000. Same result as 10.0.0.116:80 and 10.0.0.116:8000.

Bridged.

It’s a kind of urgent so I was thinking to backup the db and reinstall it. I see following error. Is this normal?

mysqldump -u root -p _118c995738a5f0d4 > erp-21-2-1.sql Enter password:
mysqldump: Got error: 1932: “Table ‘_118c995738a5f0d4.__Auth’ doesn’t exist in engine” when using LOCK TABLES

mysqldump -u root -p --skip-lock-tables _118c995738a5f0d4 > erp-21-2-1.sql
Enter password:
mysqldump: Couldn’t execute 'show create table __Auth': Table ‘_118c995738a5f0d4.__Auth’ doesn’t exist in engine (1932)

Just checking , what does sestatus return?

The bench backup command could also be used, if the bench is running.

Disabled.

Thanks for trying to help me out!

I found out there is XFS file system corruption. Not sure this is a root cause. I didn’t have a room for investigating anymore so restored it from old backup.

Thank you again!

You got form nginx “failed (13: Permission denied)” looks like a simple file permission or file system error

I haven’t changed a file permission so I guess a file system corruption could be a root cause.

Thank you.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.