403 Forbidden Nginx

Install / Update
1

Hello,

I am facing an error 403 when trying to access erpnext today.
Nginx error log says:
2021/02/01 21:35:24 [error] 1297#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.230, server: erp.jjd, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:8000/”, host: “erp.jjd”
2021/02/01 21:35:24 [error] 1297#0: *1 open() “/home/erp/.local/lib/python3.6/site-packages/bench/config/templates/502.html” failed (13: Permission denied), client: 192.168.1.230, server: erp.jjd, request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:8000/”, host: “erp.jjd”

supervisorctl status:
frappe-bench-redis:frappe-bench-redis-cache RUNNING pid 1366, uptime 0:00:43
frappe-bench-redis:frappe-bench-redis-queue RUNNING pid 1367, uptime 0:00:43
frappe-bench-redis:frappe-bench-redis-socketio RUNNING pid 1368, uptime 0:00:43
frappe-bench-web:frappe-bench-frappe-web STARTING
frappe-bench-web:frappe-bench-node-socketio RUNNING pid 2102, uptime 0:00:02
frappe-bench-workers:frappe-bench-frappe-default-worker-0 RUNNING pid 2119, uptime 0:00:02
frappe-bench-workers:frappe-bench-frappe-long-worker-0 STARTING
frappe-bench-workers:frappe-bench-frappe-schedule STARTING
frappe-bench-workers:frappe-bench-frappe-short-worker-0 RUNNING pid 2130, uptime 0:00:01

netstat -plant:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:12000 0.0.0.0:* LISTEN 1368/redis-server 1
tcp 0 0 127.0.0.1:13000 0.0.0.0:* LISTEN 1366/redis-server 1
tcp 0 0 0.0.0.0:2921 0.0.0.0:* LISTEN 898/sshd
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 991/systemd-resolve
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1293/nginx: master
tcp 0 0 127.0.0.1:11000 0.0.0.0:* LISTEN 1367/redis-server 1
tcp 0 64 10.0.0.116:2921 192.168.1.230:51270 ESTABLISHED 1999/sshd: root [pr
tcp 0 0 10.0.0.116:2921 192.168.1.230:51266 ESTABLISHED 1360/sshd: root [pr
tcp6 0 0 :::2921 :::* LISTEN 898/sshd
tcp6 0 0 :::3306 :::* LISTEN 972/mysqld
tcp6 0 0 :::5355 :::* LISTEN 991/systemd-resolve
tcp6 0 0 :::80 :::* LISTEN 1293/nginx: master

/etc/nginx/conf.d/frappe-bench.conf:

upstream frappe-bench-frappe {
server 127.0.0.1:8000 fail_timeout=0;
}

upstream frappe-bench-socketio-server {
server 127.0.0.1:9000 fail_timeout=0;
}

setup maps

server blocks

server {

listen 80;


server_name
	erp.jjd
	merp.jjdstore.kr
	;

root /home/erp/frappe-bench/sites;





add_header X-Frame-Options "SAMEORIGIN";
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";

location /assets {
	try_files $uri =404;
}

location ~ ^/protected/(.*) {
	internal;
	try_files /erp.jjd/$1 =404;
}

location /socket.io {
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "upgrade";
	proxy_set_header X-Frappe-Site-Name erp.jjd;
	proxy_set_header Origin $scheme://$http_host;
	proxy_set_header Host $host;

	proxy_pass http://frappe-bench-socketio-server;
}

location / {

	rewrite ^(.+)/$ $1 permanent;
	rewrite ^(.+)/index\.html$ $1 permanent;
	rewrite ^(.+)\.html$ $1 permanent;

	location ~ ^/files/.*.(htm|html|svg|xml) {
		add_header Content-disposition "attachment";
		try_files /erp.jjd/public/$uri @webserver;
	}

	try_files /erp.jjd/public/$uri @webserver;
}

location @webserver {
	proxy_set_header X-Forwarded-For $remote_addr;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_set_header X-Frappe-Site-Name erp.jjd;
	proxy_set_header Host $host;
	proxy_set_header X-Use-X-Accel-Redirect True;
	proxy_read_timeout 120;
	proxy_redirect off;

	proxy_pass  http://frappe-bench-frappe;
}

# error pages
error_page 502 /502.html;
location /502.html {
	root /home/erp/.local/lib/python3.6/site-packages/bench/config/templates;
	internal;
}

# optimizations
sendfile on;
keepalive_timeout 15;
client_max_body_size 50m;
client_body_buffer_size 16K;
client_header_buffer_size 1k;

# enable gzip compresion
# based on https://mattstauffer.co/blog/enabling-gzip-on-nginx-servers-including-laravel-forge
gzip on;
gzip_http_version 1.1;
gzip_comp_level 5;
gzip_min_length 256;
gzip_proxied any;
gzip_vary on;
gzip_types
	application/atom+xml
	application/javascript
	application/json
	application/rss+xml
	application/vnd.ms-fontobject
	application/x-font-ttf
	application/font-woff
	application/x-web-app-manifest+json
	application/xhtml+xml
	application/xml
	font/opentype
	image/svg+xml
	image/x-icon
	text/css
	text/plain
	text/x-component
	;
	# text/html is always compressed by HttpGzipModule

}

Any help would be much appreciated!

JS

2

Is this the first attempt to connect to a new installation or was everything working ok, then not?

What does bench setup nginx return?

3

It was working OK until today. - used over 4months.

cd ~/frappe-bench;
[erp@erp frappe-bench]$ bench setup nginx
nginx.conf already exists and this will overwrite it. Do you want to continue? [y/N]: y
Port configuration list:

Site erp.jjd assigned port: 80

Still 403 error after rebooting.

4 
sudo bench setup nginx
sudo service nginx restart

Also, how are you accessing the server?

5

Are you trying to connect on the same computer that ERPNext is running on or from a remote computer?

6

Still 403 error. From remote computer.

7

From a remote computer. ERPNEXT is on Centos 8 VM.

8

The nginx log looks like says a client , 192.168.1.230 , is trying to connect to port 8000 Are the port forwards correct or has there been a firewall change? On the CentOs host try installing lynx and going to localhost:80 and 8000 if headless, or in a browser if possible. 403 usually indicates a permission issue of some sort.

9

It’s on the same internal network so firewalld is disabled on ERPNext server. 10.0.0.116 is the IP of ERPNext server.

Lynx to erp.jjd:80 (403 error), erp.jjd:8000 ( Unable to connect to remote host.), 10.0.0.116:80 (Nginx default page), 10.0.0.116:8000 (Unable to connect to remote host.)

Looks like Ngnix port 8000 is dead or something but I am not sure how to bring it up.

10

Did you try lynx to localhost:80 or 8000 ?

What is the Adapter, Attached to settings in (i presume) virtualbox , nat or bridged?

11

Yes, tried to localhost:80 and localhost:8000. Same result as 10.0.0.116:80 and 10.0.0.116:8000.

Bridged.

It’s a kind of urgent so I was thinking to backup the db and reinstall it. I see following error. Is this normal?

mysqldump -u root -p _118c995738a5f0d4 > erp-21-2-1.sql Enter password:
mysqldump: Got error: 1932: “Table ‘_118c995738a5f0d4.__Auth’ doesn’t exist in engine” when using LOCK TABLES

mysqldump -u root -p --skip-lock-tables _118c995738a5f0d4 > erp-21-2-1.sql
Enter password:
mysqldump: Couldn’t execute ‘show create table __Auth’: Table ‘_118c995738a5f0d4.__Auth’ doesn’t exist in engine (1932)

12

Just checking , what does sestatus return?

The bench backup command could also be used, if the bench is running.

13

Disabled.

Thanks for trying to help me out!

I found out there is XFS file system corruption. Not sure this is a root cause. I didn’t have a room for investigating anymore so restored it from old backup.

Thank you again!

14

You got form nginx “failed (13: Permission denied)” looks like a simple file permission or file system error

15

I haven’t changed a file permission so I guess a file system corruption could be a root cause.

Thank you.

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.