Deployment to prod on AWS EC2 with Supervisor

i followed the guide for setting up to production server from here and got it running (see below)

frappe-bench-redis:frappe-bench-redis-cache                 RUNNING   pid 2087, uptime 0:02:48
frappe-bench-redis:frappe-bench-redis-queue                 RUNNING   pid 2086, uptime 0:02:48
frappe-bench-redis:frappe-bench-redis-socketio              RUNNING   pid 2093, uptime 0:02:48
frappe-bench-web:frappe-bench-frappe-web                    RUNNING   pid 2084, uptime 0:02:48
frappe-bench-web:frappe-bench-node-socketio                 RUNNING   pid 2085, uptime 0:02:48
frappe-bench-workers:frappe-bench-frappe-default-worker-0   RUNNING   pid 2081, uptime 0:02:48
frappe-bench-workers:frappe-bench-frappe-long-worker-0      RUNNING   pid 2082, uptime 0:02:48
frappe-bench-workers:frappe-bench-frappe-schedule           RUNNING   pid 2080, uptime 0:02:48
frappe-bench-workers:frappe-bench-frappe-short-worker-0     RUNNING   pid 2083, uptime 0:02:48

how can i access the url, AFAIK it’s just something like <public_ip_given at AWS EC2 dashboard>:<port which in this case 8000>

but whenever i do that, it says the site can’t be reached.
i even tried to reboot/stop&start again the ec2 instance but still no luck.

anyone encountered the same or anyone can help me? thanks in advance

You should be able to access it using your public_ip in this case unless you have configured it otherwise.
The default port is 80 for a production installed unless you have specified it otherwise in the nginx conf file.

do i need to set both supervisor and nginx or just having supervisor will do?

You do not need to setup anything separately if you have done a production install nginx and supervisor will be already setup. What are you getting if you type the public ip on your browser?

i’m getting site can’t be reached.
i did the manual installation for production

can you please run this at the command prompt and post the output
sudo netstat -tuplen
Also post your nginx.conf and supervisor.conf files

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      1000       16647       1726/python     
tcp        0      0 127.0.0.1:12000         0.0.0.0:*               LISTEN      1000       16479       1730/redis-server 1
tcp        0      0 127.0.0.1:13000         0.0.0.0:*               LISTEN      1000       16478       1729/redis-server 1
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      113        13781       1121/redis-server 1
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          13952       1136/sshd       
tcp        0      0 127.0.0.1:11000         0.0.0.0:*               LISTEN      1000       16480       1728/redis-server 1
tcp6       0      0 :::9000                 :::*                    LISTEN      1000       17910       1727/node       
tcp6       0      0 :::22                   :::*                    LISTEN      0          13954       1136/sshd       
udp        0      0 0.0.0.0:68              0.0.0.0:*                           0          11980       958/dhclient    

here’s the supervisor.conf generated bench setup supervisor

; Notes:
; priority=1 --> Lower priorities indicate programs that start first and shut down last
; killasgroup=true --> send kill signal to child processes too

[program:frappe-bench-frappe-web]
command=/home/ubuntu/frappe-bench/env/bin/gunicorn -b 127.0.0.1:8000 -w 2 -t 120 frappe.app:application --preload
priority=4
autostart=true
autorestart=true
stdout_logfile=/home/ubuntu/frappe-bench/logs/web.log
stderr_logfile=/home/ubuntu/frappe-bench/logs/web.error.log
user=frappe
directory=/home/ubuntu/frappe-bench/sites


[program:frappe-bench-frappe-schedule]
command=/usr/local/bin/bench schedule
priority=3
autostart=true
autorestart=true
stdout_logfile=/home/ubuntu/frappe-bench/logs/schedule.log
stderr_logfile=/home/ubuntu/frappe-bench/logs/schedule.error.log
user=frappe
directory=/home/ubuntu/frappe-bench

[program:frappe-bench-frappe-default-worker]
command=/usr/local/bin/bench worker --queue default
priority=4
autostart=true
autorestart=true
stdout_logfile=/home/ubuntu/frappe-bench/logs/worker.log
stderr_logfile=/home/ubuntu/frappe-bench/logs/worker.error.log
user=frappe
stopwaitsecs=1560
directory=/home/ubuntu/frappe-bench
killasgroup=true
numprocs=1
process_name=%(program_name)s-%(process_num)d

[program:frappe-bench-frappe-short-worker]
command=/usr/local/bin/bench worker --queue short
priority=4
autostart=true
autorestart=true
stdout_logfile=/home/ubuntu/frappe-bench/logs/worker.log
stderr_logfile=/home/ubuntu/frappe-bench/logs/worker.error.log
user=frappe
stopwaitsecs=360
directory=/home/ubuntu/frappe-bench
killasgroup=true
numprocs=1
process_name=%(program_name)s-%(process_num)d

[program:frappe-bench-frappe-long-worker]
command=/usr/local/bin/bench worker --queue long
priority=4
autostart=true
autorestart=true
stdout_logfile=/home/ubuntu/frappe-bench/logs/worker.log
stderr_logfile=/home/ubuntu/frappe-bench/logs/worker.error.log
user=frappe
stopwaitsecs=1560
directory=/home/ubuntu/frappe-bench
killasgroup=true
numprocs=1
process_name=%(program_name)s-%(process_num)d



[program:frappe-bench-redis-cache]
command=/usr/local/bin/redis-server /home/ubuntu/frappe-bench/config/redis_cache.conf
priority=1
autostart=true
autorestart=true
stdout_logfile=/home/ubuntu/frappe-bench/logs/redis-cache.log
stderr_logfile=/home/ubuntu/frappe-bench/logs/redis-cache.error.log
user=frappe
directory=/home/ubuntu/frappe-bench/sites

[program:frappe-bench-redis-queue]
command=/usr/local/bin/redis-server /home/ubuntu/frappe-bench/config/redis_queue.conf
priority=1
autostart=true
autorestart=true
stdout_logfile=/home/ubuntu/frappe-bench/logs/redis-queue.log
stderr_logfile=/home/ubuntu/frappe-bench/logs/redis-queue.error.log
user=frappe
directory=/home/ubuntu/frappe-bench/sites


[program:frappe-bench-redis-socketio]
command=/usr/local/bin/redis-server /home/ubuntu/frappe-bench/config/redis_socketio.conf
priority=1
autostart=true
autorestart=true
stdout_logfile=/home/ubuntu/frappe-bench/logs/redis-socketio.log
stderr_logfile=/home/ubuntu/frappe-bench/logs/redis-socketio.error.log
user=frappe
directory=/home/ubuntu/frappe-bench/sites


[program:frappe-bench-node-socketio]
command=/usr/bin/node /home/ubuntu/frappe-bench/apps/frappe/socketio.js
priority=4
autostart=true
autorestart=true
stdout_logfile=/home/ubuntu/frappe-bench/logs/node-socketio.log
stderr_logfile=/home/ubuntu/frappe-bench/logs/node-socketio.error.log
user=frappe
directory=/home/ubuntu/frappe-bench




[group:frappe-bench-web]
programs=frappe-bench-frappe-web,frappe-bench-node-socketio



[group:frappe-bench-workers]
programs=frappe-bench-frappe-schedule,frappe-bench-frappe-default-worker,frappe-bench-frappe-short-worker,frappe-bench-frappe-long-worker



[group:frappe-bench-redis]
programs=frappe-bench-redis-cache,frappe-bench-redis-queue,frappe-bench-redis-socketio

here’s the nginx.conf

upstream frappe-bench-frappe {
    server 127.0.0.1:8000 fail_timeout=0;
}

upstream frappe-bench-socketio-server {
    server 127.0.0.1:9000 fail_timeout=0;
}



# setup maps


# server blocks





server {
	listen 80;
	server_name
		site1.local
		;

	root /home/ubuntu/frappe-bench/sites;

	

	

	location /assets {
		try_files $uri =404;
	}

	location ~ ^/protected/(.*) {
		internal;
		try_files /site1.local/$1 =404;
	}

	location /socket.io {
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
		proxy_set_header X-Frappe-Site-Name site1.local;
		proxy_set_header Origin $scheme://$http_host;
		proxy_set_header Host $host;

		proxy_pass http://frappe-bench-socketio-server;
	}

	location / {
		try_files /site1.local/public/$uri @webserver;
	}

	location @webserver {
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Frappe-Site-Name site1.local;
		proxy_set_header Host $host;
		proxy_set_header X-Use-X-Accel-Redirect True;
		proxy_read_timeout 120;
		proxy_redirect off;

		proxy_pass  http://frappe-bench-frappe;
	}

	# error pages
	error_page 502 /502.html;
	location /502.html {
		root /home/ubuntu/bench-repo/bench/config/templates;
		internal;
	}

	# optimizations
	sendfile on;
	keepalive_timeout 15;
	client_max_body_size 50m;
	client_body_buffer_size 16K;
	client_header_buffer_size 1k;

	# enable gzip compresion
	# based on https://mattstauffer.co/blog/enabling-gzip-on-nginx-servers-including-laravel-forge
	gzip on;
	gzip_http_version 1.1;
	gzip_comp_level 5;
	gzip_min_length 256;
	gzip_proxied any;
	gzip_vary on;
	gzip_types
		application/atom+xml
		application/javascript
		application/json
		application/rss+xml
		application/vnd.ms-fontobject
		application/x-font-ttf
		application/font-woff
		application/x-web-app-manifest+json
		application/xhtml+xml
		application/xml
		font/opentype
		image/svg+xml
		image/x-icon
		text/css
		text/plain
		text/x-component
		;
		# text/html is always compressed by HttpGzipModule
}


if i try to access the Public DNS given on AWS EC2 dashboard, i got an Apache2 Ubuntu Default Page that says: It works!

but if i access the Public IP and Public DNS:8000, it’s site cannot be reached

The public dns leads you to the default port 80 which you see when you type just the public ip, but you have configured ERPNext to run on port 8000
Based on the output you have posted for sudo netstat -tuplen It seems that nginx and MySQL are not running
Since you did a manual production install did you install MySQL? Can you see if a site has been created under the frappe-bench/sites folder?

i installed nginx and MariaDB not MySQL. on frappe-bench/sites i have:

drwxrwxr-x 4 ubuntu ubuntu 4096 Sep 11 13:55 .
drwxrwxr-x 9 ubuntu ubuntu 4096 Sep  8 08:31 ..
-rw-rw-r-- 1 ubuntu ubuntu   14 Sep 11 14:04 apps.txt
drwxrwxr-x 4 ubuntu ubuntu 4096 Sep 11 14:04 assets
-rw-rw-r-- 1 ubuntu ubuntu    0 Sep 12 00:51 .build
-rw-rw-r-- 1 ubuntu ubuntu  475 Sep 12 05:43 common_site_config.json
-rw-rw-r-- 1 ubuntu ubuntu   11 Sep 11 13:55 currentsite.txt
drwxrwxr-x 6 ubuntu ubuntu 4096 Sep 11 13:54 site1.local

Okay, can you try running the following commands and try again
sudo service nginx restart
sudo service supervisor restart

Post the output of sudo netstat -tuplen after running the above commands

sudo netstat -tuplen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      1000       18540       2059/python     
tcp        0      0 127.0.0.1:12000         0.0.0.0:*               LISTEN      1000       18479       2063/redis-server 1
tcp        0      0 127.0.0.1:13000         0.0.0.0:*               LISTEN      1000       18475       2062/redis-server 1
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      113        13670       1130/redis-server 1
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      0          17307       2038/nginx -g daemo
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          14710       1108/sshd       
tcp        0      0 127.0.0.1:11000         0.0.0.0:*               LISTEN      1000       17392       2061/redis-server 1
tcp6       0      0 :::9000                 :::*                    LISTEN      1000       18510       2060/node       
tcp6       0      0 :::80                   :::*                    LISTEN      0          17308       2038/nginx -g daemo
tcp6       0      0 :::22                   :::*                    LISTEN      0          14712       1108/sshd       
udp        0      0 0.0.0.0:68              0.0.0.0:*                           0          12358       889/dhclient 

There should be something similar to this as well which is missing in this output
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 113 6777243 17716/mysqld

What do you get now when you try the public ip without any port number?

when i access the Public IP w/o port number i still get site cannot be reached but if i access the Public DNS i still get the Apache2 Ubuntu Home Page saying it works

it seems that i have an error in MySQL service:

sudo systemctl status mysql.service
● mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2017-09-12 06:17:14 UTC; 10s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 4074 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)

Sep 12 06:16:13 ip-172-31-27-0 systemd[1]: Starting LSB: Start and stop the mysql database server daemon...
Sep 12 06:16:13 ip-172-31-27-0 mysql[4074]:  * Starting MariaDB database server mysqld
Sep 12 06:17:14 ip-172-31-27-0 mysql[4074]:    ...fail!
Sep 12 06:17:14 ip-172-31-27-0 systemd[1]: mysql.service: Control process exited, code=exited status=1
Sep 12 06:17:14 ip-172-31-27-0 systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
Sep 12 06:17:14 ip-172-31-27-0 systemd[1]: mysql.service: Unit entered failed state.
Sep 12 06:17:14 ip-172-31-27-0 systemd[1]: mysql.service: Failed with result 'exit-code'.

Have you enabled the ports 80 or 8000 under security group for the server?

yes, it’s already enabled on the security group of the EC2 instance

The MySQL service is not running as I had suspected. Can you check the logs?

yes, i think it’s the MySQL service, and i’m trying to fix it now.
i tried to uninstall and install MySQL again, and i’m getting errors:

Errors were encountered while processing:
 /var/cache/apt/archives/mysql-server-5.7_5.7.19-0ubuntu0.16.04.1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

do you have any idea how can i fix this:

dpkg: dependency problems prevent configuration of mysql-server:
 mysql-server depends on mysql-server-5.7; however:
  Package mysql-server-5.7 is not configured yet.

dpkg: error processing package mysql-server (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          Processing triggers for libc-bin (2.23-0ubuntu9) ...
Processing triggers for systemd (229-4ubuntu19) ...
Processing triggers for ureadahead (0.100.0-19) ...
Errors were encountered while processing:
 mysql-server-5.7
 mysql-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

You would have to remove MySQL completely and install it again.
Alternatively you can try the Easy Install method and reinstall again.