Server does not randomly start stop. Because of 1 user fail2ban acts. By then another user accessing the server keeps refreshing the page. It causes fail2ban to add more time. I am not sure of this, it was what I observed at a client who had about 15 simultaneous users at one location.
Best solution is to get a static IP at the main which has the highest number of users. Add that to fail2ban. After that users accessing server from that IP address will never get banned no matter how many users keep hitting the server. Once you have the static IP do this
sudo fail2ban-client set nginx-proxy addignoreip IP.IP.IP.IP
We did this and since then there are no complaints of inaccessible server.