So I’ve found that the blocking is indeed from fail2ban and the particular rule causing this seems to be the following:
#Block IPs trying to use server as proxy.
failregex = <HOST>.*\" 400
This effectively blocks the ip address/network for 10minutes whenever there are up to 6 “http 400 bad request errors” within a space of 10minutes
From the Access logs, I find that requests such as the ones shown below are the cause of the issue:
41.xxx.xxx.x8 - - [24/Oct/2018:07:54:07 +0000] "POST / HTTP/1.1" 400 103 "http://erp.abc.com/desk" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36" "-"
41.xxx.xxx.x8 - - [24/Oct/2018:07:55:08 +0000] "POST /socket.io/?EIO=3&transport=polling&t=MQb4TN_&sid=FH3UE9PiDLiUauc6AAjM HTTP/1.1" 400 52 "http://erp.abc.com/desk" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" "-"
Is it normal to have up to 6 “400 bad request errors” within 10minutes? If so, what would be a more acceptable limit?
If the scenario above is abnormal and indicates a real issue, are there any pointers as to what may be the cause and how we can easily identify the offending client (most likely a web browser as indicated in the logs)?
Hoping someone can help shed more light on this