Over the weekend my site suffered from a spike in traffic that caused the WordPress interface and then the rest of the site to grind to a halt. This blog runs on a VPS that has about the same computing power as a first generation Raspberry Pi and so it doesn’t take much to exhaust the hamster and cause a rather irritating Denial of Service.
Running a site with a hugely popular tool like WordPress can cause unwanted issues because its security weaknesses are so well known and are regularly attacked. I decided to dig a little deeper to find out how the DOS occurred and a found a couple of measures that helped to fix it.
Check The Logs
First step was to check the Apache logs to try and find the origins of the spike in traffic:
cat /var/log/apache2/access.log
The culprit stood out straight away. Two IP addresses were spamming the xmlrpc.php
file on my server several times a second:
83.220.168.109 - - [07/Jun/2020:21:10:59 +0000] "POST /xmlrpc.php HTTP/1.0" 200 83.220.168.108 - - [07/Jun/2020:21:11:00 +0000] "POST /xmlrpc.php HTTP/1.0" 200 83.220.168.109 - - [07/Jun/2020:21:11:00 +0000] "POST /xmlrpc.php HTTP/1.0" 200 83.220.168.109 - - [07/Jun/2020:21:11:02 +0000] "POST /xmlrpc.php HTTP/1.0" 200 83.220.168.109 - - [07/Jun/2020:21:11:02 +0000] "POST /xmlrpc.php HTTP/1.0" 200 83.220.168.108 - - [07/Jun/2020:21:11:02 +0000] "POST /xmlrpc.php HTTP/1.0" 200 83.220.168.108 - - [07/Jun/2020:21:11:03 +0000] "POST /xmlrpc.php HTTP/1.0" 200 83.220.168.109 - - [07/Jun/2020:21:11:04 +0000] "POST /xmlrpc.php HTTP/1.0" 200 83.220.168.108 - - [07/Jun/2020:21:11:04 +0000] "POST /xmlrpc.php HTTP/1.0" 200
The two IPs had been doing this for over 24 hours. Every time they made a request to xmlrpc.php
my tiny server returned an HTTP 200 code. No wonder it got too busy to do anything else and ground to a halt.
Disabling XMLRPC.php
xmlrpc.php
is enabled by default in most WordPress installations. It’s a useful feature that allows users to integrate other apps with WordPress so that they can publish or edit content, or send pings and trackbacks. Unfortunately it’s also possible to exploit it to try and log in to a WordPress site by bruteforcing the username and password. Bruteforce attacks on WordPress are common and most are easy to block but I’d hadn’t disabled xmlrpc.php
and now my server was paying the price. You can read a little more about this exploit here.
I don’t really need any of its features, so I decided to disablexmlrpc.php
. The quick and easy solution is to disable access to it by editing the site’s .htaccess file and adding the following lines:
<files xmlrpc.php> Require all denied </files>
Instead of getting a 200 HTTP response code, the attackers now get a 403 Forbidden message. You can see the impact in the logs immediately:
83.220.168.108 - - [07/Jun/2020:21:54:43 +0000] "POST /xmlrpc.php HTTP/1.0" 200 83.220.168.108 - - [07/Jun/2020:21:54:43 +0000] "POST /xmlrpc.php HTTP/1.0" 200 83.220.168.108 - - [07/Jun/2020:21:54:44 +0000] "POST /xmlrpc.php HTTP/1.0" 200 83.220.168.108 - - [07/Jun/2020:21:54:46 +0000] "POST /xmlrpc.php HTTP/1.0" 403 83.220.168.108 - - [07/Jun/2020:21:54:46 +0000] "POST /xmlrpc.php HTTP/1.0" 403 83.220.168.108 - - [07/Jun/2020:21:54:47 +0000] "POST /xmlrpc.php HTTP/1.0" 403
It’s nice but it still doesn’t really solve the problem. My server still has to process the requests to issue the 403 status codes and uses precious resources in doing so.
Blocking XMLRPC Requests With Fail2Ban
A much better approach is to configure Fail2Ban to spot malicious xmlrpc.php
requests and drop connections from any IPs that attempt to make them. There’s no jail in the default Fail2Ban setup that will automatically do this so it’s necessary to create one.
First of all create a custom config file containing the regex that Fail2Ban needs to look for:
nano /etc/fail2ban/filter.d/xmlrpc.conf
Then in the file enter the regex to capture the HTTP requests:
[Definition] failregex = ^<HOST> .* "POST .*xmlrpc.php
Next we need to create the jail in jail.local
:
nano /etc/fail2ban/jail.local
Add the following lines to create the [xmlrpc]
jail:
[xmlrpc] enabled = true port = http,https filter = xmlrpc logpath = /var/log/apache2/access.log maxretry = 5 findtime = 600 bantime = 86400 banaction = iptables-allports
This jail tells Fail2Ban to monitor the Apache access log for any requests that match the potentially malicious regex.
maxretry = 10
and findtime = 300
sets the threshold to ten requests in 300 seconds. Setting bantime
to 86400 ensures that any IPs that meet these criteria will be blocked by the firewall for 24 hours. Setting the thresholds is a little tricky. I don’t want to unintentionally ban anyone, only stop the spammers, so I might tweak this as time goes by.
Once the changes are made, restart Fail2Ban with:
service fail2ban restart
Ensure everything went ok by checking the status:
service fail2ban status
Only a short time after implementing this change one of the malicious IPs wanted to continue probing away at xmlrpc.php
again. A quick check of the Fail2Ban log for activity brought back the details:
cat /var/log/fail2ban.log | grep xmlrpc
It didn’t take long to confirm the new changes work:
17:38:40,567 fail2ban.filter [3243]: INFO [xmlrpc] Found 83.220.168.109 17:38:40,567 fail2ban.filter [3243]: INFO [xmlrpc] Found 83.220.168.109 17:38:40,568 fail2ban.filter [3243]: INFO [xmlrpc] Found 83.220.168.109 17:38:40,568 fail2ban.filter [3243]: INFO [xmlrpc] Found 83.220.168.109 17:38:40,570 fail2ban.filter [3243]: INFO [xmlrpc] Found 83.220.168.109 17:38:40,570 fail2ban.filter [3243]: INFO [xmlrpc] Found 83.220.168.109 17:38:40,577 fail2ban.filter [3243]: INFO [xmlrpc] Found 83.220.168.109 17:38:40,583 fail2ban.filter [3243]: INFO [xmlrpc] Found 83.220.168.109 17:38:40,746 fail2ban.actions [3243]: NOTICE [xmlrpc] Ban 83.220.168.109
Fail2Ban scans the logs and finds activity that triggers the banning threshold. The jail kicks in to ensure all connections from the IP are dropped and my server can breathe a sigh of relief.
For a more detailed guide on setting up Fail2Ban on Ubuntu this tutorial is really useful. It’s for an older version of Ubuntu but the details are still valid.
Thanks for this.
The xmlrpc jail was helpful.
I’d tried blocking the IPs with UFW but that wasn’t working to stop them totally.
Found your post by Googling the IP address that was hitting you too.
Cheers
Thanks Paul, glad you found it helpful. It’s worked well for me since I added the jail to Fail2ban and no problems since.