We’re introducing a Traffic Rate Limiting feature into ShieldPRO for WordPress. It’ll help mitigate larger attacks, and put a stop to traffic abuse, by preventing excessive, overloading requests from any visitor.
In this article we’ll explain what the rate the limiting feature is and how it works, and why this is an important security tool in your WordPress site defenses.
What Is Shield’s Traffic Rate Limiting?
Simply put, rate limiting is where you restrict the number of requests a single visitor can make against your site, within a certain period of time.
There are 2 important factors in Rate Limiting your WordPress site:
- How many requests are allowed in the time period.
- How long a time period will you count the number of requests.
Let’s take the example where you limit to 10 requests within 60 seconds.
When a visitor loads a page on your site, attempts a login, or posts a comment, this will start a counter in the rate limiting system.
If they make another request, perhaps browse to another blog post, they’ll add 1 more request to that counter. If they continue to load pages and they reach 11 requests within a 60 seconds, they’ll trigger Shield’s defenses and an offense will be recorded against their IP address.
If they make a further request, still within the same 60 seconds, the offense limit will be incremented, again.
As always with Shield, when the number of offenses marked against an IP address reaches your threshold, the IP address will be blocked entirely from accessing the site.
With rate limiting activated, any visitors/bots that continue to send too many requests to your site, will be blacklisted.
Note: Visitors that exceed your rate limit don’t get blocked immediately. They will only incur an offense against the site. Too many offenses, and then the IP will be blocked.
Why Is Rate Limiting Good For WordPress Security
There are 2 primary approaches to security:
ShieldPRO tackles both of these, but focuses mainly on prevention.
Shield has important features that cure problems, too, such the WordPress Malware scanner, and the Plugin/Theme Guard that help us repair modified files after an intrusion.
We strongly believe that the earlier you catch a potential issue, a bad-actor, or an intrusion, the better off you’ll be.
It’s far, far easier to prevent a catastrophe, than to recover from one (if you can even recover!). Prevention is much more cost-effective with respect of your time, resources (money), and reputation.
Shield’s Traffic Rate Limiting for WordPress is all about prevention.
How Does Rate Limiting Prevent Security Breaches?
We’ve covered similar approaches in our series of articles about ways that Shield helps to detect malicious bots on your WordPress sites.
ShieldPRO uses these signals to determine whether a visitor is actually a bot that should be neutralised.
The sooner we can identify these bad bots, the sooner we can block access.
It’s hard to state that a visitor is a bot when it triggers just one of these signals, and even harder to say that it’s malicious.
But if the same bot triggers multiple signals, multiple times, then you probably don’t want to allow further access to your site.
Of course, there may be scenarios where we can’t make this statement, and that’s okay, turning on all of these signals just isn’t appropriate in those cases. But for the vast majority of WordPress sites, these signals play a critical role in identifying malicious bots, or bots that don’t have any redeeming quality, and so don’t need access to your site.
The same can be said about Traffic Rate Limiting.
MOST visitors to your site will be human beings, and human beings don’t usually browse your site more than a page or 2 every few seconds.
Again, there will be websites that are the exception to this generalised statement, and for them, rate limiting will need to be more carefully applied.
But if you have bots throwing 10s/100s of requests at your site within a short period of time, you can be sure they’re not up to anything good.
Better to simply block them as early on as possible. In this way you prevent them from probing your WordPress installation any more than they should.
Any attacks they want to throw at your site thereafter, based off of their analysis will be completely useless.
Does Rate Limiting Affect Search Engine Crawlers?
Shield has been filtering out Search Engine traffic for all the major search engines for a long time now. They won’t even register on the site traffic logging system in the first place.
It is this filtering of legitimate search engine traffic that allows Shield to use the “Fake Search Engine Crawler” signal, too. Since we can test which crawlers are genuine, then we can spot bad bots when they say they’re from Google, but they’re actually not.
Does Rate Limiting Monitor All Traffic?
ShieldPRO’s Rate Limiting feature uses the Traffic Logging system as its foundation. This system lets you carefully exclude traffic you don’t want to log (or limit). The only traffic that the Rate Limiter will monitor, is the of traffic that gets picked-up by the Traffic Logging system.
Does Rate Limiting Use More Resources?
The Traffic Logging feature uses a separate database table to record all requests that you’ve set it to monitor.
This of course has a cost – 1x SQL database insert per request. This is practically nothing, and to ensure there’s no slowdown as a result of the query, we execute it at the end of a page request (i.e. when PHP is shutting down) so it will go unnoticed.
The rate limiting feature will, again on PHP shutdown, assess how many requests in the database for this IP within the time frame. It’s an extra query per request, but again, it’s very small.
Also, separately, we keep the traffic log table well trimmed, so lookups are as fast as they can be.
Is There Anything I Should Be Aware Of?
There are a few things to bear in mind when you’re rate limiting your traffic.
#1 Ensure Your Visitor IP Address Source Is Correct
If Shield can’t detect the correct visitor IP address, this will cause lots of trouble, even before you try to limit traffic.
You can’t properly rate limit traffic unless you’re sure Shield has the correct IP address for each visitor. Go to General Settings > IP Source and ensure that the visitor IP address source is correct.
#2 Rate Limiting the WordPress API
If your WordPress site uses the WP REST API extensively, consider excluding the API from your Traffic Logging (and your rate limiting).
Or, if you’re confident with how it works and what sort of API usage you expect or want to allow, Shield’s Rate Limiting feature will be highly effective in throttling REST API access.
#3 Start By Being Generous
If you’re unsure of how your traffic really looks, set your rate limiting options more generously than you might at-first think.
To do this, you would set your ‘Max Request Limit’ higher and your ‘Time Interval’ lower. Doing both or either of these will reduce the chances that legitimate visitors don’t get blocked.
#4 Don’t Forget About AJAX
AJAX requests, particularly in the WordPress admin areas can be quite frequent. Some plugin use AJAX on the frontend also, so your visitors might more requests to your site than you realise. This goes back to #3 – be more generous at the start and dial it back slowly.
How Can You Get Rate Limiting For WordPress?
This security feature will be released with Shield Security PRO around mid-late March 2020. It’s undergoing final testing and quality control, but we’ll release it as soon as it’s ready.
Comments or Suggestions?
As always, we welcome your suggestions and feedback about this feature, and any other ShieldPRO feature. Please leave your comments below and we’ll get right back to you!
Great plugin which seems to work very well. Difficult for me to check what is really going on in the background because I do not have the knowledge for that. So what I can see directly is that it blocks a lot of hacking attempts and that makes me very…
Good plugin, easy to setup
Very nice plugin for more WordPress security. Works well in combination with Sucuri Security . What would be nice is to be able to add multiple languages for the custom messages of the spamfilter, e.g. when working with plugins like polylang or wpml. Keep up the good work!
Great security features! I like the flexibility and granularity.
Looks like a winner.
Working well so far. No issues.
Hey there gorgeous! Do you like what you've read here? :)
If this cool feature is something you'd like, but you haven't gone PRO yet, click here to get started today. (no risk, with a 14-day satisfaction guarantee!)
You'll get all PRO features, including Malware Scanning, WP Config Protection, Plugin FileGuard, import/export, customer support, and so much more. Not only that, you'll get that warm, fuzzy feeling that comes from supporting our work and future development.
How do I track the IP address through your plugin?
How exactly do you mean “track the IP”?
If you want to see what activity the IP is doing on your site, you can review the Audit Trail and Traffic Log (and filter by the IP you want to review).
For traffic to appear on the Audit Trail, it will need to perform an action on the site that triggers Shield in some way, and for the Traffic Log, the IP will need to have sent a request that isn’t excluded based on your Traffic Log settings.
Hope this helps.
I’m currently using the Free edition, although I would have no problem updating to pro if it is of merit.
I am wondering if this feature was also applied to the Free edition because last week my other Admin started getting locked out as he was creating new member accounts. I already have his IP address whitelisted, but that doesn’t seem to help.
My site is a forum, and we create the accounts manually as we had an issue with spam accounts. We’ve switched to this method since December and, until last week, we had no problems. We currently run the free version of Shield, ver 8.7.0.
Again I would be happy to update to Pro but first I want to make sure that Shield is not actually causing this problem.
Thank you for your comment. We can assure you, if you don’t have the PRO features enabled, then your site will not be rate limiting any visitors.
Not only this, if your admin’s IP address is whitelisted, then no Shield Security settings will be applied to them whatsoever so they wont even show up in your Traffic Log.
Please feel free to drop us a message to our support team to discuss these issues further if you need to.
Thanks for your response Paul, our issue started about a week ago and isn’t necessarily being caused by Shield. I just wanted to eliminate the possibility that the latest Shield update may have affected the site as the symptom is quite similar to this new feature. It could very well be related to the latest WordPress update, or another factor unrelated tp Shield.
I have a couple of other questions regarding the Pro version but I’ll just email them to you, as I don’t want to “hijack” this thread! I appreciate your quick response and the email you sent me, impressive as I’m not yet a Pro subscriber (but probably will be soon).
Hi, I have a question. I talked to you on wordpress.org also. I am planning to buy the pro version of the plugin but I am just trying to do as much research as possible. There is this article I read that stated that rate limiting does not distinguish between good and bad bots. That worries me quite a bit but above you have stated that SHield Pro does recognize good bots. Will this be true for all good bots? That article actually advised me to ask the security vendor these 4 questions:
1) Do you provide application-layer (L7) DDoS protection as part of your DDoS solution, or does it require an add-on WAF component?
2) Do you use behavioral learning algorithms to establish ‘legitimate’ traffic patterns?
3) How do you distinguish between good and bad traffic?
4) Do you have application-layer DDoS protection that goes beyond rate limiting?
Hope you dont feel I am grilling you. I am just tired of researching this scraping and bad bot stuff so much. Shield seems to be the answer but I want to make sure. Could you kindly address the questions?
Thanks for your questions – I’ll try to address them here but if you need further clarification, please reach out to us on help desk.
Q1) No, we don’t. The reason we don’t is because the server itself shouldn’t be the agent implementing DDoS, wherever possible. The reason DDoS works is because it overloads the server so that it can’t respond adequately to so many requests. The best solution, by-far, is to have a reverse-proxy service preventing DoS traffic even reaching your server in the first place
We recommend everyone use CloudFlare – the free tier offers wonderful DoS protection, not to mention a whole host of other benefits.
We don’t believe in reinventing solutions when they’ve been implemented elsewhere, in the best way possible, and they’re accessible by everyone.
CloudFlare offers rate limiting, but in very specific circumstances. Our rate limiting is generalised across the entire WordPress site and so has a slightly different use-case.
Q2) No, Shield doesn’t track behaviour like this at this time. It tracks offending visitors/bots over a range of activities and blocks their access if they trigger Shield’s defenses enough times to exceed your desired limit. This defaults to 10, but you can set this lower to be more strict, or higher to be more lenient.
Q3) This question is quite general and comes down to how you define good and bad. We whitelist certain traffic and services, for example Stripe, Pingdom, Statuscake etc., and also whitelist legitimate search engine bots. Other than this, bad traffic is that which triggers Shield defenses, good traffic is that which doesn’t. You decide based on the Shield settings what is good and bad.
For example consider the question: Are 404 errors “bad”?
No, probably not, here and there. But what if it’s a bot probing for known vulnerable php scripts that doesn’t exist on your site? Are those 404s bad? Yes. This is why Shield has a “limit” (see Q2) so that normal human traffic that triggers a 404 doesn’t get blocked, but bots that repeat 404s, will get blocked eventually.
Q4) This is the same as Q1 as I understand.
I hope this all helps.
I will be going with shield pro as soon as possible!
Thanks Paul! You have been very kind and informative. I will be bothering you and your support staff more once I purchase the plugin!
Hi there. Should the default of 60 and 60 avoid any issues with most/all of the good crawlers? Thanks!
Shield has special handling in there for all major search engines so that if they ever try to access your site in any way, Shield will never stop them – they’re basically whitelisted.
For example, we can confirm whether a bot is real-Google or fake-Google and use this as a ‘bad bot’ signal. If it’s a bad bot, we block it. Good bot crawler – never get blocked.
So, no matter what Traffic Rate Limiter settings you choose, it won’t block good crawlers.
Regarding the Traffic Rate Limiter settings in general, you can always get started with the default ones and monitor your activity log to see how that works for you. You can then strengthen it little bit later if you have to.
But please ensure that your settings do not block you legit site users.
We have an article here, explaining how Traffic Rate Limiter works in practice. Maybe you’ll find it helpful in some way.