March 16, 2021 by Paul G. | Blog, Features, Shield Pro, Shield Security

Discover Shield’s New Secret Weapon Against WordPress Bots: Anti-Bot Detection Engine

Shield Image

No matter how you look at WordPress security, or from which direction you approach it, it nearly always comes down to bots.

Automation is the key to hacking 1000s of WordPress websites at scale. And the simple way to automate this, is with “bots”.

But, if we can be better at detecting bots, then we can completely revolutionise our WordPress security.

The challenge we face, then, is how can we accurately detect bots hitting our WordPress sites?

ShieldPRO 11.0, due out soon, delivers our unique response to this challenge with your all-new AntiBot Detection Engine.

What Are Bots And Why Are They Such A Big Deal?

First, let’s get really clear on what we mean by “bots”.

The best way to start thinking about bots is that there are 2 types of visitors to your WordPress site:

  1. Humans, and
  2. Everything else (bots)

It’s a simplistic way to split up your traffic, but no matter which way you categorise your WordPress visitors, they fall into 1 of these 2 groups.

For this discussion, let’s put aside human traffic and examine everything else. This could include:

  • Search engine crawlers, such as Google, Bing, Baidu etc.
  • Uptime Monitoring services
  • Integrated services such as iControlWP and MainWP.
  • Comment SPAM bots
  • Login Bots
  • Probing bots and other malicious “bots”

All of these can be referred to as “bots” because they’re automatic – they’re robotic. So calling something a “bot” doesn’t mean they’re bad, they’re just non-human and automated.

The challenge we face, then, is not only in detecting which traffic is human and which is automatic, but which automatic traffic is “good bot”, and “bad bot” traffic.

If we can isolate a “bad bot” from the rest of the pack, then we can treat it differently – i.e. thwart its activity by blocking it from accessing your WordPress site.

Once it’s been blocked, the security threat to your WordPress site from that particular bad bot is neutralised.

This isn’t a trivial task, but with ShieldPRO 11.0, we’ve taken a massive step towards more accurate bad bot detection.

The question then is, can we detect these bad bots?

Yes.

How Does ShieldPRO Detect Malicious WordPress Bots

Earlier we split our traffic up into 2 main categories. Now we’re going to split it further, as follows:

  • Humans
  • Good Bots
  • Bad Bots

Our goal is to isolate “bad bots” from our traffic wherever possible – in this discussion, we’re not interested in dealing with the others.

The steps below outline how Shield goes about identifying bad bots.

Step 1: Eliminate The “Good Bots”

We never want to block the good guys.

So the first step is to identify the “good bots”, so we can eliminate them from the picture.

ShieldPRO 11.0 has hugely improved and expanded its knowledge and processing of the most common “good bots”.

These include bots like:

  • Search engine crawlers: Google, Microsoft/Bing, Yahoo!, DuckDuckGo
  • Uptime Monitoring: Pingdom, Statuscake, Nodeping
  • Performance: GTMetrix
  • WP Management: iControlWP
  • Social Media: Facebook, Pinterest
  • Others: Stripe, PayPal IPN, and more…

And the best part? This list is completely flexible. We can add, remove and modify this list of services at any time without updating the Shield plugin itself. Shield will adapt and update itself to use latest list – you don’t have to do anything!

So Shield will detect a “good bot”, and verify it where necessary. You can learn more about this in the “Fake Web Crawler” section here.

Now that we’ve eliminated the good bots from the picture, the final 2 players remain: bad bots and humans.

Step 2: Bot Signals And They’re Importance In Malicious Bot Detection

This approach isn’t new to ShieldPRO, but we’ve levelled it up for v11.0.

We use the concept of “bot signals” as a way of building a clearer picture about a visitor.

You may think that if a visitor tries to login with an invalid username, then it’s a bot and should be blocked immediately!

That’s just not practical. It’s easy for any normal, WordPress user to mistype their username, or just put in the wrong password. So we can’t block them just yet.

If a visitor performs a number of missteps, or offenses, against our WordPress site, or repeats the same offense a number of times, then we can be increasingly sure it’s a bad bot.

ShieldPRO tracks all of these bot-signals, and has done so for a long time. They include:

  • Fake web crawler idenfication (as discussed above)
  • Triggering a 404 error (used by bots to probe a site for plugins/items)
  • Failed WordPress login
  • Invalid username login
  • Completely invalid User Agents
  • Link Cheese
  • XML-RPC access

Each offenses by a visitor can’t be used in isolation to detect a bad bot, but they can be used together, in combination.

The more signals triggered, the more confidently we can say that a visitor is malicious, and then block their IP from accessing your WordPress site.

But these aren’t the only bot signals we can use.

How ShieldPRO 11.0 Expands Upon The Bot Signals System

The goal of Shield’s new AntiBot Detection Engine is to reliably categorise a visitor as a “bad bot”, or not.

Shield now creates a score for every single WordPress visitor. This score let’s Shield make a decision at any time, as to whether to allow a visitor’s request, or deny it.

To build this score, we draw upon the huge coverage that Shield has on your WordPress site, from failed logins, to comment SPAM, to firewall triggers, and more.

In our first release of the new AntiBot Detection Engine, there are over 25 different bot signals that Shield uses to identify a visitor threat.

Alongside the new…

Shield’s Not-Bot WordPress Visitor Test

A brand new addition to Shield is the visitor “Not Bot” test.

Shield adds a small snippet of Javascript (jQuery not required) to the footer of every page on a WordPress site. This Javascript fires a simple AJAX POST request to the WordPress site, silently in the background. Shield records this request to use as a new bot signal.

So what does firing this AJAX request mean?

Most simple bots don’t process Javascript well. This is either because they technically can’t do it (the vast majority of scripts/bots), or they’re directed not to (GoogleBot, for example, doesn’t fire AJAX POST requests).

If Shield records this AJAX request for a visitor, it means that in all likelihood they’re a real human being, not a bot. Shield combines this not-bot signal with other signals (or the lack of them) to determine whether a visitor is legitimate, or malicious.

Every IP Address is assigned a score based on the signals they trigger.

You can then review the full breakdown of the scores and see what signals have contributed.

How Shield builds a Bot Score for each visitor
Screenshot: An Example of how Shield builds a Bot Score for each visitor

This not-bot signal becomes incredibly useful when combating brute force login requests, comment SPAM, and contact form SPAM.

No More CAPTCHAs or “I’m a human” Checkboxes

With ShieldPRO‘s AntiBot Detection Engine we can now completely remove the need for annoying WordPress login form CAPTCHAs, and of course Shield’s own GASP “I’m a human” checkbox.

Since we’re passively testing a visitor’s status (no direct interaction required from the visitor), when they come to leave a comment, or submit a contact form, ShieldPRO can build their Bot Score and allow or deny the request.

Because we don’t need CAPTCHAs and checkboxes, integrating with custom forms, or 3rd party Contact Forms/Login Forms becomes much, much easier.

We’ve been asked countless times to integrate with Contact Form 7 and Gravity Forms, but the technical requirements to do so until now was always a little daunting.

With our AntiBot System, integration is far easier. And with the release of ShieldPRO 11.0, we’ve already integrated with 16 Contact Form providers, namely:

  1. Contact Form 7
  2. Elementor PRO
  3. Fluent Forms
  4. Formidable Forms
  5. Forminator
  6. Gravity Forms
  7. Kali Forms
  8. Ninja Forms
  9. WPForms
  10. Caldera Forms (beta)
  11. Groundhogg
  12. HappyForms
  13. Super Forms
  14. Support Candy
  15. WeForms
  16. wpForo

This covers millions of WordPress sites and we’re quite sure you’re likely using one of them.

Of course, if your form provider isn’t listed, please reach out to us and we’ll see what we can do to include it.

Future Enhancements To Shield’s Antibot Detection Engine

Our new Antibot Detection Engine in Shield 11.0 (due out soon) is just the start.

With Shield building scores for IP addresses on 1000s of WordPress sites, it makes sense to collate these scores centrally, and allow access to this data by everyone.

We plan to build this out later in the year.

Eventually we’ll be able to provide an API lookup for IP address scores and the more contributors we have offering IP reputation info, the more accurate the system becomes, for everyone.

As with every new system there can be some confusion – please do leave us any comments or questions below and we’ll get right back to you, or update the article as required.

Hello dear reader!

If you want to level-up your WordPress security with ShieldPRO, click to get started today. (risk-free, with our no-quibble 14-day satisfaction promise!)

You'll get all PRO features, including AI Malware Scanning, WP Config File Protection, Plugin and Theme File Guard, import/export, exclusive customer support, and much, much more.

We'd be honoured to have you as a member, and look forward to serving you during your journey towards powerful, WordPress security.

Try ShieldPRO Today →

ShieldPRO Testimonials
@paulbarrett1952's Gravatar @paulbarrett1952

Beats WordFence, especially on price

I run a number of private or not for profit sites where security is needed but budgets are tight, so Shield is the perfect antidote to WordFence. Easy to configure and the pro version is not a rip off. I had it installed on 4 sites. Unfortunately I had to…

@arissports's Gravatar @arissports

Great Plugin

Great plugin doing what it supposed to do.

@carlitapazmino's Gravatar @carlitapazmino

Excelent

Excelent, very easy

@captzeanie2016's Gravatar @captzeanie2016

Works great

A very good plugin

Comments (14)

    Excited for the release of Shield 11.0. Very good news.

      Thanks Charlie! We’re just as excited 😀

    Good news, it seems to be very promising. Good work

      Cheers, Raphael… we’re hopeful it’s as promising as it sounds!

    This is an exciting development! Will there be integration with Beaver Builder, Astra Pro and Mailerlite?

      Thanks for the comment, Lorraine… in what ways specifically would you like to see those 3 plugins integrated?

        Hi Paul, would love to see the AntiBot System integrated with form modules from Beaver Builder, Astra Pro and Mailerlite.

    I would appreciate if you would add PionetForms. Although new, it has great potential and from Pionet who make Elementor addons.

    I get the message »Shield Security Bot Check Failed. Warnung: Wiederholte Anmeldeversuche, die fehlschlagen, führen zu einer vollständigen Sperrung Ihrer IP-Adresse.« Username/password are definitely correct. What can i do now?

      Hi Werner,

      We’re sorry to hear about the troubles you’re having.

      Shield uses JavaScript to help identify bots versus legitimate visitors. However, caching plugins/system often interfere, preventing it loading for your visitors. This may cause your legitimate users to be identified as bots, when they’re not.

      To help you with this, we’ve opened a support ticket for you. Please check your email inbox when you have a chance.

      Thanks 🙂

        Hi Jelena, did not get a mail from you, but i found this page that helped me: https://help.getshieldsecurity.com/article/435-warning-message-shield-security-bot-check-failed

        thx, regards,
        Werner

          Hi Werner,

          Happy to hear that you managed to sort it out.

          And sorry to hear about the email. Whenever you have any problem, please feel free to reach out to us directly. We’d be happy to help.

          Regards,

          Jelena

    When will there be spam bot integration for Happy Forms?

      Hi Angus,

      Thanks so much for your suggestion.

      We can take a look to integrating this. It may take a few weeks to put that in place, but if you want us to investigate it, please send along a link to us directly at support(at)getshieldsecurity.com so we can download the latest version of the plugin if it’s a premium form plugin, or a link to its WordPress.org page if it’s on there.

      Thanks.

      Jelena

Leave a Reply to Charlie Barks Cancel reply

Your email address will not be published. Required fields are marked *

Click to access the login or register cheese