Last week we released ShieldPRO 11.2 and if you look at the changelog, we’ve released 4 “patch” updates for that release.
It wasn’t because there were bugs (there were, but), it was primarily because we’d adjusted the scoring logic for the AntiBot Detection Engine.
Often with new features, they need some realworld testing and hardening to iron out the creases and spot any gaps or weaknesses.
We released improvements to the engine in 11.2 and in version 11.3 we’ve added a much-needed enhancement to eliminate excessive Shield releases.
To explain this, we need to understand the changes introduced in 11.2.
Improvements Introduced in Shield 11.2
We added a few significant improvements in 11.2, most notably:
- removing the need for cookies to track certain elements
- improved timing of the NotBot Javascript execution to ensure more visitors performed the necessary actions
The 2 improvements should drastically reduce, and hopefully eliminate, the incidents where legitimate WordPress users would be labelled as bots.
We received numerous reports that this was happening for our customers but up until that point, it wasn’t even on our radar in our tests.
Another area we tweaked was making the scoring system a bit more lenient to err on the side of caution and reduce false positives.
By defaulting to a better reputation for new site visitors, you’d have fewer users locked out from logging into the site.
This, however, was where the trouble begain with the Shield 11.2 release.
Why We Saw More WordPress Comments SPAM and Contact SPAM
If you increase the reputation score for all new visitors, then even bots which are “new” to your site will have a better reputation than they should.
This caused their scores to be high enough to meet the minimum AntiBot level and so their SPAM comments were permitted.
And of course, if you were using any of our Contact Form integrations, the problem was the same.
3 of the patch releases to Shield 11.2 included updates to the scoring system to ensure that new visitors would be given a pass, while bots wouldn’t.
It’s a tricky balance to get right since going too far in either direction isn’t a great experience for anybody.
How We’ve Improved Bot Scores In Shield 11.3
It’s great that we can tweak the system for scoring bots quickly, but it doesn’t make sense to release a new version of Shield each time.
With Shield 11.3+ we’re moving towards a dynamic scoring system which we can centrally control, without issuing a plugin update.
Each Shield plugin will download the latest scoring logic from our ShieldNET API, so if we see the need for any improvements, we simply update the scoring on our side and your sites will update automatically.
Shield will always have a fallback scoring system within the plugin itself in-case it can’t access the API, so you’ll always be covered.
Other Improvements To Shield In 11.3
Smarter 404 Bot Signal Handling
Based on the feedback from some clients, we’ve adjusted how the 404 Bot Signal operates.
Excessive 404 errors on a site are bad, for a myriad of reasons. It’s a terrible user experience for starters.
Many themes can cause 404 errors for assets such as Javascript and CSS files, and, we’ve come to realise, these often go unnoticed or ignored, even by the most seasoned website admins.
This caused a bit of a problem for our 404 Bot Signals system which would record these 404s as offenses and also use it as a scoring factor for the AntiBot Detection Engine.
Since bots probe your site looking for particular items, such a vulnerable plugins and scripts, they typically generate a lot of 404 errors. It’s an extremely reliable bot signal.
But if we’re triggering the bot signal for legitimate users, too, this is bad.
So we needed a more nuanced approach. This has been implemeneted in Shield 11.3.
Here’s the difference:
- 404 error requests still trigger the bot signals as normal, the feature itself hasn’t changed
- However, if the 404 request is to a Javascript, CSS, or image file (PNG, JPEG), then an offense against the IP will not be triggered.
This means that legitimate users (and bots) get a PASS if their 404 requests are for common types of assets.
But we weren’t happy with this. So we decided to adjust the logic further.
Let’s say a bot wanted to check whether you had Woocommerce installed. They may try to request a CSS file it “knows” to be in the Woocommerce plugin directory.
If it requests the CSS file with Shield 11.3, no offense will be recorded against that IP for a 404. This isn’t ideal.
So we added the additional rule:
- if the request is to assets, and it’s directed towards a plugin or theme directory, and that plugin or theme isn’t actually installed, then the 404 error will still trigger an offense.
High IP Reputation Bypass
Since releasing the AntiBot Detection Engine (ADE), we’ve been deliberately reluctant to combine it with our long-establish IP blocking and offenses system.
We wanted to ensure we’d ironed out the kinks to the ADE first.
Based on customer feedback we’ve made an adjustment to allow a high-reputation IP (i.e. with a good ADE score) to bypass the IP blocking system.
The IP address will still accumulate offenses and will still be subject to Shield’s rules, but, if the number of offenses would normally lead to an IP address being blocked, but the IP reputation is good enough, the block will not be put in-place.
You can think of it like: Shield will see everything your IP does, and it’ll mark offenses against it. Once the IP has accumulated enough offenses and it’s about to block your IP address, it’ll lookup your Bot Reputation Score and if it’s high enough, you wont be blocked.
NotBot Javascript Loading Check
We’ll sometimes here from clients that they’re failing the Antibot test. The #1 reason for this is that the NotBot Javascript isn’t being loaded on the front end of the site.
This is typically because of misconfigured site optimisation plugins that aren’t correctly picking up the NotBot JS file, or aggressive page caching is affecting the normal enqueuing of assets on the site.
So we added a quick little test so you can be forewarned that the NotBot JS isn’t working on your site.
Questions, Suggestion and Feedback
As with every release, there are bug fixes and code enhancements that don’t really need to be detailed, but we’re always working to ensure that Shield is as bug-free and stable as we can make it.
If you have any questions or suggestions about anything raised in this article, please don’t hesitate to leave us a comment below. Thanks!
