What AI Is Doing to Your Website and What to Do About It

• By Scott Smith
• October 26, 2026

AI and LLMs are the hot topics of the day. Everyone’s excited, angry, and worried. The biggest concerns are that they are shortcutting traditional search, reducing job opportunities through automation, and affecting communities and the environment. Or they might bring on the apocalypse. No biggie. But while we’re waiting for the other robot shoe to drop, let’s talk about something that doesn’t get much attention.

In the not-too-recent past, Google was the only real big player in search. Their crawlers would index the entire web and drop in on sites on a regular schedule to see what’s new. There were malicious actors out there too, who would write programs to conduct DDoS attacks that could be pesky and costly.

Boy, times have changed. Now, lots of companies are crawling your site, and they are crawling CONSTANTLY. It’s easy and free to vibe code a program to exploit vulnerabilities in WordPress, Drupal, and other CMSs and their plugins, just by entering a prompt. Now the maintainers of those systems, and the websites built on them, are constantly patching.

This can cause your site to slow down or go offline entirely, and in some cases, you pay for traffic that has nothing to do with human visitors. This can turn into headaches and real money.

However, you can protect your site, and it’s pretty easy and inexpensive.

A lot of this is happening because there’s a new, exciting/scary technology that has created a lot of competition. These crawlers aren’t trying to take your site down or cost you money. They’re just trying to deliver something valuable to their own users.

Of course, another class of bot is malicious. They are looking for unlocked doors like login pages, any outdated plugins, directories that are open they can write to. Attackers are bad.

There’s a third group of bots as well, namely researchers and security firms. These penetration tests are always happening, looking for the weaknesses in popular content management systems and other technologies.

So if you want your site to stay up and to be found by humans, you can’t really block all bots or AI. That’s not the right approach.

How AI crawlers work

Let’s talk about group 1, the non-malicious crawlers, and how they cause issues.

A person coming to a site reads a page or two, and then goes on with their day. Crawler bots try to access every single possible path, even paths to content they’ve already seen.

For example, let’s say your website has a news or resources section with two filters: type and topics. You do that so people can find blog posts about housing policy, for example. Because you’re a housing nonprofit, you have dozens of those posts.

What the bot does is work through every combination of type and topic, and then through every paginated page in those results. It has already visited every post, but it keeps going. That means that the page is getting hundreds or thousands of hits.

This is why sites with lots of filters (event calendars, staff directories, grant databases, publication libraries) tend to get hit hardest.

One of the ways to deal with this is to tell crawlers to ignore URL parameters ( which are those strings like “?type=video&topic=housing” at the end of the URL). In this way, you’re telling the bot that if you’ve seen the news landing page and understand what’s here, you’ve seen everything.

General security hygiene

That’s a tough situation but even worse are the attacker bots. There are people out there flooding the internet with tiny programs looking for weaknesses. You can protect against them. This is what we do:

• We choose secure web hosts.
• We limit the number of plugins on any site. Every plugin is something that needs monitoring/updating and can be exploited.
• We make sure that all code is peer-reviewed before it’s checked in so that more human eyes are looking for mistakes.
• We protect the code repository so the live site can’t be deployed by accident.
• We push clients to use unique passwords, a password manager, and two-factor or multi-factor authentication wherever it’s offered.

Hosting on a read-only filesystem

Another thing you can do is to host your site on a mostly read-only filesystem. We host our WordPress and Drupal sites on Upsun (formerly Platform.sh). This isn’t the only host that does this, but it’s one reason we use it. We’re not getting paid by Upsun. We’re just fans.

So once the site is deployed, the filesystem remains read-only, with only a few directories the CMS can write to, such as file uploads (no JavaScript) and logs. Writing files to the server is one of the most common ways that attackers take over your site. This way, the host itself limits what a bad actor can do if they make it past your security.

Using Wordfence on WordPress sites

One thing we’ve been doing recently is to require Wordfence on every WordPress site we make. This protects against brute-force login attacks, enables two-factor authentication (and can require it), and can require strong passwords for anyone with admin/editor access. There’s a great free version that sends you alerts that can save you a lot of hassle.

Using a CDN in front of the site

Most of our sites sit behind Cloudflare. Again, we’re not paid by them. We’ve used Amazon CloudFront and Imperva. Even before AI was part of the story, we were recommending a CDN, which stands for content delivery network.

CDNs can deliver cached pages when your site goes down, so visitors might never notice when that happens. It also provides a layer of security between the internet and your server. You can filter traffic how you like to do it. And it makes it possible to connect people to servers closer to the visitor so pages load faster (which is good for SEO, as well).

The free tier of Cloudflare is great, and most organizations would benefit from it. We typically recommend the Pro tier because it provides more protections. They also have a program to provide Pro tier service to some organizations (see above).

Here’s how we use Cloudflare:

• We generally don’t block AI crawlers, but we do rate-limit them. Cloudflare lets you cap the number requests a bot makes in a given period of time. You stay findable without racking up server bills or slowing down/going down.
• We may block 10–20 countries, depending on where you do business, where a lot of bot activity originates. You’re probably not getting a lot of real traffic from North Korea or Russia. VPNs can get around this, but it helps.
• We use form protection, which is a lot like CAPTCHAs. Most of the time you don’t know it’s happening. Sometimes it asks visitors to check a box.

There are a lot of other benefits we don’t have time to get into (we’re happy to have a conversation about it), but a CDN is a way to have a security partner who knows more than you do, sitting there swatting away unwanted traffic. And they’re keeping up to date on all the new stuff, which would be impossible for a small team to do.