Hello and welcome to this guide/journal of adding a script to your website in the attempt to stop AI bots from scraping your prized info.
Before I start, a few disclaimers, I’m not a trained computer scientist or software engineer. I have a dabbling of experience with frontend coding and some wobbly attempts at JS. Nevertheless, hopefully this will be helpful to those with similar to no experience adding custom scripts to their websites. And/or maybe this will prompt a thought as to what the uninhibited reach of generative AI and LLMs means for the great information landscape of the worldwide web, plus your internet searchable data.
In this guide/journal I detail how I went about adding blockers and the references that informed the process. It’s not very long, because I wasn’t able to do all the steps lol. Read on to find out why.
Finally before we script this, from what I’ve learnt, these methods might not be full proof as many crawlers and bots can and do ignore them. Anyway, in my opinion it’s worth a try, especially when one’s original creative work is on the line.
One: Block crawlers in robots.txt file
Recommended as the first and easiest step one can take, let’s go.
Here is the script:
User-agent: GPTBot Disallow: /User-agent: ChatGPT-User Disallow: /User-agent: ClaudeBot Disallow: /User-agent: anthropic-ai Disallow: /User-agent: Google-Extended Disallow: /User-agent: CCBot Disallow: /User-agent: PerplexityBot Disallow: /User-agent: Bytespider Disallow: /User-agent: Meta-ExternalAgent Disallow: /
To add to your website’s robots.txt:
Many website builders will have a robots.txt file available to edit. Start there, add the script below what’s already written and save. Check if the update has been applied by going to insertyourwebsitenamehere/robots.txt
Seems so easy until you’re lost. Which was me when I was unable to access the robots.txt in my website builder. So I reached out to the knowledgeable support chat to find out more. Editing the robots.txt is not possible in the editing platform, so they recommended adding a more restrictive meta tag. Again, they reiterated, like many of the resources, that due to the current unregulated nature of AI and their crawling bots, these are not fire-wall proof solutions.
Two: Update Meta tags and headers
This is the route I was able to take, and would encourage you to do add this along with method one.
In the HTML file, add the following inside each page’s <head> :
<meta name="robots" content="noai, noimageai">The website builder I use has a handy Meta Tags editor in the Publish Settings window. It was a quick copy/paste and save.
To check if the meta tag has been saved to your website’s code, you can do the following in your web browser:
1. While on the homepage of your website, right click and scroll down to Inspect
2. Click Inspect, and the developer tools window will open on the right hand side
3. Click on Elements to view the HTML code of the page
4. Scroll to the top of the window with all the code until you see <!DOCTYPE html>
5. Scroll to <head>...<head> below the html opening tag
6. Click the arrow to open the <head> tag
7. Is your new meta tag there? Hooray! - if not, double check you’ve published/saved your recent updates
Three to Six: IP Address Blocking, Server level User-agent Filtering, Rate Limiting and Tarpits
Stepping into another side of your website infrastructure here. The recommended methods that follow one and two, delve into the server-side and DNS settings of your website. Not a place I have often frequented with success. If you have some trusty tech support at hand, why not get them involved. If not, and you’re eager to stop the bots, consider method seven.
Seven: Use a Bot Management Platform
There are a few options out there, mostly for a fee. The benefit of using a third-party service is a (hopefully) easy installation process, followed by extensive bot blocking. A recommendation would be to understand how often they update their methods, and the combination of techniques used to halt increasingly sophisticated AI bots.
Finally, what began as a quick search into what I was hoping would be a quick fix, plus a few words aimed at sharing the research, highlighted the vast reach of Large Language Models (LLMs), AI companies and their robots (bots), that with adept code can increasingly leap through the loopholes of blockers. Leaving with a parting thought, what giant content soup is being created, who ultimately benefits and at what expense?
References:
Falokun, Christine. “How to Block AI Bots, Llms, Scrapers and Crawlers.” DataDome, 27 May 2025, https://datadome.co/learning-center/block-ai-bots/. Accessed 10 Aug. 2026.
Google Search Central. “Robots.Txt Introduction and Guide | Google Search Central | Documentation.” 2025.
Tester, Paige. “How to Block Web Crawlers and AI Bots From Your Website.” Security Boulevard, 20 Mar. 2026, https://securityboulevard.com/2026/03/how-to-block-web-crawlers-and-ai-bots-from-your-website/. Accessed 10 Aug. 2026.