Hello and welcome to this guide/journal of adding a script to your website in the attempt to stop AI bots from scraping your prized info.
Before I start, a few disclaimers, I’m not a trained computer scientist or software engineer. I have a dabbling of experience with frontend coding and some wobbly attempts at JS. Nevertheless, hopefully this will be helpful to those with similar to no experience adding custom scripts to their websites. And/or maybe this will prompt a thought as to what the uninhibited use of generative AI and LLMs means for the great information landscape of the worldwide web, plus maybe your internet searchable data.
In this guide/journal I detail how I went about adding blockers and the references that informed the process. It’s not very long, because I wasn’t able to do all the steps lol. Read on to find out why.
Finally before we script this, from what I’ve learnt, these methods might not be full proof as many crawlers and bots can and do ignore them. Anyway, in my opinion it’s worth a try, especially when one’s original creative work is on the line.
One: Block crawlers in robots.txt file
Recommended as the first and easiest step one can take, let’s go.
Here is the script:
User-agent: GPTBot Disallow: /User-agent: ChatGPT-User Disallow: /User-agent: ClaudeBot Disallow: /User-agent: anthropic-ai Disallow: /User-agent: Google-Extended Disallow: /User-agent: CCBot Disallow: /User-agent: PerplexityBot Disallow: /User-agent: Bytespider Disallow: /User-agent: Meta-ExternalAgent Disallow: /
To add to your website’s robots.txt:
Many website builders will have a robots.txt file available to edit. Start here, add the script below what’s already written and save. Check if the update has been applied by going to [yourwebsitename]/robots.txt
Seems so easy until you’re lost. Which was me when I was unable to access the robots.txt in my website builder. So I reached out to the knowledgeable support chat to find out more. Editing the robots.txt is not possible in the editing platform, so they recommended adding a more restrictive meta tag. Again, they reiterated, like many of the blog posts, that due to the current unregulated nature of AI and their crawling bots, these are not fire-wall proof solutions.
Two: Update Meta tags and headers
This is the route I was able to take, and would encourage you to do add this along with method one.
In the HTML file, add the following inside each page’s <head> :
<meta name="robots" content="noai, noimageai">The website builder I use has a handy Meta Tags editor in the Publish Settings window. It was a quick copy/paste and save.
To check if the meta tag has been saved to your website’s code, you can do the following in your web browser:
1. While on the homepage of your website, right click and scroll down to Inspect
2. Click Inspect, and developer tools window will open on the right hand side
3. Under the Elements section you can view the HTML code
4. Scroll to the top of the window with all the code until you see <!DOCTYPE html>
5. Find <head>...<head> below the html opening tag
6. Click the arrow to open the <head> tag
7. Is your new meta tag there? Hooray! - if not, double check you’ve published/saved your recent updates
Three: IP Address Blocking
In progress, updates to follow soon...
References:
Falokun, Christine. “How to Block AI Bots, Llms, Scrapers and Crawlers.” DataDome, 27 May 2025, https://datadome.co/learning-center/block-ai-bots/. Accessed 10 Aug. 2026.
Google Search Central. “Robots.Txt Introduction and Guide | Google Search Central | Documentation.” 2025.
Tester, Paige. “How to Block Web Crawlers and AI Bots From Your Website.” Security Boulevard, 20 Mar. 2026, https://securityboulevard.com/2026/03/how-to-block-web-crawlers-and-ai-bots-from-your-website/. Accessed 10 Aug. 2026.