Is your online store blocking ChatGPT by accident?

Almost every website has a small file called robots.txt. It is a short text file that tells visiting crawlers what they may and may not look at. It has usually been there since the day your site was built, and most owners have never opened it.

The problem in one line

That file can contain a line that says: all crawlers, stay out of everything. Sometimes that was set on purpose, for example while the site was still being built, and then forgotten. But it also hits the crawlers of ChatGPT, Gemini and Perplexity. They are told at the door that they are not allowed in, and they leave without ever seeing your products.

How to check it yourself

Type your own web address in the browser and add /robots.txt at the end. For example yourstore.nl/robots.txt. You will see a plain text screen.

Look for lines with the name of an AI crawler, such as GPTBot, ClaudeBot, PerplexityBot or Google-Extended. If one of them has a line with Disallow followed by a forward slash, that crawler is blocked completely. If there is no robots.txt at all, there is usually nothing wrong, because then everyone is simply allowed in.

What to do about it

If you want to let AI assistants in, the file needs to say for each crawler that it is welcome. That takes a separate block per crawler: a line User-agent with the name of the crawler (for example GPTBot), and below it Allow followed by a forward slash.

If you are not sure about this, ask whoever built your website to do it. It is a change of a few minutes, and it is exactly the kind of thing you regret later if it has been wrong for months.

A choice, not an obligation

To be fair: some businesses do not want AI to use their content, and block these crawlers on purpose. That is a legitimate choice. The point is that it should be a choice, not something that happens by accident while you miss out on customers.