Robots.txt

Robots.txt is a config format created in 1994 by Martijn Koster.

#2100on PLDB

30Years Old

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.

Robots.txt first developed in https://web.archive.org/web/20131029200350/http://inkdroid.org/tmp/www-talk/4113.html
Read more about Robots.txt on the web: 1. 2.

Example from the web:

User-agent: googlebot        # all Google services
Disallow: /private/          # disallow this directory

User-agent: googlebot-news   # only the news service
Disallow: /                  # disallow everything

User-agent: *                # any robot
Disallow: /something/        # disallow this directory

Feature	Supported	Token	Example
Comments	✓		# A comment
Line Comments	✓	#	# A comment

Robots.txt - Config format

Language features