Google has released several updates to both the robots.txt documentation and the structured data documentation last Friday. Here is a list of those changes from Google’s site.
- Simplified the introduction page to make it clearer what is robots.txt and what is its intended use.
- Expanded the instructions about creating and updating robots.txt files.
- Removed redundant sections from our documentation about how Google handles robots.txt.
- Simplified sentences across all robots.txt documentation in English. This helps with localization.
- Removed redundant documentation about how to build, test, and release structured data. Each feature has guidance about this, embedded directly into each guide (for example, the Video structured data guide.
- Improved the troubleshooting sections across all structured data feature guides (for example, the Product structure data guide.
- Added more beginner-friendly information about how to get started with structured data.
Here is some clarity on why Google made the changes from Lizzi Harvey and Gary Illyes of Google:
@methode oh come on. there were only 148 comments, and a 4 page doc.
— Lizzi Harvey (@LizziHarvey) June 18, 2021
Insane, people sent in 148 comments that the both of them used to revise the robots.txt documents. Lizzi added that they also ended up “fixing issues across 20+ structured data docs.”
Forum discussion at Twitter.