Google Working On Improving Robots.txt Tester Tool

Gary Illyes from Google confirmed he’s slowly engaged on bettering the robots.txt tester system, which is presently on hand on the extinct model of Google Search Console. He said on Twitter “we’re presently in the system of integrating the tester with the manufacturing model of the parser.”

Gary said this on yarn of there became once a criticism that there could be a worm with either the tester system or how Google reads a mumble’s robots.txt file. Gary said it could maybe be perfect the system itself and when the system is up to this point to make utilize of the manufacturing model of the parser, you’re going to perceive take into accout that.

A reminder that Gary became once inquisitive about the robots exclusion protocol long-established project after which commence sourcing the parser code.

Here is Gary’s tweet:

we'll discover about, but it indubitably's likely a system error. we're presently in the system of integrating the tester with the manufacturing model of the parser which should fix this, if certainly it is miles a system error.

– it's my procrastination project, so no keeping breaths

— Gary 鯨理／경리 Illyes (@methode) January 6, 2021

Must you scroll up by means of the Twitter thread, that you need to most likely learn more about the criticism.

In 2014, Google revamped its robots.txt tester system but it indubitably has been up to this point rather loads of times sooner than and after that time.

Forum dialogue at Twitter.