How do I fix robots txt?
Export the list of URLs from Google Search Console and sort them alphabetically. Go through the URLs and check if it includes URLs… That you want to have indexed. If this is the case, update your robots.
How do I verify a robots txt file?
Test your robots. txt file
- Open the tester tool for your site, and scroll through the robots.
- Type in the URL of a page on your site in the text box at the bottom of the page.
- Select the user-agent you want to simulate in the dropdown list to the right of the text box.
- Click the TEST button to test access.