You should create robots.txt in the root of your wordpress installation if you don’t have this file.
Typical Example of WordPress robots.txt
sitemap: https://wbxpress.net/sitemap.xml User-agent: * # disallow all files in these directories Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /archives/ Disallow: /*?* Disallow: *?replytocom Disallow: /comments/feed/ User-agent: Mediapartners-Google* Allow: / User-agent: Googlebot-Image Allow: /wp-content/uploads/ User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Mobile Allow: /
Arsh Sran says
12th September 2017 at 1:24 amBro ! What is the use of this robots file ?
Better idexing ??
wbxpress says
17th September 2017 at 9:34 pmThis file helps search engine robots, what to index and what not.
avi says
14th December 2017 at 10:52 pmhow we can use it on blogger
sachin says
10th February 2018 at 12:32 pmhi how can i change my robot.txt so that it will not show my backlinks in moz and ahref.
maurya says
26th February 2018 at 9:45 pmhey how to use robot txt file and how can i use…
Mohammed Siraj says
18th May 2018 at 3:02 pmcan we use same for blogspot blogs. ?
wbxpress says
19th May 2018 at 7:09 pmyes you can.