• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

WBXPress.NET

Website Developement Guide

  • apache2
  • nginx
  • mysql
  • php7
  • SSL
  • wordpress
  • genesis
  • caching
  • seo

Optimized robots.txt file for WordPress

Updated on 27th August 2017 7 Comments

You should create robots.txt in the root of your wordpress installation if you don’t have this file.

Typical Example of WordPress robots.txt

sitemap: https://wbxpress.net/sitemap.xml

User-agent:  *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /archives/
Disallow: /*?*
Disallow: *?replytocom
Disallow: /comments/feed/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

Filed Under: wordpress Tagged With: seo

Reader Interactions

Comments

  1. Arsh Sran says

    12th September 2017 at 1:24 am

    Bro ! What is the use of this robots file ?
    Better idexing ??

    Reply
    • wbxpress says

      17th September 2017 at 9:34 pm

      This file helps search engine robots, what to index and what not.

      Reply
  2. avi says

    14th December 2017 at 10:52 pm

    how we can use it on blogger

    Reply
  3. sachin says

    10th February 2018 at 12:32 pm

    hi how can i change my robot.txt so that it will not show my backlinks in moz and ahref.

    Reply
  4. maurya says

    26th February 2018 at 9:45 pm

    hey how to use robot txt file and how can i use…

    Reply
  5. Mohammed Siraj says

    18th May 2018 at 3:02 pm

    can we use same for blogspot blogs. ?

    Reply
    • wbxpress says

      19th May 2018 at 7:09 pm

      yes you can.

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Most Popular Posts

  • Remove/ Defer Render-blocking JavaScripts
  • Customizing Genesis eleven40-pro Theme
  • Free Genesis Child Theme by WBXPress
  • Host Multiple Sites with SSL in Ubuntu 18.04
  • 25 Ways to Boost Website Traffic that Really Work

Recent Comments

  • Ruhul on Customizing Genesis eleven40-pro Theme
  • Priya Agarwal on Customizing Genesis eleven40-pro Theme
  • mimi roy on Remove/ Defer Render-blocking JavaScripts
  • danish choudhary on Customizing Genesis eleven40-pro Theme
  • Radhe on Customizing Genesis eleven40-pro Theme

Copyright © 2021 · Powered by WordPress · Log in