A Multisite Robots.txt Manager Plugin for WordPress

A Multisite Robots.txt Manager for WordPress. Quickly manage all Network Websites robots.txt files directly within the Network Admin.

This Plugin Was Created For Multisite Networks > Network Activations Only!

Multisite Robots.txt Manager WordPress Plugin

 

Features

Fast, easy to use, and most important – it saves you time!

  • Manage all Network Websites from one Administration area.
  • Instantly add Sitemaps URLs to all Network Websites.
  • Mass update the entire Network or update a unique Website.
  • Each Website within a Network can have a unique robots.txt file.
  • Robots.txt file examples that can be published to Websites or stored as a default.
 

Quick Info

Before you start, you should know…

  • When you first install the plugin, no robots.txt files are active.
  • The default “Network Wide” robots.txt file is NOT a live robots.txt file.
  • If you deactivate the plugin, no options are removed but the robots.txt file(s) are no longer displayed.
  • If you delete this plugin, all options and settings will be removed from the database.

Making the Plugin Work

To make the robots.txt live for a Website, either click the “publish to network” button or select the Website from the dropdown > then click the “change sites” buttons. Next, adjust the displayed robots.txt file, then click the “update this website” button. Both methods publish the robots.txt file to the Website(s), making it live. Click the [ view robots.txt ] link next to the Websites dropdown to view the changes within your browser.

The best way to learn how the tool works is to play around with the features. Start by creating your robots.txt file (even a junk one), network publish it, delete it, rebuild the default(s)….and when you’re ready, set it up how you want, mass update the Network, then enjoy the rest of the day!

Download > Install > Network Activate > Settings > MS Robots.txt


Download The Plugin

Download at WordPress.org or within your WordPress Admin, access the Plugins Menu > Add New Button and search for: MS Robots.txt


Install through the WordPress Admin

  • It is recommended that you use the built in WordPress installer to install plugins.
    • Access the Network Admin > Plugins Menu > Add New Button
  • In the Search box, enter: MS Robots.txt
  • Click Install Now and proceed through the plugin setup process.
    • Activate / Network Activate the plugin when asked.
    • If you returned to the Plugin Admin, locate “MS Robots.txt” Plugin and click the Activate link.

FTP Upload and Setup

  • If uploading, upload the /ms_robotstxt_manager/ folder to /wp-content/plugins/ folder for your Worpdress install.
  • Then open the WordPress Admin:
    • Access the Network Admin > Plugins Menu > Add New Button
  • Locate the MS Robots.txt Manager Plugin in your listing of plugins. (sort by Inactive)
  • Click the Activate link to start the plugin.

Configuring

When you first enter the MS Robots.txt Settings page, the shown robots.txt file is the default “network only” or “network wide”
working copy. Modify the default robots.txt file, save the default file, and when ready click the “publish to network” button to duplicate the robots.txt file to all Websites within the Network.

Sitemap URLs and Structure

Most WordPress Sitemap Plugins can automatically add the Sitemap URL’s to the robots.txt file for you. Be sure to disable this feature within other Plugins if you use the feature within this plugin, otherwise more than one sitemap url will be listed in the robots.txt file.

To add a Sitemap URL to a Robots.txt file, simply select the checkbox to add the sitemap url, then enter the Sitemap URL Structure to use.

Almost all WordPress Installs will use: http://WEBSITE_URL/sitemap.xml

The [brackets] within the Sitemap URL’s automatically get replaced by the plugin (You Will Use Them). Network Wide Sitemap Updates “must” use the [bracket] structure to ensure Websites have the proper Sitemap URL. Unique Website updates can use the brackets OR take the full sitemap url directly.

Sitemap URLs and Structure

  • WordPress Sitemap URLs: http://WEBSITE_URL/sitemap.xml
  • GoDaddy Sitemap URLs: http://WEBSITE_URL/sitemaps/DOMAIN-EXT.xml
  • Random Example: http://WEBSITE_URL/DOMAIN-EXT-sitemap.xml.gz

Structure Meaning:

  • [WEBSITE_URL] = domain.com
  • [DOMAIN] = domain
  • [EXT] = .com/net, etc.

Always include the http:// with the Sitemap URL Structure.
If the sitemaps are within a directory, /include-the-path/ within the sitemap url.

New Website Added to Network

If every Website uses the Networks default robots.txt file, simply click the “publish to network” button to copy the default robots.txt file over to any new Websites you have.

Per Site: Change to the Website in the dropdown. Then click the “reset this website” button to copy the default robots.txt file to this Website. If needed, modify the robots.txt file and click the “update this website” button once done.

Manage a Websites Robots.txt File

To manage a Website directly, select the Website from the dropdown, then click the “change sites” button. This will display the robots.txt file for the selected Website. Change the robots.txt file how you like, once done click the “update this website” button to publish the modification.

Disable a Website

To disable the MS Robots.txt Manager on a Website, click the “disable this website” button. This will clear the option settings for this Website, making the WordPress default robots.txt file display.

Reseting

Reset Default: Something wrong? No worries! When viewing the Networks robots.txt file, click the “reset to default” button to replace the displayed robots.txt file with the core “coded in” default robots.txt file.

Reset Website: To reset a Websites robots.txt file, change to the Website within the dropdown, then click the “reset this website” button to pull in the “Networks Default Robots.txt file” (not the coded in default file).

Presets / Examples Tab

Use the provided examples to create your own robots.txt file…. or within the dropdown, select either the Networks Robots.txt file or a Websites Robots.txt file, then click the “set as default” button to copy the example over, to the selected file.

Presets can also use the Sitemap URL Structure setting. Read above on how to use this feature.

Sitemaps and Sitemap Plugins

The Multisite Robots.txt Manager plugin has been tested with:
Google XML Sitemaps with Multisite support and BWP Google XML Sitemaps.