Jump to content

Bing(bot) and Google(bot)


Recommended Posts

Yes, when we did our last system update, a few months ago, we got a new capability treat search engine bots as "fake" users of the board.


It allows is to control how bots can access our site and it can keep them out of the parts of the site that we don't want them to see.


Here is some of the documentation from phpBB, the makers our our software:


3.10.2. Managing Search Robots


phpBB 3 introduced a new system for managing search indexer and bot accounts. It allows you to identify these automated bots by their IP or a part of their user-agent, which is a setting that normally identifies the browser of a user. After you add a bot and it is recognized, phpBB does not treat the session as anonymous, but uses the created bot account. Bots use permissions set by the predefined Bots group. Identifying bots is important so that phpBB can serve them content which is more appropriate for search engines - dead links to pages without content are omitted, e.g. posting pages, report post pages etc. Bots never receive a session ID in the URL, which should not appear in the search results. You can also assign a specific style and language to bots.

Link to comment
Share on other sites

That's how Google (or other search engines) work.


They send out crawlers (or bots) to search the web, automatically recording the web pages they find and categorizing them according to the content on each page. It's like a gigantic card catalog like you used to find in the library.


If you want to be able to find information on the internet, Google (et. al) has to index it.


FYI: The private parts of our website like the "Fur Freaks" section and the Photo Gallery are inaccessible to the crawlers. You might be able to go on Google, Bing or Yahoo and search for things on The Fur Den but you should NOT be able to find anything that has been posted in Fur Freaks, the Gallery or the private parts of the Wiki, such as the stories section.


We lock the bots out of those sections and we use those "fake" user accounts to make it easier to do.

Link to comment
Share on other sites

Another point about spiders and crawlers:

Search engine bots will follow any links they encounter. So the more sites that link to a particular site (or specific web page) generally results in a higher ranking of the site or page on the search service.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...