Website Design

Website Marketing for Lawyers: How to Stop Bots from Crawling Your Website | Good2bSocial media

Generally speaking, you want your website content to be as “explorable” as possible. It’s important that spiders – like Google’s – can see your site quickly and easily. However, there may be times when you want to block bots. Keep reading to find out why you would want to block certain bots and how to do it to improve website marketing for lawyers.

Website Marketing for Lawyers
website marketing for lawyers

What is a bot?

A lot of people don’t really know what a bot is and that makes them hard to tell. Short for “robot”, a bot is a software application designed to repeat a particular task over and over again. SEO professionals can use bots to scale their SEO campaigns by automating as many tasks as possible. They can help digital teams work smarter instead of harder, for example by fetching useful data from search engines.

Are bots and spiders harmless?

For the most part, spiders and robots are harmless. You actually need it in many cases. For example, you need Google robots to crawl and index your site in order to appear in searches. Sometimes, however, bots can cause problems and provide unwanted traffic. It is important because:

  • They can confuse where your traffic is coming from.
  • They can blur reports and make them difficult to understand (and less useful).
  • You may encounter misattribution in Google Analytics.
  • Bandwidth can be increased to accommodate additional traffic, which can increase your costs.
  • Unwanted traffic can lead to other small nuisances that require resources to manage.

Essentially, there are good bots and bad bots. The bots you want run in the background and don’t attack another user or website. Bad bots, on the other hand, break the security behind a website and can be used as a large-scale botnet to serve DDOS attacks against certain organizations. In these cases, a botnet can do what a single machine could not.

By preventing certain bots from visiting your site, you can protect your data and see other benefits such as:

  • Securing sensitive customer data and other information from forms
  • Prevent software from taking advantage of a security hole to add bad links to your site
  • Limit bandwidth costs by preventing an influx of unwanted traffic

How to stop bad bots from crawling your site

Fortunately, there are things you can do to reduce the chances of negative bots entering your website. It’s not easy to discover all the bots capable of crawling your site, but you can usually find some malicious ones that you don’t want to visit.

One method is to use robots.txt. This is a file that resides on your web server database. Sometimes it’s there by default, but usually it needs to be created. Here are some files that you might find useful.

1. To ban Googlebot from your server

Note: Don’t use this one lightly. This is for cases where you want to prevent Googlebot from crawling your server, such as preventing it from crawling your staging site.

  • User agent: Googlebot
  • To refuse:/

2. To ban all bots from your server

To completely avoid bots, use this code. You can use this when you want to keep your site private for a while before a full-scale launch.

  • User agent: *
  • To forbid : /

3. To prevent bots from crawling a specific folder

You might want to prevent bots from crawling a certain folder. To do this, use this code:

  • User agent: *
  • Forbid: /folder-name/

It is important to avoid some mistakes that are common. Major errors include using both disallow in robots.txt and noindex, not including the correct path, or not testing the robots.txt file.

There are other bot blocking methods that allow users to be more specific about which bots to block and how to do it, but they can get quite technical. Unless you have a developer on staff, you might want to ask your web design partner for advice.


Blocking bots and spiders requires a few extra steps, but it’s worth it. This keeps your site secure and ensures that you don’t fall into certain traps. By controlling some robots, you have a better ability to automate your SEO processes and improve website marketing for lawyers. All of these things will make for a much stronger site that will be useful and optimized for years to come.

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button