# Bot protection

Keitaro automatically detects bots by the following attributes:

  • IP is in the bot database of Keitaro
  • IP is in the user's list
  • UserAgent in the bot database of Keitaro
  • Part of UserAgent in the user's list
  • Spammers myip.ms in the black list (disabled by default)
  • Empty UserAgent

You can change the settings at the page “Maintenance > Settings > Bots”.

# How to protect a campaign from bots

You need to create a special stream in the campaign.

Create a stream with the type “Forced” and add the filter “Bots” in the “IS” mode. Сhoose the necessary action. For example, “Show 404 not found”.

# What types of bots are there in the bot database?

  • Bots of search engines: Google, Bing and others;
  • Bots of advertising services: Google Adwords, Facebook;
  • Bots of ad monitoring systems and disclosing of links.

# How to update, and how often are the bot databases updated?

The bot databases are updated on the page “Maintenance > Geo DBs”. Updates are issued 2 times a month.

# How to add custom bot list?

At the page “Maintenance > Settings > Bots”, press “Edit list” . You can write IPs and CIDR. Other formats are not supported.

Example:

78.12.31.1
92.12.1.0/24

Keitaro optimizes the list. It's okay when uploading 100.000, then you see 50.000 after saving.

# How to add a large number of IPs to bot list?

  1. Upload the file to the server. For example as file /var/www/keitaro/file.txt.
  2. Run these commands
cd /var/www/keitaro
php bin/cli.php bots:import file.txt

# What is the signature of bots?

It is a part of UserAgent. For example, Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)is the signature of Googlebot.

# How to add signature of bots

Open the page Settings > Bots. You can set the signatures separated by commas:

signature1, signature2, ...

# How to detect bots through a provider

The provider database is not supplied with Keitaro due to limitations imposed by the IP2Location license agreement.

You can connect the database in two ways:

  1. Buy the license. Enter the license key at the page “Maintenance > Settings > Integration”.
  2. Download the GeoISP.dat file manually to the directory /var/geoip/GeoISP/.
  3. After integrating the database, you can use the “ISP” filter in the streams.

# Protecting your ads from bots

  1. Create a campaign.
  2. Create a stream:
  3. Type «Forced»
  4. Add the filter “Bots: YES”
  5. Choose a scheme “Direct URL” and choose the redirect type “CURL”, paste here the URL of the page that you want to show for bots and moderators (“Show HTML” with the page code is also suitable)

IMPORTANT! Keitaro does not give 100% guarantee of catching bots. Improving the security limits streams for the real users with filters.

# Protecting a Website

We can protect a website from bots. To do that, we integrate KClient PHP to the website.

  1. Prepare a HTML page that you want to show to bots.
  2. Create a campaign.
  3. Create a stream for bots:
  4. Choose type “Regular”.
  5. Choose schema “Direct URL”.
  6. Choose redirect type “CURL”.
  7. Enter bot page URL to the URL field
  8. Add a filter “Bots: Yes”
  9. Create a stream for others:
  • Type “Regular”
  • Schema “Action”
  • Action “Do nothing”
  1. Open the page “Additional > Integration”.
  2. Choose “ KClient PHP”;
  3. Follow the instructions to connect the code to your website.
  4. Replace in the code $client->execute(); to $client->executeAndBreak();.

# Protecting a Static Website

Solution for websites without PHP, where the website acts as a secure page.

  1. Create a campaign;
  2. Create a stream for bots:
  3. Stream type: “Forced”;
  4. Action “Do nothing”;
  5. Add the filter “Bot: Yes”;
  6. Create a second stream.
  7. Stream type “Regular”;
  8. Scheme “Redirect”;
  9. Redirect type “Redirect for Script” (permissible also: Meta redirect and Double Meta redirect).
  10. Open the page “Integrate” (“Additional” menu).
  11. Choose “KClient JS”;
  12. Connect the code to your website according to the instruction.

# How to check protection

Use the browser extension that replaces UserAgent. For example, “UserAgent Switcher” for Chrome.

Or you can use traffic simulation.

# Protection Tips

  • Regularly update Bot DB on page “Maintenance > Geo-DBs”.
  • Always remember to create a forced stream for bots (with a filter Bots+yes).
  • Restrict your streams by filters like Country and device type.
  • Use multiple domains or subdomains.
  • Make sure your domain is not flagged. To check that use these tools http://urlvoid.com, http://virustotal.com.
Last Updated: 4/13/2020, 2:00:02 PM