Build a Bot Detection System with the User Agent API

Bots account for a significant share of web traffic -- search engine crawlers, SEO tools, uptime monitors, and scrapers all hit your endpoints. Distinguishing bots from real users is critical for accurate analytics, rate limiting, and security. The Apixies User Agent Inspector API returns an is_bot flag alongside full browser and OS details, giving you a reliable way to classify traffic without maintaining your own bot signature database.

Why UA-Based Bot Detection?

There are many approaches to bot detection (JavaScript challenges, CAPTCHAs, behavioral analysis), but UA string parsing is the fastest and simplest first layer:

UA parsing catches the vast majority of legitimate crawlers (Googlebot, Bingbot, etc.) and many malicious scrapers that use known bot signatures. It's ideal as a first-pass filter before more expensive checks.

Step-by-Step: Building a Bot Filter

Step 1 -- Parse the Incoming UA

Extract the User-Agent header from the request and send it to the API:

async function classifyRequest(req) {
  const ua = req.headers["user-agent"] || "";

  const response = await fetch(
    `https://apixies.io/api/v1/inspect-user-agent?user_agent=${encodeURIComponent(ua)}`,
    { headers: { "X-API-Key": process.env.APIXIES_KEY } }
  );

  return (await response.json()).data;
}

Step 2 -- Route Based on Bot Status

Use the is_bot flag to handle bot and human traffic differently:

app.get("/api/data", async (req, res) => {
  const parsed = await classifyRequest(req);

  if (parsed.is_bot) {
    // Log bot visit, apply stricter rate limits, or return cached content
    console.log(`Bot detected: ${parsed.browser.family}`);
    return res.json({ data: getCachedResponse() });
  }

  // Full processing for human visitors
  return res.json({ data: await getFullResponse() });
});

Step 3 -- Build an Allow/Block List

Some bots are welcome (search engine crawlers), while others should be blocked. Use the parsed browser family to decide:

import requests
import os

api_key = os.environ["APIXIES_KEY"]

ALLOWED_BOTS = {"Googlebot", "Bingbot", "DuckDuckBot", "Slurp", "facebookexternalhit"}
BLOCKED_BOTS = {"AhrefsBot", "SemrushBot", "MJ12bot", "DotBot"}

def check_bot(ua_string):
    response = requests.get(
        "https://apixies.io/api/v1/inspect-user-agent",
        params={"user_agent": ua_string},
        headers={"X-API-Key": api_key},
    )
    data = response.json()["data"]

    if not data["is_bot"]:
        return "human"

    bot_name = data["browser"]["family"]
    if bot_name in BLOCKED_BOTS:
        return "blocked"
    if bot_name in ALLOWED_BOTS:
        return "allowed"
    return "unknown_bot"

Step 4 -- Middleware Integration (Express.js)

Wrap the bot check in middleware so it runs on every request:

async function botFilter(req, res, next) {
  const ua = req.headers["user-agent"] || "";

  // Skip parsing for known static assets
  if (req.path.match(/\.(css|js|png|jpg|ico)$/)) {
    return next();
  }

  try {
    const parsed = await classifyRequest(req);
    req.isBot = parsed.is_bot;
    req.botName = parsed.browser?.family || "Unknown";
    req.parsedUA = parsed;
  } catch {
    // Fail open -- don't block traffic if API is unreachable
    req.isBot = false;
  }

  next();
}

app.use(botFilter);

Logging Bot Traffic for Analytics

Track bot visits separately to keep your analytics clean:

app.use((req, res, next) => {
  if (req.isBot) {
    logBotVisit({
      bot: req.botName,
      path: req.path,
      timestamp: new Date(),
      ip: req.ip,
    });
  }
  next();
});

This lets you answer questions like "How often does Googlebot crawl us?" and "Which pages do scrapers target?" without polluting your user analytics.

Next Steps

• User Agent Inspector API documentation -- full parameter reference and error codes.
• User Agent Parser API Tutorial -- complete getting-started guide.
• Browser Analytics with the User Agent Parser API -- build a traffic breakdown dashboard.
• All guides -- browse every tutorial and use case.

Get your free API key and start filtering bots today.