Free Robots.txt Parser API - Crawl Rules

Parameters

name	in	type	required	description
`domain`	query	string	required	Without https://

Request

Send your key in the X-API-Key header. More languages and patterns live on the code examples page.

curl -s "https://apixies.io/api/v1/robots-txt?domain=github.com" \
  -H "X-API-Key: YOUR_API_KEY"

const res = await fetch("https://apixies.io/api/v1/robots-txt?domain=github.com", {
  headers: { "X-API-Key": "YOUR_API_KEY" }
});
const data = await res.json();
console.log(data);

$ch = curl_init("https://apixies.io/api/v1/robots-txt?domain=github.com");
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => ["X-API-Key: YOUR_API_KEY"],
]);
$data = json_decode(curl_exec($ch), true);
curl_close($ch);

print_r($data);

import requests

res = requests.get(
    "https://apixies.io/api/v1/robots-txt",
    params={"domain": "github.com"},
    headers={"X-API-Key": "YOUR_API_KEY"},
)
print(res.json())

Response

The result sits under data in the standard envelope. See the response format guide for errors and status codes.

returns

data.user_agents: Rules per user agent with allow, disallow and crawl_delay.
data.sitemaps: Sitemap URLs listed in the file.
data.found: False when the site doesn't have a robots.txt.
data.raw: The file exactly as served.
data.status_code: HTTP status of the robots.txt request.

example response 200

{
    "status": "success",
    "http_code": 200,
    "code": "SUCCESS",
    "message": "Robots.txt parsed successfully",
    "data": {
        "domain": "laravel.com",
        "url": "https://laravel.com/robots.txt",
        "found": true,
        "status_code": 200,
        "raw": "User-agent: *\nDisallow: /cdn-cgi/\nContent-Signal: ai-train=no, search=yes, ai-input=yes\n",
        "user_agents": {
            "*": {
                "allow": [],
                "disallow": [
                    "/cdn-cgi/"
                ],
                "crawl_delay": null
            }
        },
        "sitemaps": [],
        "parsed_at": "2026-09-18T12:00:00+00:00"
    }
}

Authentication and limits

This endpoint needs an API key. Use the sandbox ("try it" above) to test it without registering. See the authentication guide for details.

Standard rate limits apply to this endpoint.

Common use cases

SEO crawlability auditing

Analyze robots.txt files across your domains to ensure important pages aren't accidentally blocked from search engine crawlers.

Competitor analysis

Examine competitors' robots.txt files to understand their crawl strategy, discover hidden sitemaps, and identify sections they prioritize or restrict.

Multi-site governance

Monitor robots.txt configurations across all your organization's domains. Alert when changes could affect SEO or expose restricted content.

related endpoints

GET Meta Tag Extractor Extract OpenGraph, Twitter Card, and standard meta tags from any URL GET Link Checker Check if a URL is reachable and return its HTTP status code and response time GET DNS Lookup Resolve DNS records (A, AAAA, MX, TXT, CNAME, NS, SOA, SRV, CAA, PTR) for any domain

getting started

authentication code examples response format

explore

all api endpoints guides and tutorials online tools

resources

homepage changelog suggest an endpoint

Get a free API key

A temporary key takes one click and no signup. Register and you get 75 requests/day. Free tier is for development & small projects.

get a temporary key create an account