POST/v1/map

Sitemaps, robots rules, feeds and link counts for a whole site, in one call. Plan a crawl before paying for it.

$0.005
sitemaps
robots
feeds

What it does

One paragraph, no pitch.

Discover a site's crawlable surface without crawling it: parse robots.txt allow/disallow rules, collect URLs from sitemap.xml and sitemap indexes, find RSS, Atom and JSON feeds, and count internal and external links on the homepage. Use the returned URLs to plan a /v1/scrape or /v1/crawl.

Price

Quoted by the 402 before anything runs.

$0.005per call
60 s default, 95 s maximumwhole-call budget
billedonly when content is delivered
no keythe payment is the credential

x402 settlement is live on eip155:8453: sign a USDC transfer and retry with PAYMENT-SIGNATURE.

When to call it

Cheapest endpoint that answers the question, first.

Call it when the sentence above is the whole job. Its neighbours solve the same URL with a different trade-off — a browser instead of plain HTTP, a site instead of one page, one section instead of a document. /choose walks the decision in thirteen routes, cheapest first, browser last.

Not sure the page is worth paying for at all? /v1/probe is free and returns the status, content type, size, title, language and whether a bot wall is waiting — no body.

Request fields

JSON body. Send this to https://readfetch.com/v1/map.

Field	Type	Default	What it does
`url` required	string	—	Public http(s) URL of the site to map.
`egress`	string	—	Egress pool to leave through: `direct` (the default route, no proxy) or any pool the operator configured, e.g. `residential` (the home exit) or `local`. Omit to route by host rules.
`egress_fallback`	string	—	Pool to retry through once when the first attempt is refused (bot check, wall, empty extraction, origin 403/429/5xx); must differ from `egress`. Absent keeps one exit for the whole call.
`limit`	integer	200	Maximum number of URLs to return (1-1000).
`timeout_s`	number	60	Total budget for the whole call in seconds; every sitemap fetch and the browser fallback share it. 95 s is the edge ceiling (Cloudflare closes at 100 s).
`max_bytes`	integer	5000000	Abort if any single document exceeds this many bytes.

Make the call

Unpaid, this answers 402 with the quote. Sign accepts[0] and resend with PAYMENT-SIGNATURE.

curl

curl -s https://readfetch.com/v1/map -H 'content-type: application/json' -d '{"url":"https://example.com"}'

json body

{
  "url": "https://example.com"
}

Response

Fields shown are the ones a caller branches on.

200 ok

{
  "feeds": [
    "https://example.com/feed.xml"
  ],
  "host": "example.com",
  "links": {
    "external": 6,
    "internal": 34
  },
  "robots": {
    "allow": [],
    "crawl_delay": null,
    "disallow": [
      "/admin"
    ],
    "sitemaps": [
      "https://example.com/sitemap.xml"
    ]
  },
  "sitemaps": [
    "https://example.com/sitemap.xml"
  ],
  "status": "success",
  "status_code": 200,
  "total_found": 2,
  "url": "https://example.com",
  "url_count": 2,
  "urls": [
    "https://example.com/",
    "https://example.com/blog/post"
  ]
}

Limits

Requests are JSON, 1 MiB maximum. Redirects are re-validated at every hop and capped at 5; the final URL is reported. 60 s default, 95 s maximum is the whole-call budget — DNS, connect, redirects, retries and any render share it, and no step starts a fresh clock.

Failures come back as {"error": "…", "hint": "…"} with a status to branch on: 402 for the quote, 429 with Retry-After, 502 and 504 for the upstream. /docs lists the shapes.

Billing

Settled after the work, only when something was delivered.

$0.005 per delivered call. A response with billed: false settles nothing and names the next call to try.

billed: false means nothing settled: a bot wall, an empty extraction, an unsupported type or a transport failure. The hint on that response names the next call to try.

Every settlement carries a signed RECEIPT bound to the transaction. If a paid call delivered nothing, a claim issues a single-use credit for it — the rail cannot refund.