Web search API vs. page fetch for LLM research: why snippets are not enough

For claims that depend on dates, conditions or surrounding context, a search snippet alone is insufficient evidence for an LLM research answer. Use a web search API to discover candidate URLs, then fetch the selected pages to read their content. Retain both the search record and the extracted passages, so each claim can be traced to a source the application actually read.

This separation also helps scheduled monitors and dataset enrichment jobs that run without a model. Search decides which sources to inspect; fetch makes their contents available for verification. Neither operation establishes that a source is accurate or current.

Why aren't search snippets enough?

A title, URL, and snippet can answer a navigational question: where is the documentation, release note, or likely source? They may omit the date, conditions, exceptions, or surrounding paragraph that changes a claim's meaning. A useful result summary is a reason to open the page, not a substitute for reading it.

AIVAX Web Search follows this discovery-first approach. Use Fetch and OCR when a URL is known or a result needs closer inspection. Check publication dates, prefer primary sources, and compare conflicting accounts before storing a fact or generating an answer.

Some search services can return longer page content. Do not infer completeness from the field name text: inspect what the response contains and whether it covers the claim you need to verify. AIVAX returns a common result shape, not a guarantee of complete page text.

How do I chain search and fetch?

For a backend-controlled workflow, call authenticated POST /api/v1/web/search, select candidate URLs, then call POST /api/v1/web/fetch. Both require a positive account balance. No LLM inference call is needed for the text-only example below.

Search accepts a required non-empty query and optional topn, country, language, and includeDomains. topn defaults to 10 and accepts 1–25; it is a requested maximum, not a promise that many sources will be returned. Country and language handling can vary with the search provider, so verify the returned sources rather than treating those fields as strict guarantees. Domain filters guide discovery; they are not an outbound-access security policy.

The search response exposes data.results, with url, title, and text for each result. Fetch accepts a non-empty contents array of URLs or base64 data URIs. Its data.results includes a zero-based index into that array, extractedText, processingUnits, and error, plus the optional JSON-conversion fields described in the Fetch reference. Use index to associate results with inputs, especially if failed items are omitted.

Run this JavaScript server-side with Bun or a Node.js version that supports fetch and top-level await. Set AIVAX_API_KEY in the process environment; never put the key in browser code or source control. The example requests documentation sources and reads the first five returned results. That selection is illustrative: production callers should apply their source and destination policies before fetching.

const apiKey = process.env.AIVAX_API_KEY;

if (!apiKey) {
  throw new Error("AIVAX_API_KEY is required");
}

const startedAt = new Date().toISOString();
const searchRequest = {
  query: "browser automation release notes September 2026",
  topn: 10,
  includeDomains: ["developer.chrome.com", "playwright.dev"],
};
const headers = {
  Authorization: `Bearer ${apiKey}`,
  "Content-Type": "application/json",
};

const searchResponse = await fetch(
  "https://inference.aivax.net/api/v1/web/search",
  {
    method: "POST",
    headers,
    body: JSON.stringify(searchRequest),
  },
);

if (!searchResponse.ok) {
  throw new Error(`Search failed with HTTP ${searchResponse.status}`);
}

const searchPayload = await searchResponse.json();
const discovered = searchPayload.data.results;
const candidates = discovered.slice(0, 5);

if (candidates.length === 0) {
  throw new Error("Search returned no candidate sources");
}

const fetchResponse = await fetch(
  "https://inference.aivax.net/api/v1/web/fetch",
  {
    method: "POST",
    headers,
    body: JSON.stringify({
      contents: candidates.map((result) => result.url),
      returnErrors: true,
    }),
  },
);

if (!fetchResponse.ok) {
  throw new Error(`Fetch failed with HTTP ${fetchResponse.status}`);
}

const fetchPayload = await fetchResponse.json();
const evidence = {
  startedAt,
  collectedAt: new Date().toISOString(),
  searchRequest,
  discovered,
  selectionPolicy: "First five returned results",
  read: fetchPayload.data.results.map((result) => ({
    index: result.index,
    url: candidates[result.index].url,
    extractedText: result.extractedText,
    processingUnits: result.processingUnits,
    error: result.error,
  })),
};

console.log(JSON.stringify(evidence, null, 2));

This example stops before summarization. Inspect each item's error and text before selecting passages for inference; an HTTP-successful batch can still contain failed extractions. returnErrors defaults to true. Setting it to false omits failed items.

What should I store as evidence?

Keep enough information to reconstruct how a claim reached the answer:

• The question or monitor rule, exact search request, and all returned URLs, titles, and text.
• The selection policy and URLs submitted to fetch.
• Each extraction's status, timestamp, and text, or a durable reference to that text.
• The passages supporting each downstream claim, paired with their source URLs.
• Retry history and the downstream record or decision that consumed the evidence.

The example prints one record; it does not implement durable storage, retries, or recovery from a failed HTTP request. In a production workflow, persist discovery before fetching so a later failure does not erase it. Keep the URL and the extracted version: a page may change after the answer was written.

If findings become long-lived agent context, apply provenance and write controls as described in persistent memory for agents. Permission to read a page should not automatically grant permission to store its claims as trusted memory.

How do I evaluate each stage?

Evaluate discovery, extraction, and the final answer separately. A single “research succeeded” score cannot show which step needs repair.

1. Discovery: for representative questions, check whether the returned URLs include relevant primary sources. Record missing sources, duplicates, and irrelevant results.
2. Extraction: compare the fetched text with the original page or document. Check coverage, dates, tables, identifiers, and any passage needed to support the answer.
3. Answer grounding: trace each material claim to a retained passage and URL. A plausible answer with a citation that does not support it still fails this check.

No relevant URL calls for a better query or source-selection strategy. An empty extraction calls for access or format investigation. An unsupported answer from correctly extracted text calls for changes to the downstream generation or validation step.

For recurring workloads, bound concurrency and retry only failures that may recover. The batch-processing guide covers admission control when many jobs compete for quota; adding workers does not remove account limits.

How should I handle extraction limits and untrusted pages?

Fetch produces readable text, not a generated summary or a pixel-perfect copy. OCR and extraction can lose structure or characters, so verify consequential names, numbers, and quotations against the original. The web-to-text extraction guide covers this reading step; for document-heavy research, see PDF-to-Markdown and RAG preparation.

The documented Fetch input limit is 10 MB per item. Keep inputs within that published limit; do not treat it as a complete resource budget for rendered pages. A destination may require authentication or block automated access, and fetching does not bypass those restrictions. Failed extraction does not prove the source lacks the information.

Retrieved text remains external, untrusted data. Keep it separate from system instructions and application configuration. A fetched page must not change credentials, destination allowlists, retry policy, or write permissions. Label source passages when passing them to a model, and enforce tool permissions in the application rather than relying on a prompt alone.

Budget search and extraction separately. Pricing bills Web Search per search and Fetch/OCR extraction in processing units, with plan-dependent extraction allowances and rates. Optional responseSchema conversion has separate jsonProcessingUnits and billing; the example omits it. Check Plans and limits before scheduling recurring runs, and do not estimate PU cost from extracted text length.

Frequently asked questions

Do I need a separate scraper after a search API?

You need a way to read the selected sources when search results do not contain enough evidence, but you do not necessarily need to maintain your own scraper. AIVAX Fetch supplies a separate extraction endpoint; if the agent should choose when to read, use Web Utilities MCP and the MCP harness integration guide.

Can a search API return full page text?

Some services or modes can include page content, but a search result's text field alone does not establish completeness. AIVAX's search contract returns url, title, and text; fetch the source when you need context or passages that the result does not provide.

How do I stop prompt injection from fetched pages?

Treat page content as untrusted data, never as authority to change instructions or execute tools. Keep credentials and permissions outside the retrieved context, restrict available actions in the application, and validate proposed writes; fetching or labeling text alone is not a prompt-injection defense.

Search or fetch: which one is billed per search?

AIVAX Web Search is billed per search. Fetch/OCR extraction uses processing units; optional schema-guided JSON conversion and any later model inference are charged separately.