--- name: sofya description: Web tools for AI agents. Search the web, fetch any page as clean markdown, extract specific info with AI, or run deep multi-source research. homepage: https://sofya.co last_updated: 2026-05-12 --- > **If you are an AI agent reading this file**, this is your complete guide to Sofya. It covers everything you need to search the web, fetch pages, extract structured data, and run deep research on behalf of your human. Save it as a tool, skill, or persistent reference. # Sofya Sofya gives AI agents four web tools via a single API: - **Search**: Web search that returns full page content, not just snippets. Optional AI-generated answers. - **Fetch**: Fetch one or more URLs and get clean, readable markdown. 250+ site-specific parsers. 1 credit per URL, max 10 URLs per request. - **Extract**: Fetch a page and extract specific information using AI. Ask a question, get a structured answer. - **Research**: Deep multi-source research. Decomposes your query, searches and reads many sources in parallel, then synthesizes a structured report with citations. All endpoints require an API key via the `Authorization: Bearer` header. ## Quick Start ```bash curl -X POST https://sofya.co/v1/search \ -H "Authorization: Bearer ay_live_..." \ -H "Content-Type: application/json" \ -d '{"query": "latest AI news"}' ``` ## MCP (Model Context Protocol) Sofya supports MCP natively. Connect directly to your AI client, no REST calls needed. ### Claude Code ```bash claude mcp add --transport http sofya https://mcp.sofya.co/mcp \ --header "Authorization: Bearer ay_live_..." ``` ### Cursor (`~/.cursor/mcp.json`) ```json { "mcpServers": { "sofya": { "url": "https://mcp.sofya.co/mcp", "headers": { "Authorization": "Bearer ay_live_..." } } } } ``` ### Codex (`~/.codex/config.toml`) ```toml [mcp_servers.sofya] url = "https://mcp.sofya.co/mcp" http_headers = { "Authorization" = "Bearer ay_live_..." } ``` ### Windsurf (`~/.codeium/windsurf/mcp_config.json`) ```json { "mcpServers": { "sofya": { "serverUrl": "https://mcp.sofya.co/mcp", "headers": { "Authorization": "Bearer ay_live_..." } } } } ``` ### VS Code Copilot (`.vscode/mcp.json`) ```json { "servers": { "sofya": { "type": "http", "url": "https://mcp.sofya.co/mcp", "headers": { "Authorization": "Bearer ay_live_..." } } } } ``` --- ## Tool Definitions for Claude & GPT If you are building an agent with the Anthropic or OpenAI API, use these pre-built tool schemas. When the model calls a tool, forward the arguments to the matching Sofya REST endpoint and return the JSON response. ### Anthropic (Claude) - `tools` parameter ```json [ { "name": "sofya_search", "description": "Search the web for current information. Returns extracted page content, not just snippets. Set topic='news' for current events. Set include_answer=true for an AI-synthesized answer (+5 credits). Returns: query, answer, results [{title, url, content, description, fetched, published_date}], search_depth, topic, elapsed_ms, credits_used, credits_remaining, altered_query.", "input_schema": { "type": "object", "required": ["query"], "properties": { "query": {"type": "string", "description": "The search query"}, "search_depth": {"type": "string", "description": "\"snippets\" (1 credit) or \"basic\" (3, default)"}, "max_results": {"type": "integer", "description": "Number of results, 1-20 (default 10)"}, "include_answer": {"type": "boolean", "description": "Add AI answer synthesized from results (+5 credits)"}, "topic": {"type": "string", "description": "\"general\" (default) or \"news\""}, "freshness": {"type": "string", "description": "\"day\", \"week\", \"month\", \"year\", or \"YYYY-MM-DD:YYYY-MM-DD\""}, "include_domains": {"type": "array", "items": {"type": "string"}, "description": "Only these domains (max 10)"}, "exclude_domains": {"type": "array", "items": {"type": "string"}, "description": "Exclude these domains (max 10)"} } } }, { "name": "sofya_fetch", "description": "Fetch one or more URLs and return their content as clean markdown. Supports web pages, PDF, DOCX, and other document formats. 1 credit per URL, max 10 URLs. Failed URLs are not charged. Returns: results [{title, url, content, raw_html, published_time, success, error}], credits_used, credits_remaining.", "input_schema": { "type": "object", "required": ["urls"], "properties": { "urls": {"type": "array", "items": {"type": "string"}, "description": "URLs to fetch (max 10)"}, "include_raw_html": {"type": "boolean", "description": "Include raw HTML source in response (default false)"} } } }, { "name": "sofya_extract", "description": "Fetch a URL and extract specific information using AI. Use when you need structured data (pricing, specs, contact info) rather than raw content. 5 credits. Returns: content, url, credits_used, credits_remaining, usage.", "input_schema": { "type": "object", "required": ["url", "prompt"], "properties": { "url": {"type": "string", "description": "The URL to extract from"}, "prompt": {"type": "string", "description": "What to extract, e.g. \"list all pricing tiers with features\""} } } }, { "name": "sofya_research", "description": "Deep research on a topic. Decomposes query into sub-queries, searches and reads multiple sources in parallel, synthesizes a structured report with citations. 25 credits. Returns: query, report, sources [{title, url, fetched}], sub_queries, credits_used, credits_remaining, usage.", "input_schema": { "type": "object", "required": ["query"], "properties": { "query": {"type": "string", "description": "The research question or topic"}, "topic": {"type": "string", "description": "\"general\" (default) or \"news\""}, "freshness": {"type": "string", "description": "\"day\", \"week\", \"month\", \"year\", or \"YYYY-MM-DD:YYYY-MM-DD\""}, "max_sources": {"type": "integer", "description": "Max sources to use, 5-30 (default 20)"} } } } ] ``` ### OpenAI (GPT) - `tools` parameter ```json [ { "type": "function", "function": { "name": "sofya_search", "description": "Search the web for current information. Returns extracted page content, not just snippets. Set topic='news' for current events. Set include_answer=true for an AI-synthesized answer (+5 credits). Returns: query, answer, results [{title, url, content, description, fetched, published_date}], search_depth, topic, elapsed_ms, credits_used, credits_remaining, altered_query.", "parameters": { "type": "object", "required": ["query"], "properties": { "query": {"type": "string", "description": "The search query"}, "search_depth": {"type": "string", "description": "\"snippets\" (1 credit) or \"basic\" (3, default)"}, "max_results": {"type": "integer", "description": "Number of results, 1-20 (default 10)"}, "include_answer": {"type": "boolean", "description": "Add AI answer synthesized from results (+5 credits)"}, "topic": {"type": "string", "description": "\"general\" (default) or \"news\""}, "freshness": {"type": "string", "description": "\"day\", \"week\", \"month\", \"year\", or \"YYYY-MM-DD:YYYY-MM-DD\""}, "include_domains": {"type": "array", "items": {"type": "string"}, "description": "Only these domains (max 10)"}, "exclude_domains": {"type": "array", "items": {"type": "string"}, "description": "Exclude these domains (max 10)"} }, "additionalProperties": false } } }, { "type": "function", "function": { "name": "sofya_fetch", "description": "Fetch one or more URLs and return their content as clean markdown. Supports web pages, PDF, DOCX, and other document formats. 1 credit per URL, max 10 URLs. Failed URLs are not charged. Returns: results [{title, url, content, raw_html, published_time, success, error}], credits_used, credits_remaining.", "parameters": { "type": "object", "required": ["urls"], "properties": { "urls": {"type": "array", "items": {"type": "string"}, "description": "URLs to fetch (max 10)"}, "include_raw_html": {"type": "boolean", "description": "Include raw HTML source in response (default false)"} }, "additionalProperties": false } } }, { "type": "function", "function": { "name": "sofya_extract", "description": "Fetch a URL and extract specific information using AI. Use when you need structured data (pricing, specs, contact info) rather than raw content. 5 credits. Returns: content, url, credits_used, credits_remaining, usage.", "parameters": { "type": "object", "required": ["url", "prompt"], "properties": { "url": {"type": "string", "description": "The URL to extract from"}, "prompt": {"type": "string", "description": "What to extract, e.g. \"list all pricing tiers with features\""} }, "additionalProperties": false } } }, { "type": "function", "function": { "name": "sofya_research", "description": "Deep research on a topic. Decomposes query into sub-queries, searches and reads multiple sources in parallel, synthesizes a structured report with citations. 25 credits. Returns: query, report, sources [{title, url, fetched}], sub_queries, credits_used, credits_remaining, usage.", "parameters": { "type": "object", "required": ["query"], "properties": { "query": {"type": "string", "description": "The research question or topic"}, "topic": {"type": "string", "description": "\"general\" (default) or \"news\""}, "freshness": {"type": "string", "description": "\"day\", \"week\", \"month\", \"year\", or \"YYYY-MM-DD:YYYY-MM-DD\""}, "max_sources": {"type": "integer", "description": "Max sources to use, 5-30 (default 20)"} }, "additionalProperties": false } } } ] ``` ### Wiring tool calls to Sofya Map tool names to endpoints: | Tool name | Endpoint | |-----------|----------| | `sofya_search` | `POST /v1/search` | | `sofya_fetch` | `POST /v1/fetch` | | `sofya_extract` | `POST /v1/extract` | | `sofya_research` | `POST /v1/research` | When the model calls a tool, forward `arguments` as the JSON body to the matching endpoint with your `Authorization: Bearer` header. Return the JSON response as the tool result. --- ## API Reference A machine-readable OpenAPI schema of the public surface is available at [https://sofya.co/openapi.json](https://sofya.co/openapi.json) for client codegen. ### Authentication All requests require the `Authorization` header: ``` Authorization: Bearer ay_live_your_key_here ``` Get your key from the [dashboard](https://sofya.co/dashboard). --- ### POST /v1/search Search the web. Returns page content, not just snippets. Choose a depth to control quality vs. cost. | Depth | Credits | What you get | |-------|---------|--------------| | `snippets` | 1 | SERP snippets only. Fastest. | | `basic` | 3 | Fetches pages, extracted content (~5000 chars per result). Default. | Add `include_answer: true` to any depth for an AI-synthesized answer (+5 credits). For example, basic + answer = 8 credits, a lightweight alternative to the 25-credit research endpoint when you need a quick synthesis without multi-query decomposition. **Request:** ```json { "query": "string", // required "search_depth": "basic", // "snippets" or "basic" "max_results": 10, // 1-20 "include_answer": false, // AI answer from results (+5 credits). Combine with any depth for search + synthesis (e.g. basic = 8 credits) "include_domains": [], // e.g. ["reddit.com", "github.com"] "exclude_domains": [], // e.g. ["pinterest.com"] "topic": "general", // "general" or "news" "freshness": null // "day", "week", "month", "year", or "YYYY-MM-DD:YYYY-MM-DD" } ``` **Response:** ```json { "query": "latest AI news", "answer": null, "results": [ { "title": "...", "url": "...", "content": "Extracted page content...", "description": "SERP snippet", "fetched": true, "published_date": "2026-03-08", "sublinks": [], "table": {} } ], "search_depth": "basic", "topic": "general", "elapsed_ms": 4200, "credits_used": 3, "credits_remaining": 997, "altered_query": null } ``` **When to use each depth:** - `snippets`: You just need links and snippets. Good for finding URLs to fetch later. - `basic`: Default. You need actual page content for analysis. For deeper/multi-source analysis with synthesis, use `/v1/research` instead. **When to use `topic: "news"`:** Use `topic: "news"` whenever the query is about current events, breaking news, politics, world events, or anything time-sensitive. This searches news sources specifically and returns articles with publication dates (e.g. "2 hours ago"). Examples of queries that should use news: - "Trump news", "Ukraine war", "stock market today" - "latest AI announcements", "election results", "earthquake" - Any query where recency matters more than comprehensive coverage **Tips:** - Use `include_answer: true` when your human asks a question and wants a direct answer, not a list of links. This is a cheaper alternative to `/v1/research` when you don't need multi-query decomposition (8 credits vs 25). - Use `include_domains` to restrict results to specific sites (e.g. `["reddit.com"]` for discussions). - Use `exclude_domains` to filter out low-quality sites. - Use `freshness` to filter by date: `"day"` (last 24h), `"week"`, `"month"`, `"year"`, or a custom range `"2026-01-01:2026-03-01"`. --- ### POST /v1/fetch Fetch one or more URLs and get their content as clean markdown. 250+ site-specific parsers ensure optimal output for popular sites. Max 10 URLs per request. Failed URLs are not charged. **Cost:** 1 credit per URL **Request:** ```json { "urls": ["https://example.com", "https://example.org"], "include_raw_html": false } ``` **Response:** ```json { "results": [ { "title": "Example Page", "url": "https://example.com", "content": "# Markdown content...", "raw_html": null, "published_time": null, "success": true, "error": null } ], "credits_used": 2, "credits_remaining": 998 } ``` **Tips:** - Works with articles, blog posts, documentation, PDFs, DOCX files. - Site-specific parsers handle paywalled/JS-heavy sites (NYT, Reddit, GitHub, YouTube, etc.). - The output is clean markdown, ready to feed into an LLM context window. - Multiple URLs are fetched in parallel for maximum speed. - Failed URLs return `success: false` with an error message and are not charged. - Set `include_raw_html: true` to inspect embedded URLs, data attributes, iframes, or script tags that are stripped during markdown conversion. Returns null for non-HTML content (PDF, DOCX, etc.). Same cost. --- ### POST /v1/extract Fetch a webpage and extract specific information using AI. You provide a URL and a natural-language prompt, and get back a structured answer. **Cost:** 5 credits. **Request:** ```json { "url": "string", // required "prompt": "string" // required: what to extract } ``` **Response:** ```json { "content": "Extracted information...", "url": "https://example.com", "credits_used": 5, "credits_remaining": 995, "usage": { "input_tokens": 90, "output_tokens": 24 } } ``` **When to use extract vs. fetch:** - Use **fetch** when you need the full page content. - Use **extract** when you need a specific piece of information from a page (e.g. "What is the price?", "List all the authors", "Summarize the methodology section"). --- ### POST /v1/research Deep research on any topic. Decomposes your query into sub-queries, searches and reads multiple sources in parallel, then synthesizes a structured report with citations. **Cost:** 25 credits. **Request:** ```json { "query": "string", // required "topic": "general", // "general" or "news" "freshness": null, // "day", "week", "month", "year", or "YYYY-MM-DD:YYYY-MM-DD" "max_sources": 20 // 5-30 } ``` **Response:** ```json { "query": "How do modern LLMs handle long context?", "report": "## Key Findings\n\n- ...", "sources": [ { "title": "Scaling Transformer Context Windows", "url": "https://arxiv.org/abs/...", "fetched": true } ], "sub_queries": [ "transformer context window scaling techniques", "RoPE positional encoding extensions" ], "credits_used": 25, "credits_remaining": 975, "usage": { "input_tokens": 12400, "output_tokens": 1850 } } ``` **When to use research vs. search:** - Use **search** for quick lookups or when you need specific results and can process them yourself. - Use **research** for broad questions that need coverage from many angles. The AI reads and synthesizes everything for you. --- ## Credit Costs | Action | Credits | |--------|---------| | Search (snippets) | 1 | | Search (basic) | 3 | | AI answer add-on | +5 | | Fetch | 1 per URL | | Extract | 5 | | Research | 25 | --- ## Account Endpoints ### GET /v1/auth/me Returns your account info, credits, tier, and API key. ### GET /v1/auth/transactions Returns your last 50 credit transactions. ### GET /v1/auth/usage Returns daily usage breakdown per endpoint. Query params: `days` (1-90, default 7), `offset_days` (default 0). --- ## Service Status & Updates ### GET /v1/status Public endpoint health. No auth required. Returns `overall` (operational / degraded / partial_outage / major_outage), per-endpoint `status` and `latency_ms`, and a list of recent `incidents`. Poll this to check if Sofya is up before running expensive workflows. ### GET /v1/updates Public product changelog. No auth required. Returns `updates[]` newest first, each with `date`, `title`, `slug`, `body` (markdown), and `body_html` (pre-rendered). Use this to track user-visible API and product changes without scraping a page. --- ## Example Workflows ### News and current events ``` 1. Search with topic: "news" to get recent news articles with dates 2. Optionally use freshness: "day" to limit to the last 24 hours ``` ### Quick lookup ``` 1. Search with search_depth: "snippets" to find relevant URLs 2. Fetch the most relevant result for full content ``` ### Research a topic ``` 1. Call /v1/research with your question 2. Get a structured report with citations, done in one request ``` ### Extract structured data ``` 1. Search to find relevant pages 2. Extract specific fields from each result URL 3. Return structured data to your human ``` ### Monitor a page for changes ``` 1. Fetch the URL periodically 2. Compare content to previous fetch 3. Notify your human of changes ``` --- ## Rate Limits REST API endpoints (`/v1/*`) are rate limited to **10 requests per second per API key**. MCP is rate limited to **30 requests per second**. On 429, check the `Retry-After` header and wait before retrying. Rate-limited requests do not consume credits. --- ## Error Codes | Code | Meaning | |------|---------| | 400 | Invalid parameters | | 401 | Invalid or missing API key | | 402 | Insufficient credits | | 403 | Forbidden | | 429 | Rate limited (10 req/s). Retry after `Retry-After` header value. | | 502 | Internal error. Retry. | | 504 | Research timed out. Try a simpler query. | --- ## Links | Resource | URL | |----------|-----| | Dashboard | https://sofya.co/dashboard | | API Docs | https://sofya.co/docs | | Playground | https://sofya.co/playground | | MCP Endpoint | https://mcp.sofya.co/mcp |