# ContentAPI - Complete API Reference

> Universal content extraction API for RAG pipelines, AI agents, and LLM applications.
> Base URL: https://api.getcontentapi.com/api/v1

ContentAPI extracts structured content from YouTube, web pages, Twitter/X, and Reddit.
It provides built-in chunking for RAG workflows, batch processing, web search with
content extraction, a semantic content tree for web pages, and an MCP server for
direct AI assistant integration.

---

## Authentication

Every request requires an API key. Two methods are supported (use either):

```
Authorization: Bearer sk_live_your_key_here
```

```
X-API-Key: sk_live_your_key_here
```

Get a free API key at https://getcontentapi.com (no credit card required).

---

## Universal Query Parameters

These parameters work on all content endpoints (transcript, extract, tweet, thread, post):

| Parameter     | Type | Description                                         |
|---------------|------|-----------------------------------------------------|
| chunk_size    | int  | Split content into chunks of N characters (for RAG) |
| chunk_overlap | int  | Overlap between chunks in characters                |

When chunking is enabled, the response includes a `chunks` array instead of a single
`content` string. Each chunk contains the text segment and its index.

---

## YouTube Endpoints

### GET /youtube/transcript

Extract the full transcript of a YouTube video with timestamps.

Parameters:
- url (required) — YouTube video URL or video ID
- lang (optional) — Language code for captions (e.g., "en", "es"). Defaults to primary language.
- chunk_size (optional) — Chunk size in characters
- chunk_overlap (optional) — Overlap between chunks in characters

Request:
```
curl "https://api.getcontentapi.com/api/v1/youtube/transcript?url=https://youtube.com/watch?v=dQw4w9WgXcQ&lang=en" \
  -H "Authorization: Bearer sk_live_xxx"
```

Response:
```json
{
  "success": true,
  "data": {
    "source": "youtube",
    "video_id": "dQw4w9WgXcQ",
    "title": "Rick Astley - Never Gonna Give You Up",
    "channel": "Rick Astley",
    "duration": 213,
    "language": "en",
    "content": "We're no strangers to love. You know the rules and so do I...",
    "transcript": [
      {"start": 18.64, "duration": 3.24, "text": "We're no strangers to love"},
      {"start": 22.64, "duration": 4.32, "text": "You know the rules and so do I"}
    ],
    "metadata": {
      "title": "Rick Astley - Never Gonna Give You Up",
      "channel": "Rick Astley",
      "duration": 213,
      "view_count": 1500000000
    }
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

### GET /youtube/metadata

Get video metadata without extracting the transcript.

Parameters:
- url (required) — YouTube video URL or video ID

Request:
```
curl "https://api.getcontentapi.com/api/v1/youtube/metadata?url=https://youtube.com/watch?v=dQw4w9WgXcQ" \
  -H "X-API-Key: sk_live_xxx"
```

Response:
```json
{
  "success": true,
  "data": {
    "source": "youtube",
    "video_id": "dQw4w9WgXcQ",
    "title": "Rick Astley - Never Gonna Give You Up",
    "channel": "Rick Astley",
    "description": "The official video for Never Gonna Give You Up...",
    "duration": 213,
    "view_count": 1500000000,
    "like_count": 15000000,
    "thumbnail": "https://i.ytimg.com/vi/dQw4w9WgXcQ/maxresdefault.jpg",
    "published_at": "2009-10-25T00:00:00Z"
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

### GET /youtube/languages

List all available caption languages for a video.

Parameters:
- url (required) — YouTube video URL or video ID

Request:
```
curl "https://api.getcontentapi.com/api/v1/youtube/languages?url=https://youtube.com/watch?v=dQw4w9WgXcQ" \
  -H "X-API-Key: sk_live_xxx"
```

Response:
```json
{
  "success": true,
  "data": {
    "video_id": "dQw4w9WgXcQ",
    "available_languages": [
      {"code": "en", "name": "English", "auto_generated": false},
      {"code": "es", "name": "Spanish", "auto_generated": true},
      {"code": "pt", "name": "Portuguese", "auto_generated": true}
    ]
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

---

### GET /youtube/channel

Get YouTube channel metadata and recent videos.

Parameters:
- handle (required) — Channel handle (@mkbhd), full URL (https://youtube.com/@mkbhd), or channel ID (UCBcRF18a7Qf58cCRy5xuWwQ)

Request:
```
curl "https://api.getcontentapi.com/api/v1/youtube/channel?handle=@mkbhd" \
  -H "X-API-Key: sk_live_xxx"
```

Response:
```json
{
  "success": true,
  "data": {
    "channel_id": "UCBJycsmduvYEL83R_U4JriQ",
    "name": "Marques Brownlee",
    "handle": "@mkbhd",
    "description": "MKBHD: Quality Tech Videos...",
    "subscribers": "20.7M",
    "video_count": "1,800",
    "avatar_url": "https://yt3.googleusercontent.com/...",
    "banner_url": "https://yt3.googleusercontent.com/...",
    "recent_videos": [
      {
        "video_id": "abc123",
        "title": "Latest Video Title",
        "published": "2 days ago",
        "views": "2.5M",
        "duration": "15:30",
        "thumbnail": "https://i.ytimg.com/vi/abc123/hqdefault.jpg"
      }
    ]
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

---

### GET /youtube/playlist

Extract all videos from a YouTube playlist.

Parameters:
- url (required) — Playlist URL or playlist ID
- limit (optional) — Max videos to return (default 50, max 200)

Request:
```
curl "https://api.getcontentapi.com/api/v1/youtube/playlist?url=https://youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf&limit=10" \
  -H "X-API-Key: sk_live_xxx"
```

Response:
```json
{
  "success": true,
  "data": {
    "playlist_id": "PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
    "title": "Playlist Title",
    "description": "Playlist description...",
    "author": "Channel Name",
    "video_count": 50,
    "videos": [
      {
        "video_id": "abc123",
        "title": "Video Title",
        "channel": "Channel Name",
        "duration": "15:30",
        "thumbnail": "https://i.ytimg.com/vi/abc123/hqdefault.jpg",
        "position": 1
      }
    ]
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

---

## Web Endpoints

### GET /web/extract

Extract readable content from any web page. Returns clean text plus a semantic
structure tree representing the page layout.

Parameters:
- url (required) — Any web page URL
- expand (optional, boolean) — Include div/span/section containers in structure tree
- chunk_size (optional) — Chunk size in characters
- chunk_overlap (optional) — Overlap between chunks in characters

Request:
```
curl "https://api.getcontentapi.com/api/v1/web/extract?url=https://example.com/article&expand=true" \
  -H "Authorization: Bearer sk_live_xxx"
```

Response:
```json
{
  "success": true,
  "data": {
    "source": "web",
    "content": "Full extracted text of the article...",
    "metadata": {
      "title": "Example Article Title",
      "description": "A brief summary of the article.",
      "author": "Jane Doe",
      "published_at": "2025-01-15T10:00:00Z",
      "url": "https://example.com/article",
      "domain": "example.com",
      "og_image": "https://example.com/images/article.jpg"
    },
    "structure": [
      {
        "type": "heading",
        "level": 1,
        "text": "Example Article Title",
        "children": [
          {
            "type": "paragraph",
            "text": "This is the first paragraph of the article."
          },
          {
            "type": "heading",
            "level": 2,
            "text": "Section One",
            "children": [
              {
                "type": "paragraph",
                "text": "Content under section one."
              },
              {
                "type": "list",
                "ordered": false,
                "items": ["Item one", "Item two", "Item three"]
              },
              {
                "type": "code_block",
                "language": "python",
                "text": "print('hello world')"
              }
            ]
          },
          {
            "type": "heading",
            "level": 2,
            "text": "Section Two",
            "children": [
              {
                "type": "table",
                "headers": ["Name", "Value"],
                "rows": [["Alpha", "1"], ["Beta", "2"]]
              },
              {
                "type": "blockquote",
                "text": "This is an important quote."
              },
              {
                "type": "image",
                "src": "https://example.com/images/chart.png",
                "alt": "A chart showing growth"
              }
            ]
          }
        ]
      }
    ]
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

### Structured Content Tree

The `structure` field in web extraction responses contains a semantic JSON tree
that preserves the document hierarchy. Node types:

| Type        | Description                                    | Key Fields                          |
|-------------|------------------------------------------------|-------------------------------------|
| heading     | Section heading (h1-h6)                        | level, text, children               |
| paragraph   | Text paragraph                                 | text                                |
| list        | Ordered or unordered list                      | ordered, items                      |
| code_block  | Code snippet                                   | language, text                      |
| table       | Data table                                     | headers, rows                       |
| blockquote  | Quoted text                                    | text                                |
| image       | Image element                                  | src, alt                            |
| link_block  | Prominent link or card                         | href, text                          |
| media       | Embedded media (video, audio, iframe)          | src, type                           |
| nav         | Navigation section                             | children                            |

Headings automatically nest their children by hierarchy level. An h1 contains
everything until the next h1; an h2 contains everything until the next h2 or h1.

With `expand=true`, container elements (div, span, section) are preserved in the
tree, which is useful for understanding page layout but increases response size.

### Smart Fetch

Web extraction uses a 5-strategy cascade for maximum reliability:

1. Direct fetch — Standard HTTP request
2. Geo-matched proxy — Proxy from the same region as the target site
3. Proxy rotation — Try different proxy servers
4. Google Cache — Retrieve cached version from Google
5. Archive.org fallback — Retrieve from the Wayback Machine

This means ContentAPI can extract content even from sites with aggressive bot
detection or regional restrictions.

PDF files are auto-detected and their text is extracted automatically. No special
parameters are needed; just pass the PDF URL to /web/extract.

### GET /web/metadata

Get page metadata without extracting full content.

Parameters:
- url (required) — Any web page URL

Request:
```
curl "https://api.getcontentapi.com/api/v1/web/metadata?url=https://example.com/article" \
  -H "X-API-Key: sk_live_xxx"
```

Response:
```json
{
  "success": true,
  "data": {
    "source": "web",
    "metadata": {
      "title": "Example Article Title",
      "description": "A brief summary of the article.",
      "author": "Jane Doe",
      "published_at": "2025-01-15T10:00:00Z",
      "url": "https://example.com/article",
      "domain": "example.com",
      "og_image": "https://example.com/images/article.jpg",
      "og_title": "Example Article Title",
      "og_description": "A brief summary of the article."
    }
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

### GET /web/search

Search the web and optionally extract content from results.

Parameters:
- q (required) — Search query string
- count (optional, int) — Number of results to return (default: 5)
- extract (optional, boolean) — If true, extract full content from each result URL
- chunk_size (optional) — Chunk size in characters (applied to extracted content)
- chunk_overlap (optional) — Overlap between chunks in characters

Request (search only):
```
curl "https://api.getcontentapi.com/api/v1/web/search?q=rust+programming+language&count=3" \
  -H "Authorization: Bearer sk_live_xxx"
```

Response (search only):
```json
{
  "success": true,
  "data": {
    "source": "web",
    "query": "rust programming language",
    "results": [
      {
        "title": "Rust Programming Language",
        "url": "https://www.rust-lang.org/",
        "snippet": "A language empowering everyone to build reliable and efficient software."
      },
      {
        "title": "Rust (programming language) - Wikipedia",
        "url": "https://en.wikipedia.org/wiki/Rust_(programming_language)",
        "snippet": "Rust is a multi-paradigm, general-purpose programming language..."
      },
      {
        "title": "The Rust Programming Language - Book",
        "url": "https://doc.rust-lang.org/book/",
        "snippet": "An introductory book about Rust..."
      }
    ]
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

Request (search + extract):
```
curl "https://api.getcontentapi.com/api/v1/web/search?q=rust+async+tutorial&count=2&extract=true" \
  -H "Authorization: Bearer sk_live_xxx"
```

Response (search + extract):
```json
{
  "success": true,
  "data": {
    "source": "web",
    "query": "rust async tutorial",
    "results": [
      {
        "title": "Async in Rust",
        "url": "https://example.com/rust-async",
        "snippet": "Learn async programming in Rust...",
        "content": "Full extracted article text here...",
        "metadata": {
          "title": "Async in Rust",
          "author": "John Smith"
        }
      }
    ]
  },
  "credits": {"used": 2, "remaining": 4997}
}
```

Note: When extract=true, each result URL counts as an additional credit.

---

## Twitter/X Endpoints

### GET /twitter/tweet

Extract a single tweet with author info and engagement metrics.

Parameters:
- url (required) — Tweet URL (twitter.com or x.com)
- chunk_size (optional) — Chunk size in characters
- chunk_overlap (optional) — Overlap between chunks in characters

Request:
```
curl "https://api.getcontentapi.com/api/v1/twitter/tweet?url=https://x.com/elonmusk/status/1234567890" \
  -H "Authorization: Bearer sk_live_xxx"
```

Response:
```json
{
  "success": true,
  "data": {
    "source": "twitter",
    "content": "This is the full tweet text content...",
    "metadata": {
      "author": "Elon Musk",
      "username": "elonmusk",
      "posted_at": "2025-01-15T14:30:00Z",
      "likes": 50000,
      "retweets": 12000,
      "replies": 8000,
      "url": "https://x.com/elonmusk/status/1234567890"
    }
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

### GET /twitter/thread

Extract a full Twitter thread (original tweet plus all replies by the author).

Parameters:
- url (required) — URL of any tweet in the thread
- chunk_size (optional) — Chunk size in characters
- chunk_overlap (optional) — Overlap between chunks in characters

Request:
```
curl "https://api.getcontentapi.com/api/v1/twitter/thread?url=https://x.com/user/status/1234567890" \
  -H "Authorization: Bearer sk_live_xxx"
```

Response:
```json
{
  "success": true,
  "data": {
    "source": "twitter",
    "content": "Full thread text concatenated...",
    "metadata": {
      "author": "Tech Writer",
      "username": "techwriter",
      "thread_length": 5,
      "posted_at": "2025-01-15T14:30:00Z"
    }
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

---

## Reddit Endpoint

### GET /reddit/post

Extract a Reddit post with its content and comments.

Parameters:
- url (required) — Reddit post URL
- chunk_size (optional) — Chunk size in characters
- chunk_overlap (optional) — Overlap between chunks in characters

Request:
```
curl "https://api.getcontentapi.com/api/v1/reddit/post?url=https://www.reddit.com/r/programming/comments/abc123/example_post/" \
  -H "Authorization: Bearer sk_live_xxx"
```

Response:
```json
{
  "success": true,
  "data": {
    "source": "reddit",
    "content": "Post title and body text, followed by comments...",
    "metadata": {
      "title": "Example Post Title",
      "author": "reddit_user",
      "subreddit": "programming",
      "score": 1500,
      "num_comments": 234,
      "posted_at": "2025-01-15T10:00:00Z",
      "url": "https://www.reddit.com/r/programming/comments/abc123/example_post/"
    }
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

---

## Batch Endpoint

### POST /batch

Process up to 50 URLs in a single request. URLs are processed in parallel and
source types are auto-detected (YouTube, web, Twitter, Reddit).

Request:
```
curl -X POST "https://api.getcontentapi.com/api/v1/batch" \
  -H "Authorization: Bearer sk_live_xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://youtube.com/watch?v=dQw4w9WgXcQ",
      "https://example.com/article",
      "https://x.com/user/status/123456"
    ]
  }'
```

Response:
```json
{
  "success": true,
  "data": {
    "results": [
      {
        "url": "https://youtube.com/watch?v=dQw4w9WgXcQ",
        "success": true,
        "data": {
          "source": "youtube",
          "content": "Transcript text...",
          "metadata": {"title": "Never Gonna Give You Up", "channel": "Rick Astley"}
        }
      },
      {
        "url": "https://example.com/article",
        "success": true,
        "data": {
          "source": "web",
          "content": "Article text...",
          "metadata": {"title": "Example Article"}
        }
      },
      {
        "url": "https://x.com/user/status/123456",
        "success": true,
        "data": {
          "source": "twitter",
          "content": "Tweet text...",
          "metadata": {"author": "user"}
        }
      }
    ],
    "total": 3,
    "succeeded": 3,
    "failed": 0
  },
  "credits": {"used": 3, "remaining": 4997}
}
```

Each URL in the batch counts as one credit. If a URL fails, the rest still
process and the failed URL is included in results with success=false and an
error message.

The batch endpoint also supports chunk_size and chunk_overlap in the request body:
```json
{
  "urls": ["https://example.com/article1", "https://example.com/article2"],
  "chunk_size": 1000,
  "chunk_overlap": 200
}
```

---

## Chunking for RAG

All content endpoints support chunking parameters for direct use in RAG pipelines.

Parameters:
- chunk_size — Number of characters per chunk
- chunk_overlap — Number of overlapping characters between consecutive chunks

Example request with chunking:
```
curl "https://api.getcontentapi.com/api/v1/web/extract?url=https://example.com/article&chunk_size=1000&chunk_overlap=200" \
  -H "Authorization: Bearer sk_live_xxx"
```

When chunking is enabled, the response includes a `chunks` array:
```json
{
  "success": true,
  "data": {
    "source": "web",
    "content": "Full text of the article...",
    "chunks": [
      {"index": 0, "text": "First 1000 characters of the article..."},
      {"index": 1, "text": "...overlapping 200 chars... next 1000 characters..."},
      {"index": 2, "text": "...overlapping 200 chars... final segment..."}
    ],
    "metadata": {"title": "Example Article"}
  },
  "credits": {"used": 1, "remaining": 4999}
}
```

Recommended chunk sizes:
- 500-1000 characters for precise retrieval (Q&A, search)
- 1000-2000 characters for general RAG (summaries, analysis)
- 2000-4000 characters for broad context (fewer chunks, more context per chunk)

A typical overlap of 10-20% of chunk_size works well (e.g., chunk_size=1000, chunk_overlap=150).

---

## Error Codes

| Code | Meaning                                                    |
|------|------------------------------------------------------------|
| 400  | Invalid URL or missing required parameters                 |
| 401  | Invalid or missing API key                                 |
| 403  | Quota exceeded (upgrade plan or wait for monthly reset)    |
| 404  | Content not found (page does not exist, no captions, etc.) |
| 429  | Rate limit exceeded (wait and retry)                       |
| 500  | Internal server error (contact support if persistent)      |
| 503  | Source temporarily blocking requests (retry after delay)   |

Error response format:
```json
{
  "success": false,
  "error": {
    "code": 429,
    "message": "Rate limit exceeded. Please wait before making more requests."
  }
}
```

---

## Rate Limits

| Plan       | Requests per Minute |
|------------|---------------------|
| Free       | 10/min              |
| Pro        | 300/min             |
| Enterprise | Custom              |

Rate limits are applied per API key. When exceeded, the API returns HTTP 429.
The response includes a Retry-After header indicating seconds to wait.

---

## Pricing

| Plan       | Price      | Requests/Month | Rate Limit |
|------------|------------|----------------|------------|
| Free       | $0         | 5,000          | 10/min     |
| Pro        | $49/mo     | 100,000        | 300/min    |
| Enterprise | Custom     | Unlimited      | Custom     |

Free tier requires no credit card. Sign up at https://getcontentapi.com.

---

## MCP Server

ContentAPI provides an MCP (Model Context Protocol) server for direct integration
with AI assistants including Claude Desktop, Cursor, Windsurf, and any MCP-compatible client.

### Installation

The MCP server is distributed as an npm package. No manual install required;
npx downloads it on first use.

Add this to your MCP configuration file:

```json
{
  "mcpServers": {
    "contentapi": {
      "command": "npx",
      "args": ["-y", "contentapi-mcp-server"],
      "env": {
        "CONTENTAPI_API_KEY": "sk_live_your_key_here"
      }
    }
  }
}
```

The package name is `contentapi-mcp-server` on npm: https://npmjs.com/package/contentapi-mcp-server

### Available MCP Tools

| Tool                | Description                                      |
|---------------------|--------------------------------------------------|
| youtube_transcript  | Extract YouTube video transcript with timestamps |
| youtube_metadata    | Get YouTube video metadata                       |
| youtube_channel     | Get YouTube channel metadata and recent videos   |
| youtube_playlist    | Extract all videos from a YouTube playlist        |
| web_extract         | Extract readable content from any web page       |
| web_metadata        | Get web page metadata (title, description, etc.) |
| web_search          | Search the web, optionally extract content       |
| twitter_tweet       | Get tweet content and engagement metrics         |
| twitter_thread      | Get full Twitter/X thread                        |
| reddit_post         | Get Reddit post content and comments             |
| batch_extract       | Process multiple URLs in parallel                |

All tools support chunk_size and chunk_overlap parameters for RAG workflows.

---

## Code Examples

### Python

```python
import requests

API_KEY = "sk_live_your_key_here"
BASE_URL = "https://api.getcontentapi.com/api/v1"
HEADERS = {"Authorization": f"Bearer {API_KEY}"}

# Extract YouTube transcript
response = requests.get(
    f"{BASE_URL}/youtube/transcript",
    params={"url": "https://youtube.com/watch?v=dQw4w9WgXcQ", "lang": "en"},
    headers=HEADERS
)
data = response.json()["data"]
transcript_text = data["content"]

# Extract web page with chunking for RAG
response = requests.get(
    f"{BASE_URL}/web/extract",
    params={
        "url": "https://example.com/article",
        "chunk_size": 1000,
        "chunk_overlap": 200
    },
    headers=HEADERS
)
chunks = response.json()["data"]["chunks"]

# Web search with content extraction
response = requests.get(
    f"{BASE_URL}/web/search",
    params={"q": "python async tutorial", "count": 3, "extract": "true"},
    headers=HEADERS
)
results = response.json()["data"]["results"]

# Batch extraction
response = requests.post(
    f"{BASE_URL}/batch",
    headers={**HEADERS, "Content-Type": "application/json"},
    json={
        "urls": [
            "https://youtube.com/watch?v=dQw4w9WgXcQ",
            "https://example.com/article",
            "https://x.com/user/status/123"
        ]
    }
)
results = response.json()["data"]["results"]
```

### JavaScript / TypeScript

```javascript
const API_KEY = "sk_live_your_key_here";
const BASE_URL = "https://api.getcontentapi.com/api/v1";
const headers = { "Authorization": `Bearer ${API_KEY}` };

// Extract YouTube transcript
const transcriptRes = await fetch(
  `${BASE_URL}/youtube/transcript?url=https://youtube.com/watch?v=dQw4w9WgXcQ&lang=en`,
  { headers }
);
const { data: transcriptData } = await transcriptRes.json();

// Extract web page with chunking
const webRes = await fetch(
  `${BASE_URL}/web/extract?url=https://example.com/article&chunk_size=1000&chunk_overlap=200`,
  { headers }
);
const { data: webData } = await webRes.json();
const chunks = webData.chunks;

// Web search with extraction
const searchRes = await fetch(
  `${BASE_URL}/web/search?q=react+hooks+tutorial&count=3&extract=true`,
  { headers }
);
const { data: searchData } = await searchRes.json();

// Batch extraction
const batchRes = await fetch(`${BASE_URL}/batch`, {
  method: "POST",
  headers: { ...headers, "Content-Type": "application/json" },
  body: JSON.stringify({
    urls: [
      "https://youtube.com/watch?v=dQw4w9WgXcQ",
      "https://example.com/article"
    ],
    chunk_size: 1000,
    chunk_overlap: 200
  })
});
const { data: batchData } = await batchRes.json();
```

### curl

```bash
# YouTube transcript
curl "https://api.getcontentapi.com/api/v1/youtube/transcript?url=https://youtube.com/watch?v=dQw4w9WgXcQ&lang=en" \
  -H "Authorization: Bearer sk_live_xxx"

# YouTube metadata
curl "https://api.getcontentapi.com/api/v1/youtube/metadata?url=https://youtube.com/watch?v=dQw4w9WgXcQ" \
  -H "X-API-Key: sk_live_xxx"

# Available caption languages
curl "https://api.getcontentapi.com/api/v1/youtube/languages?url=https://youtube.com/watch?v=dQw4w9WgXcQ" \
  -H "X-API-Key: sk_live_xxx"

# Web extraction with chunking
curl "https://api.getcontentapi.com/api/v1/web/extract?url=https://example.com/article&chunk_size=1000&chunk_overlap=200" \
  -H "Authorization: Bearer sk_live_xxx"

# Web extraction with expanded structure tree
curl "https://api.getcontentapi.com/api/v1/web/extract?url=https://example.com/article&expand=true" \
  -H "Authorization: Bearer sk_live_xxx"

# Web metadata
curl "https://api.getcontentapi.com/api/v1/web/metadata?url=https://example.com/article" \
  -H "X-API-Key: sk_live_xxx"

# Web search
curl "https://api.getcontentapi.com/api/v1/web/search?q=rust+programming&count=5" \
  -H "Authorization: Bearer sk_live_xxx"

# Web search with content extraction
curl "https://api.getcontentapi.com/api/v1/web/search?q=rust+programming&count=3&extract=true" \
  -H "Authorization: Bearer sk_live_xxx"

# Tweet
curl "https://api.getcontentapi.com/api/v1/twitter/tweet?url=https://x.com/user/status/123" \
  -H "Authorization: Bearer sk_live_xxx"

# Twitter thread
curl "https://api.getcontentapi.com/api/v1/twitter/thread?url=https://x.com/user/status/123" \
  -H "Authorization: Bearer sk_live_xxx"

# Reddit post
curl "https://api.getcontentapi.com/api/v1/reddit/post?url=https://reddit.com/r/programming/comments/abc123/post/" \
  -H "Authorization: Bearer sk_live_xxx"

# Batch extraction
curl -X POST "https://api.getcontentapi.com/api/v1/batch" \
  -H "Authorization: Bearer sk_live_xxx" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://youtube.com/watch?v=dQw4w9WgXcQ", "https://example.com/article"]}'
```

---

## RAG Pipeline Integration Guide

ContentAPI is designed for direct integration into RAG (Retrieval-Augmented
Generation) pipelines. Here is a recommended workflow:

### Step 1: Extract Content

Use ContentAPI to extract content from your sources. For multiple URLs,
use the batch endpoint for efficiency.

```python
import requests

API_KEY = "sk_live_xxx"
BASE = "https://api.getcontentapi.com/api/v1"

# Extract with built-in chunking
response = requests.get(
    f"{BASE}/web/extract",
    params={"url": "https://example.com/article", "chunk_size": 1000, "chunk_overlap": 150},
    headers={"Authorization": f"Bearer {API_KEY}"}
)
chunks = response.json()["data"]["chunks"]
```

### Step 2: Generate Embeddings

Use your preferred embedding model (OpenAI, Cohere, Voyage, or a local model).

```python
from openai import OpenAI

client = OpenAI()
embeddings = []
for chunk in chunks:
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=chunk["text"]
    )
    embeddings.append({
        "text": chunk["text"],
        "embedding": resp.data[0].embedding,
        "metadata": {
            "source_url": "https://example.com/article",
            "chunk_index": chunk["index"]
        }
    })
```

### Step 3: Store in Vector Database

Insert embeddings into your vector database.

```python
# Example with ChromaDB
import chromadb

client = chromadb.Client()
collection = client.get_or_create_collection("documents")

collection.add(
    ids=[f"chunk_{e['metadata']['chunk_index']}" for e in embeddings],
    embeddings=[e["embedding"] for e in embeddings],
    documents=[e["text"] for e in embeddings],
    metadatas=[e["metadata"] for e in embeddings]
)
```

### Step 4: Query and Retrieve

When a user asks a question, embed the query and retrieve relevant chunks.

```python
query_embedding = client.embeddings.create(
    model="text-embedding-3-small",
    input="What is the main argument of the article?"
).data[0].embedding

results = collection.query(
    query_embeddings=[query_embedding],
    n_results=5
)
context = "\n\n".join(results["documents"][0])
```

### Step 5: Generate Answer with LLM

```python
answer = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": f"Answer based on this context:\n\n{context}"},
        {"role": "user", "content": "What is the main argument of the article?"}
    ]
)
```

### Batch RAG Ingestion

For ingesting many sources at once:

```python
urls = [
    "https://example.com/article1",
    "https://example.com/article2",
    "https://youtube.com/watch?v=VIDEO_ID",
    "https://reddit.com/r/programming/comments/abc/post/"
]

response = requests.post(
    f"{BASE}/batch",
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
    json={"urls": urls, "chunk_size": 1000, "chunk_overlap": 150}
)

for result in response.json()["data"]["results"]:
    if result["success"]:
        for chunk in result["data"].get("chunks", []):
            # embed and store each chunk
            pass
```

---

## Common Questions

**How do I extract content from a YouTube video?**
Use the /youtube/transcript endpoint. Pass the video URL and optionally a language code. ContentAPI handles anti-bot detection, supports auto-generated and manual captions, and returns timestamped segments.

**How do I extract content from a web page?**
Use the /web/extract endpoint. It returns clean text content plus a semantic structure tree. Smart Fetch ensures high reliability even against bot detection. PDF files are auto-detected and extracted.

**How do I get tweets or Twitter threads?**
Use /twitter/tweet for a single tweet or /twitter/thread for the full thread. Both twitter.com and x.com URLs are supported.

**How do I get Reddit posts and comments?**
Use /reddit/post with the Reddit post URL. Returns the post body and comments.

**How do I search the web and extract content from results?**
Use /web/search with the q parameter for your query. Add extract=true to also extract full content from each result URL.

**How do I process multiple URLs at once?**
Use POST /batch with a JSON body containing a urls array (up to 50 URLs). Source types are auto-detected.

**How do I chunk content for RAG?**
Add chunk_size and chunk_overlap query parameters to any content endpoint. The response will include a chunks array with indexed text segments.

**What languages are supported for YouTube transcripts?**
All languages that have captions on the video. Use /youtube/languages to check available languages before requesting a transcript.

**Can ContentAPI extract PDFs?**
Yes. Pass a PDF URL to /web/extract and the text is extracted automatically. No special parameters needed.

**What happens when a website blocks the request?**
Smart Fetch tries 5 strategies in sequence: direct fetch, geo-matched proxy, proxy rotation, Google Cache, and Archive.org fallback. Most content is accessible through at least one strategy.

**How do I handle rate limits?**
When you hit the rate limit (HTTP 429), wait and retry. The Retry-After header indicates how long to wait. Free plan allows 10 requests/min; Pro allows 300/min.

**What is the difference between content and structure in web responses?**
The content field is plain text extracted from the page. The structure field is a semantic JSON tree preserving the document hierarchy (headings, paragraphs, lists, tables, code blocks, etc.).

**Is there a way to use ContentAPI without writing code?**
Yes. Install the MCP server (contentapi-mcp-server) and use it with Claude Desktop, Cursor, or any MCP-compatible AI assistant. The AI can call ContentAPI tools directly.

---

## URL Monitor (Change Detection)

Monitor any URL for content changes. ContentAPI periodically checks the page, detects differences, and sends webhook notifications when changes are found.

### POST /monitor

Create a new URL monitor:

```
curl -X POST "https://api.getcontentapi.com/api/v1/monitor" \
  -H "X-API-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://competitor.com/pricing",
    "webhook_url": "https://myapp.com/hook",
    "interval_hours": 24,
    "notify_on": "any_change",
    "format": "markdown"
  }'
```

Parameters:
- url (required): The URL to monitor
- interval_hours (required): How often to check (min depends on plan)
- webhook_url: URL to receive change notifications
- notify_on: "any_change" (default)
- format: "markdown" (default) or "text"

Response:
```json
{
  "success": true,
  "data": {
    "monitor_id": "mon_abc123",
    "url": "https://competitor.com/pricing",
    "webhook_url": "https://myapp.com/hook",
    "interval_hours": 24,
    "status": "active",
    "next_check": "2026-02-15T18:00:00Z",
    "created_at": "2026-02-14T18:00:00Z"
  }
}
```

### GET /monitor

List all your monitors:

```
curl "https://api.getcontentapi.com/api/v1/monitor" \
  -H "X-API-Key: your_api_key"
```

### GET /monitor/:id

Get monitor details and change history:

```
curl "https://api.getcontentapi.com/api/v1/monitor/mon_abc123" \
  -H "X-API-Key: your_api_key"
```

Response includes checks array with diff summaries, word counts, and timestamps.

### DELETE /monitor/:id

Delete a monitor:

```
curl -X DELETE "https://api.getcontentapi.com/api/v1/monitor/mon_abc123" \
  -H "X-API-Key: your_api_key"
```

### Webhook Payload

When a change is detected, the webhook receives:
```json
{
  "event": "monitor.change_detected",
  "monitor_id": "mon_abc123",
  "url": "https://competitor.com/pricing",
  "checked_at": "2026-02-14T18:00:00Z",
  "diff_summary": "Content changed: 5 lines added, 2 removed (+10 words)",
  "word_count_before": 500,
  "word_count_after": 510,
  "current_content_preview": "First 500 chars of new content..."
}
```

### Plan Limits

| Plan       | Max Monitors | Min Interval |
|------------|-------------|-------------|
| Free       | 3           | 24 hours    |
| Pro        | 50          | 1 hour      |
| Enterprise | Unlimited   | 15 minutes  |

Each check costs 1 credit (deducted automatically).

---

## Edge Cases and Notes

- YouTube URLs can be in any format: youtube.com/watch?v=ID, youtu.be/ID, or just the video ID.
- Twitter/X URLs work with both twitter.com and x.com domains.
- Reddit URLs should be full post URLs including the subreddit and post ID.
- Web extraction works on most public web pages. JavaScript-rendered SPAs may return limited content.
- The expand parameter only applies to /web/extract. It has no effect on other endpoints.
- Batch requests that partially fail still return results for successful URLs.
- Credits are consumed per URL, not per request. A batch of 10 URLs uses 10 credits.
- The free tier resets monthly. No credit card is required.

---

## Links

- Website: https://getcontentapi.com
- Documentation: https://getcontentapi.com/docs
- API Base URL: https://api.getcontentapi.com/api/v1
- Pricing: https://getcontentapi.com/pricing
- MCP Package: https://npmjs.com/package/contentapi-mcp-server
- Quick Reference: https://getcontentapi.com/llms.txt