> ## Documentation Index
> Fetch the complete documentation index at: https://firecrawl-docs-firecrawl-replit-connector.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# SourceSync.ai

> Firecrawl 与 SourceSync.ai 集成，提供网页抓取能力。

[SourceSync.ai](https://sourcesync.ai) 是一个“检索增强生成”（Retrieval Augmented Generation, RAG）即服务平台，帮助你使用自有数据构建 AI 应用。本文介绍如何将 Firecrawl 与 SourceSync.ai 搭配使用，以启用网页抓取能力。

<div id="setup">
  ## 设置
</div>

1. 先在你的 [Firecrawl 控制台](https://www.firecrawl.dev/app)获取 Firecrawl API 密钥。

2. 将你的 SourceSync.ai 命名空间配置为使用 Firecrawl 作为网页抓取提供方：

<CodeGroup>
  ```bash cURL theme={null}
  curl -X PATCH https://api.sourcesync.ai/v1/namespaces/YOUR_NAMESPACE_ID \
    -H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "webScraperConfig": {
        "provider": "FIRECRAWL",
        "apiKey": "YOUR_FIRECRAWL_API_KEY"
      }
    }'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch(
    'https://api.sourcesync.ai/v1/namespaces/YOUR_NAMESPACE_ID',
    {
      method: 'PATCH',
      headers: {
        Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        webScraperConfig: {
          provider: 'FIRECRAWL',
          apiKey: 'YOUR_FIRECRAWL_API_KEY',
        },
      }),
    }
  )
  ```

  ```python Python theme={null}
  import requests

  response = requests.patch(
      'https://api.sourcesync.ai/v1/namespaces/YOUR_NAMESPACE_ID',
      headers={
          'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
          'Content-Type': 'application/json',
      },
      json={
          'webScraperConfig': {
              'provider': 'FIRECRAWL',
              'apiKey': 'YOUR_FIRECRAWL_API_KEY',
          },
      },
  )
  ```
</CodeGroup>

<div id="usage">
  ## 用法
</div>

完成配置后，你可以将 SourceSync.ai 的网页抓取端点与 Firecrawl 的能力配合使用。以下是主要的导入方式：

<div id="url-list-ingestion">
  ### URL 列表导入
</div>

抓取指定 URL：

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST https://api.sourcesync.ai/v1/ingest/urls \
    -H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "namespaceId": "YOUR_NAMESPACE_ID",
      "ingestConfig": {
        "source": "URLS_LIST",
        "config": {
          "urls": [
            "https://example.com/page1",
            "https://example.com/page2"
          ],
          "scrapeOptions": {
            "includeSelectors": ["article", "main"],
            "excludeSelectors": [".navigation", ".footer"]
          }
        }
      }
    }'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch('https://api.sourcesync.ai/v1/ingest/urls', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      namespaceId: 'YOUR_NAMESPACE_ID',
      ingestConfig: {
        source: 'URLS_LIST',
        config: {
          urls: ['https://example.com/page1', 'https://example.com/page2'],
          scrapeOptions: {
            includeSelectors: ['article', 'main'],
            excludeSelectors: ['.navigation', '.footer'],
          },
        },
      },
    }),
  })
  ```

  ```python Python theme={null}
  response = requests.post(
      'https://api.sourcesync.ai/v1/ingest/urls',
      headers={
          'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
          'Content-Type': 'application/json',
      },
      json={
          'namespaceId': 'YOUR_NAMESPACE_ID',
          'ingestConfig': {
              'source': 'URLS_LIST',
              'config': {
                  'urls': ['https://example.com/page1', 'https://example.com/page2'],
                  'scrapeOptions': {
                      'includeSelectors': ['article', 'main'],
                      'excludeSelectors': ['.navigation', '.footer'],
                  },
              },
          },
      },
  )
  ```
</CodeGroup>

<div id="website-crawling">
  ### 网站爬取
</div>

按自定义规则爬取整个网站：

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST https://api.sourcesync.ai/v1/ingest/website \
    -H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "namespaceId": "YOUR_NAMESPACE_ID",
      "ingestConfig": {
        "source": "WEBSITE",
        "config": {
          "url": "https://example.com",
          "maxDepth": 3,
          "maxLinks": 100,
          "includePaths": ["/docs", "/blog"],
          "excludePaths": ["/admin"],
          "scrapeOptions": {
            "includeSelectors": ["article", "main"],
            "excludeSelectors": [".navigation", ".footer"]
          }
        }
      }
    }'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch('https://api.sourcesync.ai/v1/ingest/website', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      namespaceId: 'YOUR_NAMESPACE_ID',
      ingestConfig: {
        source: 'WEBSITE',
        config: {
          url: 'https://example.com',
          maxDepth: 3,
          maxLinks: 100,
          includePaths: ['/docs', '/blog'],
          excludePaths: ['/admin'],
          scrapeOptions: {
            includeSelectors: ['article', 'main'],
            excludeSelectors: ['.navigation', '.footer'],
          },
        },
      },
    }),
  })
  ```

  ```python Python theme={null}
  response = requests.post(
      'https://api.sourcesync.ai/v1/ingest/website',
      headers={
          'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
          'Content-Type': 'application/json',
      },
      json={
          'namespaceId': 'YOUR_NAMESPACE_ID',
          'ingestConfig': {
              'source': 'WEBSITE',
              'config': {
                  'url': 'https://example.com',
                  'maxDepth': 3,
                  'maxLinks': 100,
                  'includePaths': ['/docs', '/blog'],
                  'excludePaths': ['/admin'],
                  'scrapeOptions': {
                      'includeSelectors': ['article', 'main'],
                      'excludeSelectors': ['.navigation', '.footer'],
                  },
              },
          },
      },
  )
  ```
</CodeGroup>

<div id="sitemap-processing">
  ### 站点地图处理
</div>

处理站点地图中的所有 URL：

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST https://api.sourcesync.ai/v1/ingest/sitemap \
    -H "Authorization: Bearer YOUR_SOURCE_SYNC_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "namespaceId": "YOUR_NAMESPACE_ID",
      "ingestConfig": {
        "source": "SITEMAP",
        "config": {
          "url": "https://example.com/sitemap.xml",
          "scrapeOptions": {
            "includeSelectors": ["article", "main"],
            "excludeSelectors": [".navigation", ".footer"]
          }
        }
      }
    }'
  ```

  ```javascript JavaScript theme={null}
  const response = await fetch('https://api.sourcesync.ai/v1/ingest/sitemap', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${SOURCE_SYNC_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      namespaceId: 'YOUR_NAMESPACE_ID',
      ingestConfig: {
        source: 'SITEMAP',
        config: {
          url: 'https://example.com/sitemap.xml',
          scrapeOptions: {
            includeSelectors: ['article', 'main'],
            excludeSelectors: ['.navigation', '.footer'],
          },
        },
      },
    }),
  })
  ```

  ```python Python theme={null}
  response = requests.post(
      'https://api.sourcesync.ai/v1/ingest/sitemap',
      headers={
          'Authorization': f'Bearer {SOURCE_SYNC_API_KEY}',
          'Content-Type': 'application/json',
      },
      json={
          'namespaceId': 'YOUR_NAMESPACE_ID',
          'ingestConfig': {
              'source': 'SITEMAP',
              'config': {
                  'url': 'https://example.com/sitemap.xml',
                  'scrapeOptions': {
                      'includeSelectors': ['article', 'main'],
                      'excludeSelectors': ['.navigation', '.footer'],
                  },
              },
          },
      },
  )
  ```
</CodeGroup>

<div id="features">
  ## 功能
</div>

将 Firecrawl 与 SourceSync.ai 搭配使用时，你可以获得：

* 支持 JavaScript 渲染
* 自动速率限制
* 基于 CSS 选择器的内容提取
* 递归抓取并可控制深度
* 站点地图处理

<div id="resources">
  ## 资源
</div>

* [SourceSync.ai 文档](https://sourcesync.ai)
* [网页抓取指南](https://sourcesync.ai/web-scraping)
* [API 参考](https://sourcesync.ai/api-reference/data-ingestion#ingest-urls)

如需更多支持：

* 邮件： [support@sourcesync.ai](mailto:support@sourcesync.ai)
* Discord： [加入社区](https://discord.gg/Fx3GnFKnRT)