> ## Documentation Index
> Fetch the complete documentation index at: https://firecrawl-docs-firecrawl-replit-connector.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# 快速开始

> Firecrawl 可将整个网站转换为适配 LLM 的 Markdown

<img className="block" src="https://mintlify.s3.us-west-1.amazonaws.com/firecrawl-docs-firecrawl-replit-connector/images/turn-websites-into-llm-ready-data--firecrawl.jpg" alt="浅色主视觉" />

<div id="welcome-to-firecrawl">
  ## 欢迎使用 Firecrawl
</div>

[Firecrawl](https://firecrawl.dev?ref=github) 是一项 API 服务，可接收 URL、进行爬取，并将其转换为干净的 Markdown。我们会爬取所有可访问的子页面，并为每个页面提供干净的 Markdown。无需提供 sitemap。

<div id="how-to-use-it">
  ## 如何使用？
</div>

我们提供托管版的易用 API。你可以在[此处](https://firecrawl.dev/playground)找到 playground 和文档。你也可以选择自托管后端。

从以下资源开始：

* [x] **API**： [文档](https://docs.firecrawl.dev/api-reference/introduction)
* [x] **SDKs**： [Python](https://docs.firecrawl.dev/sdks/python)、[Node](https://docs.firecrawl.dev/sdks/node)、[Go](https://docs.firecrawl.dev/sdks/go)、[Rust](https://docs.firecrawl.dev/sdks/rust)
* [x] **LLM 框架**： [LangChain（Python）](https://python.langchain.com/docs/integrations/document_loaders/firecrawl/)、[LangChain（JS）](https://js.langchain.com/docs/integrations/document_loaders/web_loaders/firecrawl)、[LlamaIndex](https://docs.llamaindex.ai/en/latest/examples/data_connectors/WebPageDemo/#using-firecrawl-reader)、[Crew.ai](https://docs.crewai.com/)、[Composio](https://composio.dev/tools/firecrawl/all)、[PraisonAI](https://docs.praison.ai/firecrawl/)、[Superinterface](https://superinterface.ai/docs/assistants/functions/firecrawl)、[Vectorize](https://docs.vectorize.io/integrations/source-connectors/firecrawl)
* [x] **低代码框架**： [Dify](https://dify.ai/blog/dify-ai-blog-integrated-with-firecrawl)、[Langflow](https://docs.langflow.org/)、[Flowise AI](https://docs.flowiseai.com/integrations/langchain/document-loaders/firecrawl)、[Cargo](https://docs.getcargo.io/integration/firecrawl)、[Pipedream](https://pipedream.com/apps/firecrawl/)
* [x] **其他**： [Zapier](https://zapier.com/apps/firecrawl/integrations)、[Pabbly Connect](https://www.pabbly.com/connect/integrations/firecrawl/)
* [ ] 需要某个 SDK 或集成？欢迎提交 issue 告诉我们。

**自托管：** 请参考[此处](/zh/contributing/self-host)的指南。

<div id="api-key">
  ### API Key
</div>

要使用 API，你需要在 [Firecrawl](https://firecrawl.dev) 注册并获取 API Key。

<div id="crawling">
  ## 爬取
</div>

用于爬取一个 URL 及其所有可访问的子页面。该操作会提交一个爬取任务，并返回任务 ID 以便查询爬取状态。

<div id="installation">
  ### 安装
</div>

<CodeGroup>
  ```bash Python theme={null}
  pip install firecrawl-py
  ```

  ```bash JavaScript theme={null}
  npm install firecrawl
  ```

  ```bash Go theme={null}
  go get github.com/mendableai/firecrawl-go
  ```

  ```toml Rust theme={null}
  # 在 Cargo.toml 中添加以下内容

  [dependencies]
  firecrawl = "^0.1"
  tokio = { version = "^1", features = ["full"] }
  serde = { version = "^1.0", features = ["derive"] }
  serde_json = "^1.0"
  uuid = { version = "^1.10", features = ["v4"] }

  [build-dependencies]
  tokio = { version = "1", features = ["full"] }
  ```
</CodeGroup>

<div id="usage">
  ### 用法
</div>

<CodeGroup>
  ```python Python theme={null}
  from firecrawl import Firecrawl

  app = Firecrawl(api_key="YOUR_API_KEY")

  crawl_result = app.crawl_url('docs.firecrawl.dev', {'crawlerOptions': {'excludes': ['blog/*']}})

  # 获取 Markdown
  for result in crawl_result:
      print(result['markdown'])
  ```

  ```js JavaScript theme={null}
  import { Firecrawl } from "firecrawl";

  // 使用你的 API 密钥初始化 Firecrawl
  const app = new Firecrawl({ apiKey: "YOUR_API_KEY" });

  // 抓取网站
  const crawlResult = await app.crawlUrl("docs.firecrawl.dev", {
    crawlerOptions: { excludes: ["blog/*"] },
  });

  // 输出 Markdown
  console.log(crawlResult.map((result) => result.markdown));
  ```

  ```go Go theme={null}
  import (
    "fmt"
    "log"

    "github.com/mendableai/firecrawl-go"
  )

  func main() {
    // 使用你的 API 密钥初始化 Firecrawl
    app, err := firecrawl.NewFirecrawlApp("YOUR_API_KEY")
    if err != nil {
      log.Fatalf("初始化 Firecrawl 失败: %v", err)
    }

    // 抓取网站
    params := map[string]any{
      "crawlerOptions": map[string]any{
        "excludes": []string{"blog/*"},
      },
    }
    crawlResult, err := app.CrawlURL("docs.firecrawl.dev", params)
    if err != nil {
      log.Fatalf("抓取时发生错误: %v", err)
    }

    // 获取 Markdown
    for _, result := range crawlResult {
      fmt.Println(result.Markdown)
    }
  }
  ```

  ```rust Rust theme={null}
  use firecrawl::Firecrawl;

  #[tokio::main]
  async fn main() {
    // 使用 API 密钥初始化 Firecrawl
    let api_key = "YOUR_API_KEY";
    let api_url = "https://api.firecrawl.dev";
    let app = Firecrawl::new(api_key, api_url).expect("初始化 Firecrawl 失败");

    // 抓取 URL
    let crawl_params = json!({
      "crawlerOptions": {
          "excludes": ["blog/*"]
      }
    });

    let crawl_result = app
        .crawl_url("https://example.com", Some(crawl_params), true, 2, None)
        .await;

    // 打印抓取结果
    match crawl_result {
        Ok(data) => println!("抓取结果:\n{}", data),
        Err(e) => eprintln!("抓取失败: {}", e),
    }
  }
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.firecrawl.dev/v0/crawl \
      -H 'Content-Type: application/json' \
      -H 'Authorization: Bearer YOUR_API_KEY' \
      -d '{
        "url": "https://docs.firecrawl.dev"
      }'
  ```
</CodeGroup>

如果你不使用 SDK，或更倾向于使用 webhook 或其他轮询方式，可以将 `wait_until_done` 设置为 `false`。
这将返回一个 jobId。

对于 cURL，`/crawl` 始终会返回一个 jobId，可用它来检查抓取状态。

```json theme={null}
{ "jobId": "1234-5678-9101" }
```

<div id="check-crawl-job">
  ### 检查爬取任务
</div>

用于查看爬取任务的状态并获取其结果。

<CodeGroup>
  ```python Python theme={null}
  status = app.check_crawl_status(job_id)
  ```

  ```js JavaScript theme={null}
  const status = await app.checkCrawlStatus(jobId);
  ```

  ```go Go theme={null}
  status, err := app.CheckCrawlStatus(jobId)
  if err != nil {
    log.Fatalf("Failed to check crawl status: %v", err)
  }
  ```

  ```rust Rust theme={null}
  let status = match app.check_crawl_status(jobId).await {
      Ok(status) => status,
      Err(e) => panic!("Failed to check crawl status: {:?}", e),
  };

  println!("Crawl Status: {:?}", status);
  ```

  ```bash cURL theme={null}
  curl -X GET https://api.firecrawl.dev/v0/crawl/status/1234-5678-9101 \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY'
  ```
</CodeGroup>

<div id="response">
  #### 响应
</div>

```json theme={null}
{
  "status": "已完成",
  "current": 22,
  "total": 22,
  "data": [
    {
      "content": "原始内容",
      "markdown": "# Markdown 内容",
      "provider": "web-scraper",
      "metadata": {
        "title": "Firecrawl | 为你的 LLM 稳定可靠地抓取网页",
        "description": "用于客户体验与销售的 AI"
        "language": null,
        "sourceURL": "https://docs.firecrawl.dev/"
      }
    }
  ]
}
```

<div id="scraping">
  ## 抓取
</div>

要抓取单个 URL，请使用 `scrape_url` 方法。该方法以 URL 作为参数，返回包含抓取结果的字典。

<CodeGroup>
  ```python Python theme={null}
  from firecrawl import Firecrawl

  app = Firecrawl(api_key="YOUR_API_KEY")

  content = app.scrape_url("https://docs.firecrawl.dev")
  ```

  ```JavaScript JavaScript theme={null}
  import { Firecrawl } from 'firecrawl-js';

  const app = new Firecrawl({ apiKey: 'YOUR_API_KEY' });

  const content = await app.scrapeUrl('https://docs.firecrawl.dev');
  ```

  ```go Go theme={null}
  import (
    "log"

    "github.com/mendableai/firecrawl-go"
  )

  func main() {
    app, err := firecrawl.NewFirecrawlApp("YOUR_API_KEY")
    if err != nil {
      log.Fatalf("Failed to initialize Firecrawl: %v", err)
    }

    content, err := app.ScrapeURL("docs.firecrawl.dev", nil)
    if err != nil {
      log.Fatalf("Failed to scrape URL: %v", err)
    }
  }
  ```

  ```rust Rust theme={null}
  use firecrawl::Firecrawl;

  #[tokio::main]
  async fn main() {
      // 使用 API 密钥初始化 Firecrawl
      let api_key = "YOUR_API_KEY";
      let api_url = "https://api.firecrawl.dev";
      let app = Firecrawl::new(api_key, api_url).expect("Failed to initialize Firecrawl");

      // 抓取 URL
      let scrape_result = app.scrape_url("https://example.com", None).await;

      // 打印抓取结果
    match scrape_result {
      Ok(data) => println!("Scrape Result:\n{}", data["markdown"]),
      Err(e) => eprintln!("Scrape failed: {}", e),
    }
  }
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.firecrawl.dev/v0/scrape \
      -H 'Content-Type: application/json' \
      -H 'Authorization: Bearer YOUR_API_KEY' \
      -d '{
        "url": "https://docs.firecrawl.dev"
      }'
  ```
</CodeGroup>

### 返回

```json theme={null}
{
  "success": true,
  "data": {
    "markdown": "<字符串>",
    "content": "<字符串>",
    "html": "<字符串>",
    "rawHtml": "<字符串>",
    "metadata": {
      "title": "<字符串>",
      "description": "<字符串>",
      "language": "<字符串>",
      "sourceURL": "<字符串>",
      "<其他元数据>": "<字符串>",
      "pageStatusCode": 123,
      "pageError": "<字符串>"
    },
    "llm_extraction": {},
    "warning": "<字符串>"
  }
}
```

<div id="extraction">
  ## 提取
</div>

借助 LLM 提取功能，你可以轻松从任意 URL 提取结构化数据。我们也支持 Pydantic 架构，便于定义数据结构。使用方法如下：

<CodeGroup>
  ```python Python theme={null}
  class ArticleSchema(BaseModel):
      title: str
      points: int 
      by: str
      commentsURL: str

  class TopArticlesSchema(BaseModel):
  top: List[ArticleSchema] = Field(..., max_items=5, description="Top 5 stories")

  data = app.scrape_url('https://news.ycombinator.com', {
  'extractorOptions': {
  'extractionSchema': TopArticlesSchema.model_json_schema(),
  'mode': 'llm-extraction'
  },
  'pageOptions':{
  'onlyMainContent': True
  }
  })
  print(data["llm_extraction"])
  ```

  ```js JavaScript theme={null}
  import { Firecrawl } from "firecrawl";
  import { z } from "zod";

  const app = new Firecrawl({
    apiKey: "fc-YOUR_API_KEY",
  });

  // 定义提取内容所用的模式
  const schema = z.object({
    top: z
      .array(
        z.object({
          title: z.string(),
          points: z.number(),
          by: z.string(),
          commentsURL: z.string(),
        })
      )
      .length(5)
      .describe("Top 5 stories on Hacker News"),
  });

  const scrapeResult = await app.scrapeUrl("https://news.ycombinator.com", {
    extractorOptions: { extractionSchema: schema },
  });

  console.log(scrapeResult.data["llm_extraction"]);
  ```

  ```go Go theme={null}
  import (
    "fmt"
    "log"

    "github.com/mendableai/firecrawl-go"
  )

  app, err := NewFirecrawlApp(TEST_API_KEY, API_URL)
  if err != nil {
    log.Fatalf("初始化 Firecrawl 失败：%v", err)
  }

  params := map[string]any{
    "extractorOptions": ExtractorOptions{
      Mode:             "llm-extraction",
      ExtractionPrompt: "Based on the information on the page, find what the company's mission is and whether it supports SSO, and whether it is open source",
      ExtractionSchema: map[string]any{
        "type": "object",
        "properties": map[string]any{
          "company_mission": map[string]string{"type": "string"},
          "supports_sso":    map[string]string{"type": "boolean"},
          "is_open_source":  map[string]string{"type": "boolean"},
        },
        "required": []string{"company_mission", "supports_sso", "is_open_source"},
      },
    },
  }

  scrapeResult, err := app.ScrapeURL("https://news.ycombinator.com", params)
  if err != nil {
    log.Fatalf("抓取 URL 失败：%v", err)
  }
  fmt.Println(scrapeResult.LLMExtraction)
  ```

  ```rust Rust theme={null}
  use firecrawl::Firecrawl;

  #[tokio::main]
  async fn main() {
      // 使用 API 密钥初始化 Firecrawl
      let api_key = "YOUR_API_KEY";
      let api_url = "https://api.firecrawl.dev";
      let app = Firecrawl::new(api_key, api_url).expect("Failed to initialize Firecrawl");

      // 定义用于提取内容的架构
      let json_schema = json!({
          "type": "object",
          "properties": {
              "top": {
                  "type": "array",
                  "items": {
                      "type": "object",
                      "properties": {
                          "title": {"type": "string"},
                          "points": {"type": "number"},
                          "by": {"type": "string"},
                          "commentsURL": {"type": "string"}
                      },
                      "required": ["title", "points", "by", "commentsURL"]
                  },
                  "minItems": 5,
                  "maxItems": 5,
                  "description": "Top 5 stories on Hacker News"
              }
          },
          "required": ["top"]
      });

      let llm_extraction_params = json!({
          "extractorOptions": {
              "extractionSchema": json_schema,
              "mode": "llm-extraction"
          },
          "pageOptions": {
              "onlyMainContent": true
          }
      });

      let llm_extraction_result = app
          .scrape_url("https://news.ycombinator.com", Some(llm_extraction_params))
          .await;
      match llm_extraction_result {
          Ok(data) => println!("LLM Extraction Result:\n{}", data["llm_extraction"]),
          Err(e) => eprintln!("LLM Extraction failed: {}", e),
      }
  }
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.firecrawl.dev/v0/scrape \
      -H 'Content-Type: application/json' \
      -H 'Authorization: Bearer YOUR_API_KEY' \
      -d '{
        "url": "https://docs.firecrawl.dev/",
        "extractorOptions": {
          "mode": "llm-extraction",
          "extractionPrompt": "Based on the information on the page, extract the information from the schema. ",
          "extractionSchema": {
            "type": "object",
            "properties": {
              "company_mission": {
                        "type": "string"
              },
              "supports_sso": {
                        "type": "boolean"
              },
              "is_open_source": {
                        "type": "boolean"
              },
              "is_in_yc": {
                        "type": "boolean"
              }
            },
            "required": [
              "company_mission",
              "supports_sso",
              "is_open_source",
              "is_in_yc"
            ]
          }
        }
      }'
  ```
</CodeGroup>

<div id="contributing">
  ## 参与贡献
</div>

我们欢迎各类贡献！在提交 pull request 之前，请先阅读我们的[贡献指南](https://github.com/firecrawl/firecrawl/blob/main/CONTRIBUTING.md)。