🌐 Firecrawl website content extractor

🌐 Firecrawl Website Content Extractor (n8n Workflow)

This n8n automation workflow uses Firecrawl API to extract structured data (e.g., quotes and authors) from web pages — such as Quotes to Scrape — and handles retries in case of delayed extraction.

---

🔁 Workflow Overview

🎯 Purpose:

• Crawl and extract structured web data using Firecrawl
• Wait for asynchronous scraping to complete
• Retrieve and validate results
• Support retries if content is not ready

---

🔧 Step-by-Step Node Breakdown

1. 🧪 Manual Trigger

• Node: When clicking ‘Test workflow’
• Used to manually test or execute the workflow during setup or debugging.

---

2. 📤 Firecrawl Extract API Request

• Node: Extract
• Sends a POST request to https://api.firecrawl.dev/v1/extract
• Payload includes:
  • urls: List of pages to crawl (https://quotes.toscrape.com/*)
  • prompt: "Extract all quotes and their corresponding authors from the website."
  • schema: JSON schema defining expected structure (quotes[], each with text and author)

> 📌 Uses an HTTP Header Auth credential for Firecrawl API

---

3. ⏱️ Wait for 30 Seconds

• Node: 30 Secs
• Gives Firecrawl time to finish processing in the background
• Prevents hitting the API before results are ready

---

4. 📥 Get Results

• Node: Get Results
• Performs a GET request to the status URL using {{ $('Extract').item.json.id }} to retrieve extraction results.

---

5. ✅❌ Condition Check

• Node: If
• Checks if the data array is empty (i.e., no results yet)
• If data is empty:
  • Waits 10 more seconds and retries
• If data is available:
  • Passes data to the next step (e.g., processing or storage)

---

6. 🔁 Retry Delay

• Node: 10 Seconds
• Waits briefly before sending another GET request to Firecrawl

---

7. 🛠️ Edit Fields (Optional Output Formatting)

• Node: Edit Fields
• Placeholder to structure or format the extracted results (quotes and authors)

---

🧾 Sticky Note: Firecrawl Setup Guide

Included as an embedded reference:

• 🔗 10% Firecrawl Discount
• 🧰 Instructions to:
  • Add Firecrawl API credentials in n8n
  • Use Firecrawl Community Node for self-hosted instances
  • Set up the schema and prompt for targeted data extraction

---

✅ Key Features

• 🔌 API-based crawling with schema-structured output
• ⏱️ Smart waiting + retry mechanism
• 🧠 AI prompt integration for intelligent data parsing
• ⚙️ Flexible for different URLs, prompts, and schemas

---

📦 Sample Output Schema

{
  "quotes": [
    {
      "text": "The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.",
      "author": "Albert Einstein"
    },
    {
      "text": "It is our choices, Harry, that show what we truly are, far more than our abilities.",
      "author": "J.K. Rowling"
    }
  ]
}