Scrape ProductHunt using Google Gemini

Open on n8n.io

$20/month : Unlimited workflows

2500 executions/month

Try free

THE #1 IN WEB SCRAPING

Scrape any website without limits

Try free

HOSTINGER

Early Deal
DISCOUNT 20%

Self-hosted n8n

Unlimited workflows - from $4.99/mo

Try free

#1 hub for scraping, AI & automation

6000+ actors - $5 credits/mo

Try free

Important notice

This workflow is provided as-is. Please review and test before using in production.

1. Workflow Overview

This workflow automates the extraction of product data from Product Hunt by combining webhook interactions, HTML processing, AI based data analysis, and structured output formatting. It is designed...

Best for

Market Research automation workflows
AI Summarization automation workflows
intermediate n8n builders looking for reusable templates

Tools used

n8n-nodes-base.webhook, n8n-nodes-base.httprequest, n8n-nodes-base.code, @n8n/n8n-nodes-langchain.chainllm, @n8n/n8n-nodes-langchain.lmchatgooglegemini, @n8n/n8n-nodes-langchain.outputparserstructured, n8n-nodes-base.respondtowebhook, n8n-nodes-base.stickynote

Source and attribution

This workflow is cataloged by N8N Workflows and links back to its original n8n.io source page by Mauricio Perera.

Original n8n.io source

1.1 Workflow description

Title: Scrape ProductHunt using Google Gemini
Workflow name: Scrape ProductHunt using Google Gemini

Workflow Description: Product Data Extractor

This workflow automates the extraction of product data from Product Hunt by combining webhook interactions, HTML processing, AI-based data analysis, and structured output formatting. It is designed to handle incoming requests dynamically and return detailed JSON responses for further usage.

Overview

The workflow processes a product name submitted through a webhook. It fetches the corresponding Product Hunt page, extracts and analyzes inline scripts, and structures the data into a well-defined JSON format using AI tools. The final JSON response is returned to the client through the webhook.

Workflow Steps

1. Webhook Listener

Node: Receive Product Request
Function: Captures incoming requests containing the product name to process.
Details: Accepts HTTP requests and extracts the product parameter from the query string, such as <custom_webhook_url>/?product=epigram.

2. Fetch Product HTML

Node: Fetch Product HTML
Function: Sends an HTTP request to retrieve the HTML content of the specified Product Hunt page.
Details: Constructs a dynamic URL using the product name and fetches the page data.

3. Extract Inline Scripts

Node: Extract Inline Scripts
Function: Parses the HTML content to extract inline scripts located within the <head> section.
Details: Excludes scripts containing src attributes and validates the presence of inline scripts.

4. Process Data with LLM

Node: Process Script with LLM
Function: Analyzes the extracted scripts using a language model to identify key product data.
Details: Processes the script to derive structured and meaningful insights.

5. Refine Data with Google Gemini

Node: Analyze Script with Google Gemini
Function: Leverages Google Gemini AI for enhanced analysis of script data.
Details: Ensures the extracted data is precise and enriched.

6. Format Product Data to JSON

Node: Format Product Data to JSON
Function: Structures the processed data into a clean JSON format.
Details: Defines a schema to ensure all relevant fields are included in the output.

7. Send JSON Response to Client

Node: Send JSON Response to Client
Function: Returns the final structured JSON response to the client.
Details: Sends the response back via the same webhook that initiated the request. For example, <custom_webhook_url>.

Key Features

Versatile Use Cases: This workflow can be used to gather Product Hunt data for creating blog posts or as a tool for AI agents to research products efficiently.
Dynamic Processing: Adapts to various product names through dynamic URL construction.
AI Integration: Utilizes the Gemini 1.5 8B AI model, offering reduced latency and minimal or no cost depending on the use case.
Selector Independence: Functions even if Product Hunt's DOM structure changes, as it does not rely on direct DOM selectors.
Reliable Data Output: A low temperature setting (0) and a precisely defined JSON schema ensure accurate and real data extraction.
Dynamic Processing: Adapts to various product names through dynamic URL construction.
AI Integration: Utilizes advanced language models for data extraction and refinement.
Structured Output: Ensures the output JSON adheres to a predefined schema for consistency.
Error Handling: Includes validations to handle missing or malformed data gracefully.

Customization Options

Limitations

Dependency on Product Hunt: Significant changes to the way Product Hunt loads data on its pages might require modifications to the workflow.
Adaptability: Even if changes occur, the workflow can be updated to maintain functionality due to its reliance on AI and not direct DOM selectors.
Modify the webhook path to suit your application.
Adjust the prompt for the language model to include additional fields.
Extend the JSON schema to capture more data fields as needed.

Expected Output

Performance Metrics

Response Time: Typically ~6 seconds per product.
Accuracy: Data extracted with >95% precision due to the pre-defined JSON schema.

A JSON object containing detailed information about the specified product. Below is an example of a complete response for the product Epigram:

{
  "id": "861675",
  "slug": "epigram",
  "followersCount": 181,
  "name": "Epigram",
  "tagline": "Open-Source, Free, and AI-Powered News in Short",
  "reviewsRating": 0,
  "logoUuid": "735c2528-554c-467c-9dcf-745ee4b8bbdd.png",
  "postsCount": 1,
  "websiteUrl": "https://epigram.news",
  "websiteDomain": "epigram.news",
  "metaTitle": "Epigram - Open-source, free, and ai-powered news in short",
  "postName": "Epigram",
  "postTagline": "Open-source, free, and ai-powered news in short",
  "dailyRank": "3",
  "description": "An open-source, AI-powered news app for busy people. Stay updated with bite-sized news, real-time updates, and in-depth analysis. Experience balanced, trustworthy reporting tailored for fast-paced lifestyles in a sleek, user-friendly interface.",
  "pricingType": "free",
  "userName": "Fazle Rahman",
  "userHeadline": "Co-founder & CEO, Hashnode",
  "userUsername": "fazlerocks",
  "userAvatarUrl": "https://ph-avatars.imgix.net/129147/f84e1796-548b-4d6f-9dcf-745ee4b8bbdd.jpeg",
  "makerName1": "Fazle Rahman",
  "makerHeadline1": "Co-founder & CEO, Hashnode",
  "makerUsername1": "fazlerocks",
  "makerAvatarUrl1": "https://ph-avatars.imgix.net/129147/f84e1796-548b-4d6f-9dcf-745ee4b8bbdd.jpeg",
  "makerName2": "Sandeep Panda",
  "makerHeadline2": "Co-Founder @ Hashnode",
  "makerUsername2": "sandeepg33k",
  "makerAvatarUrl2": "https://ph-avatars.imgix.net/101872/80b0b618-a540-4110-a6d1-74df39675ad0.jpeg",
  "primaryLinkUrl": "https://epigram.news/",
  "media1OriginalHeight": 1080,
  "media1OriginalWidth": 1440,
  "media1ImageUuid": "ac426fd1-3854-4734-b43d-34a5e06347ea.gif",
  "media1MediaType": "video",
  "media1MetadataUrl": "https://www.loom.com/share/b1a48a9b3cac4ba89ce772a3fbcc2847?sid=75efc771-25fa-4ac0-bb1b-5e38fc447deb",
  "media1VideoId": "b1a48a9b3cac4ba89ce772a3fbcc2847",
  "media2OriginalHeight": 630,
  "media2OriginalWidth": 1200,
  "media2ImageUuid": "8521a6bd-7640-487b-abd6-29b9f65fee32",
  "media2MediaType": "image",
  "media2MetadataUrl": null,
  "launchState": "featured",
  "thumbnailImageUuid": "735c2528-554c-467c-9dcf-745ee4b8bbdd.png",
  "link1StoreName": "Website",
  "link1WebsiteName": "epigram.news",
  "link2StoreName": "Github",
  "link2WebsiteName": "github.com",
  "latestScore": 233,
  "launchDayScore": 233,
  "userId": "129147",
  "topic1": "News",
  "topic2": "Open Source",
  "topic3": "Artificial Intelligence",
  "weeklyRank": "24",
  "commentsCount": 20,
  "postUrl": "https://www.producthunt.com/posts/epigram"
}

Target Audience

This workflow is ideal for developers, marketers, and data analysts seeking to automate the extraction and structuring of product data from Product Hunt for analytics, reporting, or integration with other tools.

1.2 Logical Blocks

This catalog entry is organized from the workflow JSON. The node-level section below shows the executable blocks available for review before importing the template.

2. Block-by-Block Analysis

Block 1 - Receive Product Request

Type / Role: n8n-nodes-base.webhook - webhook
Config choices: Version 2

Block 2 - Fetch Product HTML

Type / Role: n8n-nodes-base.httpRequest - httpRequest
Config choices: Version 4.2

Block 3 - Extract Inline Scripts

Type / Role: n8n-nodes-base.code - code
Config choices: Version 2

Block 4 - Process Script with LLM

Type / Role: @n8n/n8n-nodes-langchain.chainLlm - chainLlm
Config choices: Version 1.5

Block 5 - Analyze Script with Google Gemini

Type / Role: @n8n/n8n-nodes-langchain.lmChatGoogleGemini - lmChatGoogleGemini
Config choices: Version 1

Block 6 - Format Product Data to JSON

Type / Role: @n8n/n8n-nodes-langchain.outputParserStructured - outputParserStructured
Config choices: Version 1.2

Block 7 - Send JSON Response to Client

Type / Role: n8n-nodes-base.respondToWebhook - respondToWebhook
Config choices: Version 1.1

Block 8 - Sticky Note

Type / Role: n8n-nodes-base.stickyNote - stickyNote
Config choices: Version 1

Block 9 - Sticky Note1

Type / Role: n8n-nodes-base.stickyNote - stickyNote
Config choices: Version 1

Block 10 - Sticky Note2

Type / Role: n8n-nodes-base.stickyNote - stickyNote
Config choices: Version 1

Block 11 - Sticky Note3

Type / Role: n8n-nodes-base.stickyNote - stickyNote
Config choices: Version 1

Block 12 - Sticky Note4

Type / Role: n8n-nodes-base.stickyNote - stickyNote
Config choices: Version 1

Block 13 - Sticky Note5

Type / Role: n8n-nodes-base.stickyNote - stickyNote
Config choices: Version 1

Block 14 - Sticky Note6

Type / Role: n8n-nodes-base.stickyNote - stickyNote
Config choices: Version 1

3. Summary Table

Workflow	Scrape ProductHunt using Google Gemini
Complexity	intermediate
Nodes	14
Categories	Market Research, AI Summarization
Author	Mauricio Perera
Published	05 Jan 2025

4. Reproducing the Workflow from Scratch

1. Download the workflow JSON

Use the JSON export at /data/workflows/2698/2698.json as the source template for this automation.
2. Import the template into n8n

Open n8n, import the downloaded JSON, and review each node before activating the workflow.
3. Configure credentials and variables

Replace placeholder credentials, API keys, webhook URLs, account IDs, and environment-specific values with your own settings.
4. Test with sample data

Run the workflow manually or in a staging workspace, inspect node output, and confirm downstream systems receive the expected data.
5. Activate and monitor

Enable the workflow only after testing, then monitor executions, errors, and rate limits during the first production runs.

5. General Notes & Resources

Review imported nodes carefully before activation. This catalog entry is intended to help you inspect the workflow structure, understand required services, and find related templates faster.

Node names, credentials, schedules, webhook paths, and external service limits may need adjustment for your workspace.

Download workflow JSON Original n8n.io source Market Research workflows AI Summarization workflows

Frequently asked questions

What does Scrape ProductHunt using Google Gemini do?

What do I need before importing this workflow?

Review the workflow JSON, configure any required credentials in n8n, and test the automation in a safe workspace before using it in production.

Can I customize this workflow?

Yes. Use the block-by-block analysis and the downloadable JSON to inspect each node, then adjust credentials, prompts, schedules, filters, or destinations for your Market Research, AI Summarization use case.

Mauricio Perera

9 workflows

Nodes

n8n-nodes-base.webhook n8n-nodes-base.httprequest n8n-nodes-base.code @n8n/n8n-nodes-langchain.chainllm @n8n/n8n-nodes-langchain.lmchatgooglegemini @n8n/n8n-nodes-langchain.outputparserstructured n8n-nodes-base.respondtowebhook n8n-nodes-base.stickynote

Complexity

intermediate

Published 05 Jan 2025

Likes 0

View on n8n.io Download Workflow

Install path: /data/workflows/2698/2698.json

Share Your Workflow

Have a useful automation to share? Publish it and help the community.

Submit Your Template How to Submit

Related Workflows

Answer BigQuery questions in Microsoft Teams with Claude 3.5 dual agents

## Quick overview Turn Microsoft Teams into a natural language BI tool. This dual-agent Claude workflow intelligently classifies user questions, generates safe BigQuery SQL, executes the query, and returns a plain-English summary alongside a data preview directly to your Teams channel. ## How it works 1. The workflow listens for new messages in a designated Microsoft Teams channel or chat. 2. It checks a 24-hour static data cache for your BigQuery schema. If the cache is missing or expired, it automatically fetches the latest INFORMATION_SCHEMA metadata. 3. A main Claude 3.5 Sonnet agent classifies the user's intent (analytical, conversational, or unsafe) and identifies the relevant database tables. 4. If the intent is analytical, a specialized Claude SQL sub-agent generates a read-only SELECT query tailored exactly to your schema. 5. The workflow executes the query against BigQuery (safely constrained to a maximum of 100 rows). 6. A final Claude agent translates the raw query results into a concise, plain-English summary. 7. The answer is posted back to Teams as an Adaptive Card, complete with a formatted data preview and the generated SQL code. 8. Non-analytical questions or pipeline errors are intelligently routed to receive standard conversational replies in Teams. ## Setup 1. CRITICAL SECURITY: Restrict your Google Cloud Service Account IAM roles to BigQuery Data Viewer and BigQuery Job User ONLY. Do not use a highly privileged account. 2. Configure your Microsoft Teams trigger and posting credentials, specifying the target team and channel for the bot to operate in. 3. Configure your Google BigQuery credentials to allow reading metadata and executing queries. 4. Configure your Anthropic credentials for the three Claude LangChain nodes. 5. Update the "Configure Variables" node with your specific GCP Project ID and BigQuery Dataset ID. ## Requirements - A Google Cloud project with BigQuery enabled and existing structured datasets. - An Anthropic API key with access to the Claude 3.5 Sonnet model. - Microsoft Teams administrator access to authorize the n8n integration. ## Customization - Distributed Environments: If you are running n8n with multiple worker nodes (Queue Mode), you must swap the $getWorkflowStaticData() schema cache logic to read/write from a dedicated Key-Value store (like Redis or Supabase) to ensure state synchronization. - UI Layout: Modify the Create Teams Adaptive Card node to add corporate branding, custom colors, or alter the FactSet data preview structure. ## Additional info This architecture features built-in pipeline protections, including strict 40,000-character payload limits to prevent LLM context window blowouts, and fallback handling for "zero-row" query results to ensure users are never left without a response.

View

Evaluate LinkedIn profiles with Apify and Groq Llama chat model

## Quick overview This workflow collects a LinkedIn profile URL via an n8n form, scrapes the public profile data using the Apify LinkedIn Profile Scraper actor, analyzes the results with a Groq-hosted Llama model, and displays an HTML-formatted evaluation back to the user. ## How it works 1. Receives a LinkedIn Profile URL when a user submits the n8n form. 2. Prepares the LinkedIn URL along with the Apify API token and Groq API key used for downstream requests. 3. Sends the profile URL to the Apify `harvestapi/linkedin-profile-scraper` actor and retrieves the scraped dataset items. 4. Passes the scraped profile JSON to a Groq chat model via an LLM chain to generate an HTML evaluation with a score and recommendations. 5. Displays the generated HTML evaluation to the user as the form completion message. ## Setup 1. Add a valid Apify API token and ensure the Apify actor endpoint (`harvestapi/linkedin-profile-scraper`) is accessible for your account. 2. Add Groq API credentials in n8n and confirm the selected model (meta-llama/llama-4-scout-17b-16e-instruct) is available to your Groq account. 3. Replace the placeholder API key values in the workflow (or move them to n8n credentials/environment variables) and verify the form field label matches `LinkedIn Profile URL`.

View

Validate startup ideas with Google Gemini, Groq, and Gmail

## Quick overview This workflow collects a startup idea via an n8n form, uses Google Gemini to generate a SWOT-style analysis, assigns a viability score, uses Groq to turn the results into a structured validation report, and emails the final report through Gmail. ## How it works 1. Receives a startup idea, industry, and target audience from an n8n Form trigger. 2. Sends the submission to Google Gemini to generate a professional analysis covering the problem, business model, and SWOT factors. 3. Generates a startup viability score (currently a random value from 80–100) and packages it with the Gemini analysis. 4. Sends the score and analysis to Groq (Llama 3.3 70B) to compile a full validation report with an executive summary, market potential, recommendations, and final verdict. 5. Emails the generated validation report to the configured recipient using Gmail. ## Setup 1. Add credentials for Google Gemini (PaLM) API, Groq API, and a Gmail OAuth2 connection. 2. Replace the placeholder recipient address in the Gmail step ("[email protected]") with your target email address before activating the workflow. 3. Publish the n8n form, then share the form URL with the users who should submit their startup ideas for validation.

View

Need Custom Automation?

Get help designing a custom n8n workflow that connects your stack and fits your process.

Scrape ProductHunt using Google Gemini

Workflow preview

Important notice

1. Workflow Overview

Best for

Tools used

Source and attribution

1.1 Workflow description

Overview

Workflow Steps

1. Webhook Listener

2. Fetch Product HTML

3. Extract Inline Scripts

4. Process Data with LLM

5. Refine Data with Google Gemini

6. Format Product Data to JSON

7. Send JSON Response to Client

Key Features

Customization Options

Limitations

Expected Output

Target Audience

1.2 Logical Blocks

2. Block-by-Block Analysis

Block 1 - Receive Product Request

Block 2 - Fetch Product HTML

Block 3 - Extract Inline Scripts

Block 4 - Process Script with LLM

Block 5 - Analyze Script with Google Gemini

Block 6 - Format Product Data to JSON

Block 7 - Send JSON Response to Client

Block 8 - Sticky Note

Block 9 - Sticky Note1

Block 10 - Sticky Note2

Block 11 - Sticky Note3

Block 12 - Sticky Note4

Block 13 - Sticky Note5

Block 14 - Sticky Note6

3. Summary Table

4. Reproducing the Workflow from Scratch

1. Download the workflow JSON

2. Import the template into n8n

3. Configure credentials and variables

4. Test with sample data

5. Activate and monitor

5. General Notes & Resources

Frequently asked questions