Job post to sales lead pipeline with Scrape.do, Apollo.io & OpenAI

Lead Sourcing by Job Posts For Outreach With Scrape.do API & Open AI & Google Sheets

Overview

This n8n workflow automates the complete lead generation process by scraping job postings from Indeed, enriching company data via Apollo.io, identifying decision-makers, and generating personalized LinkedIn outreach messages using OpenAI. It integrates with Scrape.do for reliable web scraping, Apollo.io for B2B data enrichment, OpenAI for AI-powered personalization, and Google Sheets for centralized data storage.

Perfect for: Sales teams, recruiters, business development professionals, and marketing agencies looking to automate their outbound prospecting pipeline.

---

Workflow Components

1. ⏰ Schedule Trigger

Property	Value
Type	Schedule Trigger
Purpose	Automatically initiates workflow on a recurring schedule
Frequency	Weekly (Every Monday)
Time	00:00 UTC

Function: Ensures consistent, hands-off lead generation by running the pipeline automatically without manual intervention.

---

2. 🔍 Scrape.do Indeed API

Property	Value
Type	HTTP Request (GET)
Purpose	Scrapes job listings from Indeed via Scrape.do proxy API
Endpoint	https://api.scrape.do
Output Format	Markdown

Request Parameters:

Parameter	Value	Description
token	API Token	Scrape.do authentication
url	Indeed Search URL	Target job search page
super	true	Uses residential proxies
geoCode	us	US-based content
render	true	JavaScript rendering enabled
device	mobile	Mobile viewport for cleaner HTML
output	markdown	Lightweight text output

Function: Fetches Indeed job listings with anti-bot bypass, returning clean markdown for easy parsing.

---

3. 📋 Parse Indeed Jobs

Property	Value
Type	Code Node (JavaScript)
Purpose	Extracts structured job data from markdown
Mode	Run once for all items

Extracted Fields:

Field	Description	Example
jobTitle	Position title	"Senior Data Engineer"
jobUrl	Indeed job link	"https://indeed.com/viewjob?jk=abc123"
jobId	Indeed job identifier	"abc123"
companyName	Hiring company	"Acme Corporation"
location	City, State	"San Francisco, CA"
salary	Pay range	"$120,000 - $150,000"
jobType	Employment type	"Full-time"
source	Data source	"Indeed"
dateFound	Scrape date	"2025-01-15"

Function: Parses markdown using regex patterns, filters invalid entries, and deduplicates by company name.

---

4. 📊 Add New Company (Google Sheets)

Property	Value
Type	Google Sheets Node
Purpose	Stores parsed job postings for tracking
Operation	Append rows
Target Sheet	"Add New Company"

Function: Creates a historical record of all discovered job postings and companies for pipeline tracking.

---

5. 🏢 Apollo Organization Search

Property	Value
Type	HTTP Request (POST)
Purpose	Enriches company data via Apollo.io API
Endpoint	https://api.apollo.io/v1/organizations/search
Authentication	HTTP Header Auth (x-api-key)

Request Body:

{
  "q_organization_name": "Company Name",
  "page": 1,
  "per_page": 1
}

Response Fields:

Field	Description
id	Apollo organization ID
name	Official company name
website_url	Company website
linkedin_url	LinkedIn company page
industry	Business sector
estimated_num_employees	Company size
founded_year	Year established
city, state, country	Location details
short_description	Company overview

Function: Retrieves comprehensive company intelligence including LinkedIn profiles, industry classification, and employee count.

---

6. 📤 Extract Apollo Org Data

Property	Value
Type	Code Node (JavaScript)
Purpose	Parses Apollo response and merges with original data
Mode	Run once for each item

Function: Extracts relevant fields from Apollo API response and combines with job posting data for downstream processing.

---

7. 👥 Apollo People Search

Property	Value
Type	HTTP Request (POST)
Purpose	Finds decision-makers at target companies
Endpoint	https://api.apollo.io/v1/mixed_people/search
Authentication	HTTP Header Auth (x-api-key)

Request Body:

{
  "organization_ids": ["apollo_org_id"],
  "person_titles": [
    "CTO",
    "Chief Technology Officer",
    "VP Engineering",
    "Head of Engineering",
    "Engineering Manager",
    "Technical Director",
    "CEO",
    "Founder"
  ],
  "page": 1,
  "per_page": 3
}

Response Fields:

Field	Description
first_name	Contact first name
last_name	Contact last name
title	Job title
email	Email address
linkedin_url	LinkedIn profile URL
phone_number	Direct phone

Function: Identifies key stakeholders and decision-makers based on configurable title filters.

---

8. 📝 Format Leads

Property	Value
Type	Code Node (JavaScript)
Purpose	Structures lead data for outreach
Mode	Run once for all items

Function: Combines person data with company context, creating comprehensive lead profiles ready for personalization.

---

9. 🤖 Generate Personalized Message (OpenAI)

Property	Value
Type	OpenAI Node
Purpose	Creates custom LinkedIn connection messages
Model	gpt-4o-mini
Max Tokens	150
Temperature	0.7

System Prompt:

You are a professional outreach specialist. Write personalized LinkedIn connection request messages. Keep messages under 300 characters. Be friendly, professional, and mention a specific reason for connecting based on their role and company.

User Prompt Variables:

Variable	Source
Name	$json.fullName
Title	$json.title
Company	$json.companyName
Industry	$json.industry
Job Context	$json.jobTitle

Function: Generates unique, contextual outreach messages that reference specific hiring activity and company details.

---

10. 🔗 Merge Lead + Message

Property	Value
Type	Code Node (JavaScript)
Purpose	Combines lead data with generated message
Mode	Run once for each item

Function: Merges OpenAI response with lead profile, creating the final enriched record.

---

11. 💾 Save Leads to Sheet

Property	Value
Type	Google Sheets Node
Purpose	Stores final lead data with personalized messages
Operation	Append rows
Target Sheet	"Leads"

Data Mapping:

Column	Data
First Name	Lead's first name
Last Name	Lead's last name
Title	Job title
Company	Company name
LinkedIn URL	Profile link
Country	Location
Industry	Business sector
Date Added	Timestamp
Source	"Indeed + Apollo"
Personalized Message	AI-generated outreach text

Function: Creates actionable lead database ready for outreach campaigns.

---

Workflow Flow

⏰ Schedule Trigger
       │
       ▼
🔍 Scrape.do Indeed API ──► Fetches job listings with JS rendering
       │
       ▼
📋 Parse Indeed Jobs ──► Extracts company names, job details
       │
       ▼
📊 Add New Company ──► Saves to Google Sheets (Companies)
       │
       ▼
🏢 Apollo Org Search ──► Enriches company data
       │
       ▼
📤 Extract Apollo Org Data ──► Parses API response
       │
       ▼
👥 Apollo People Search ──► Finds decision-makers
       │
       ▼
📝 Format Leads ──► Structures lead profiles
       │
       ▼
🤖 Generate Personalized Message ──► AI creates custom outreach
       │
       ▼
🔗 Merge Lead + Message ──► Combines all data
       │
       ▼
💾 Save Leads to Sheet ──► Final storage (Leads)

---

Configuration Requirements

API Keys & Credentials

Credential	Purpose	Where to Get
Scrape.do API Token	Web scraping with anti-bot bypass	scrape.do/dashboard
Apollo.io API Key	B2B data enrichment	apollo.io/settings/integrations
OpenAI API Key	AI message generation	platform.openai.com
Google Sheets OAuth2	Data storage	n8n Credentials Setup

n8n Credential Setup

Credential Type	Configuration
HTTP Header Auth (Apollo)	Header: x-api-key, Value: Your Apollo API key
OpenAI API	API Key: Your OpenAI API key
Google Sheets OAuth2	Complete OAuth flow with Google

---

Key Features

🔍 Intelligent Job Scraping

• Anti-Bot Bypass: Residential proxy rotation via Scrape.do
• JavaScript Rendering: Full headless browser for dynamic content
• Mobile Optimization: Cleaner HTML with mobile viewport
• Markdown Output: Lightweight, easy-to-parse format

🏢 B2B Data Enrichment

• Company Intelligence: Industry, size, location, LinkedIn
• Decision-Maker Discovery: Title-based filtering
• Contact Information: Email, phone, LinkedIn profiles
• Real-Time Data: Fresh information from Apollo.io

🤖 AI-Powered Personalization

• Contextual Messages: References specific hiring activity
• Character Limit: Optimized for LinkedIn (300 chars)
• Variable Temperature: Balanced creativity and consistency
• Role-Specific: Tailored to recipient's title and company

📊 Automated Data Management

• Dual Sheet Storage: Companies + Leads separation
• Timestamp Tracking: Historical records
• Deduplication: Prevents duplicate entries
• Ready for Export: CSV-compatible format

---

Use Cases

🎯 Sales Prospecting

• Identify companies actively hiring in your target market
• Find decision-makers at companies investing in growth
• Generate personalized cold outreach at scale
• Track pipeline from discovery to contact

👥 Recruiting & Talent Acquisition

• Monitor competitor hiring patterns
• Identify companies building specific teams
• Connect with hiring managers directly
• Build talent pipeline relationships

📈 Market Intelligence

• Track industry hiring trends
• Monitor competitor expansion signals
• Identify emerging market opportunities
• Benchmark salary ranges by role

🤝 Partnership Development

• Find companies investing in complementary areas
• Identify potential integration partners
• Connect with technical leadership
• Build strategic relationship pipeline

---

Technical Notes

Specification	Value
Processing Time	2-5 minutes per run (depending on job count)
Jobs per Run	~25 unique companies
API Calls per Run	1 Scrape.do + ~25 Apollo Org + ~25 Apollo People + ~75 OpenAI
Data Accuracy	90%+ for company matching
Success Rate	99%+ with proper error handling

Rate Limits to Consider

Service	Free Tier Limit	Recommendation
Scrape.do	1,000 credits/month	~40 runs/month
Apollo.io	100 requests/day	Add Wait nodes if needed
OpenAI	Based on usage	Monitor costs (~$0.01-0.05/run)
Google Sheets	300 requests/minute	No issues expected

---

Setup Instructions

Step 1: Import Workflow

1. Copy the JSON workflow configuration
2. In n8n: Workflows → Import from JSON
3. Paste configuration and save

Step 2: Configure Scrape.do

1. Sign up at scrape.do
2. Navigate to Dashboard → API Token
3. Copy your token
4. Token is embedded in URL query parameter (already configured)

To customize search:

Change the `url` parameter in "Scrape.do Indeed API" node:
- q=data+engineer (search term)
- l=Remote (location)
- fromage=7 (last 7 days)

Step 3: Configure Apollo.io

1. Sign up at apollo.io
2. Go to Settings → Integrations → API Keys
3. Create new API key
4. In n8n: Credentials → Add Credential → Header Auth
   • Name: x-api-key
   • Value: Your Apollo API key
5. Select this credential in both Apollo HTTP nodes

Step 4: Configure OpenAI

1. Go to platform.openai.com
2. Create new API key
3. In n8n: Credentials → Add Credential → OpenAI
4. Paste API key
5. Select credential in "Generate Personalized Message" node

Step 5: Configure Google Sheets

1. Create new Google Spreadsheet
2. Create two sheets:
   • Sheet 1: "Add New Company"
     • Columns: companyName | jobTitle | jobUrl | location | salary | source | postedDate
   • Sheet 2: "Leads"
     • Columns: First Name | Last Name | Title | Company | LinkedIn URL | Country | Industry | Date Added | Source | Personalized Message
3. Copy Sheet ID from URL
4. In n8n: Credentials → Add Credential → Google Sheets OAuth2
5. Update both Google Sheets nodes with your Sheet ID

Step 6: Test and Activate

1. Manual Test: Click "Execute Workflow" button
2. Verify Each Node: Check outputs step by step
3. Review Data: Confirm data appears in Google Sheets
4. Activate: Toggle workflow to "Active"

---

Error Handling

Common Issues

Issue	Cause	Solution
"Invalid character: ["	Empty/malformed company name	Check Parse Indeed Jobs output
"Node does not have credentials"	Credential not linked	Open node → Select credential
Empty Parse Results	Indeed HTML structure changed	Check Scrape.do raw output
Apollo Rate Limit (429)	Too many requests	Add 5-10s Wait node between calls
OpenAI Timeout	Too many tokens	Reduce batch size or max_tokens
"Your request is invalid"	Malformed JSON body	Verify expression syntax in HTTP nodes

Troubleshooting Steps

1. Verify Credentials: Test each credential individually
2. Check Node Outputs: Use "Execute Node" for debugging
3. Monitor API Usage: Check Apollo and OpenAI dashboards
4. Review Logs: Check n8n execution history for details
5. Test with Sample: Use known company name to verify Apollo

Recommended Error Handling Additions

For production use, consider adding:

- IF node after Apollo Org Search to handle empty results
- Error Workflow trigger for notifications
- Wait nodes between API calls for rate limiting
- Retry logic for transient failures

---

Performance Specifications

Metric	Value
Execution Time	2-5 minutes per scheduled run
Jobs Discovered	~25 per Indeed page
Leads Generated	1-3 per company (based on title matches)
Message Quality	Professional, contextual, <300 chars
Data Freshness	Real-time from Indeed + Apollo
Storage Format	Google Sheets (unlimited rows)

---

API Reference

Scrape.do API

Endpoint	Method	Purpose
https://api.scrape.do	GET	Direct URL scraping

Documentation: scrape.do/documentation

Apollo.io API

Endpoint	Method	Purpose
/v1/organizations/search	POST	Company lookup
/v1/mixed_people/search	POST	People search

Documentation: apolloio.github.io/apollo-api-docs

OpenAI API

Endpoint	Method	Purpose
/v1/chat/completions	POST	Message generation

Documentation: [platform.openai.com