Async Lead Generation Agent with Playwright Web Scraping and Supabase

Overview

AI-powered lead generation agent with a client-facing dashboard. User inputs business type, city, and number of leads, the system scrapes Google Maps via Apify, extracts emails from each website using a self-hosted Playwright API (FastAPI + Docker on VPS), and delivers a structured spreadsheet by email. Built across 3 n8n workflows: async polling prevents webhook timeouts on long scraping jobs, Supabase lock mechanism prevents duplicate runs, and phone numbers are sanitized before saving. Results delivered via branded HTML email with XLSX and CSV attachments. Extraction rate improved to 25%.

Technical Highlights

• 1Self-hosted Playwright on VPS removes cloud timeout constraints (30s kill)
• 2Email extraction rate: ~3% (HTTP fetch) → ~25% (full JS rendering)
• 3Async Apify polling: fire-and-poll, never blocks the workflow
• 4Supabase lock state prevents duplicate concurrent scraping jobs
• 5AI Agent enriches leads with LinkedIn profile search on top of email scraping