Todd Hoff @[email protected]

You can Crawl entire website with Claude 3.5 or GPT4 with @firecrawl 💯 Its open-sourced and code in github - Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API. - Crawls all accessible subpages and give you clean data for each. No sitemap required. - The greatest benefit is that the extracted data is catered for LLM-based pipelines. - The api is self hostable and opensource ----- Some benefits of firecrawl 1. handles crawling (with or without sitemaps) 2. runs headless browsers scalably 3. handles bot protections and proxies 4. a team of dedicated engineers to solve the millions of edge-cases on the web for you 5. quality formatting to markdown by default Beautiful soup doesn't generalize, thats why we built firecrawl