
The best Web Scraping framework that handles everything from a single request to a full-scale crawl by @D4Vinci1 Follow for daily tips & tricks.
Joined April 2025
- Tweets345
- Following1
- Followers1.5K
- Likes1K
🕷️ Scrapling tip: pierce_shadow=True brings open Shadow DOM into the response so normal CSS queries can reach it. This demo goes from 0 paragraphs to 2, including slotted text. Closed roots stay out. No custom JavaScript tree walk 🎯
2
136
🕷️ Scrapling tip: crawldir saves a spider's pending requests and duplicate filter. Press Ctrl+C once, then rerun with the same folder to resume without starting over. Save scraped items separately; the checkpoint holds crawl state. 🎯
3
121
🕷️ Scrapling tip: the adaptive feature can match an old element against new HTML with relocate(). No saved database needed. This demo changes both prices, renames the CSS class and swaps product order. It still finds the desk's new price 🎯
4
126
🕷️ Scrapling tip: page_action can save rows before a scrolling list removes them from the page. Collect each visible batch by ID, then scroll. This demo keeps all 20 products even though the final HTML holds only 5. 🎯
2
145
🕷️ Scrapling tip: AsyncStealthySession can share two browser tabs across many JavaScript pages. Set max_pages=2 and use asyncio.gather(): extra fetches wait for a free tab, then reuse it. One browser, bounded tabs, parallel scraping. ⚡
5
223
🕷️ Scrapling tip: find_similar() missing rows with unique IDs? Add data-id to ignore_attributes so those values do not affect matching. Start from one known item, find the similar rows, then add the seed back. Demo: 0 matches becomes 2 🎯
2
154
🕷️ Scrapling tip: the adaptive feature stores matches by domain + identifier, not full URL. Saving .price on many product pages can overwrite earlier matches. Use a product ID in identifier to keep each saved match separate 🎯
2
201
🕷️ Scrapling tip: page_setup can rewrite API requests before the page sends them. Use page.route() to raise a supported page size from 10 to 100 and load more items per call. The API must allow the larger limit. ⚡
2
168
🕷️ Scrapling tip: Network history spans page loads. Use search(after_id=cursor) to read only new saved API replies, then advance the cursor with reply.meta["network_id"]. No more reading page 1's replies again on page 2. ⚡
1
5
217
🕷️ Scrapling tip: page_action lets you read nested frames that response.css() cannot see. Get the frame's HTML with frame.content(), then parse it with Selector. Reuse the loaded browser page without a second HTTP fetch. 🎯
9
443
🕷️ Scrapling tip: The adaptive feature has a match cutoff: percentage=80. If a price vanishes, the closest match may be "Sold out". This demo rejects that match. Tune the cutoff on real pages; it is a similarity score, not confidence. 🎯
4
193
🕷️ Scrapling tip: Two prices in one tag? Pass ignore_tags=("del", "s") to get_all_text() to skip the crossed-out price. It keeps the text right after </del>, even the lone "$" in this example. You get "$29.95" in one call. 🎯
4
229
🕷️ Scrapling tip: Paginating a JSON API with POST? Keep the spider's duplicate filter on. It checks the JSON body as well as the URL. Send page 1, page 2, then page 2 again to the same endpoint. Only two requests reach the server. 🎯
7
277
🕷️ Scrapling tip: You can use a regex inside XPath. Bind the EXSLT regex namespace and use re:test() to match a SKU on each row. One query gets Lamp and Desk and skips the Flash sale row. 🎯
3
260














