
The best Web Scraping framework that handles everything from a single request to a full-scale crawl by @D4Vinci1 Follow for daily tips & tricks.
Joined April 2025
- Tweets345
- Following1
- Followers1.5K
- Likes1K
Scrapling MCP extends the limits of current AI models 🥀
Some CAPTCHAs need more than a screenshot. They need motion and precise mouse control while observing the page.
Scrapling MCP v0.5.1 gives AI models both. 🔥
1. See motion
Scrapling records a 2-second video and sends it to the model as timestamped frames. The model can inspect how the page changes, then use chained browser actions to act.
2. Adjust a slider without letting go
The model can move a slider and ask Scrapling to keep the mouse button held down. It can then inspect the page, adjust the position, and release when ready.
That means it can check and adjust during a drag, before releasing the mouse button.
In the video, I demonstrate this on hCaptcha, DataDome, and GeeTest using Codex with Astra at light effort. 🚀
Scrapling v0.5.1 is available now for free. Link below 👇
Let me know what you think.
8
861
🕷️ Scrapling tip: pierce_shadow=True brings open Shadow DOM into the response so normal CSS queries can reach it. This demo goes from 0 paragraphs to 2, including slotted text. Closed roots stay out. No custom JavaScript tree walk 🎯
2
136
🕷️ Scrapling tip: crawldir saves a spider's pending requests and duplicate filter. Press Ctrl+C once, then rerun with the same folder to resume without starting over. Save scraped items separately; the checkpoint holds crawl state. 🎯
3
121
🕷️ Scrapling tip: the adaptive feature can match an old element against new HTML with relocate(). No saved database needed. This demo changes both prices, renames the CSS class and swaps product order. It still finds the desk's new price 🎯
4
126
🕷️ Scrapling tip: page_action can save rows before a scrolling list removes them from the page. Collect each visible batch by ID, then scroll. This demo keeps all 20 products even though the final HTML holds only 5. 🎯
2
145
🕷️ Scrapling tip: AsyncStealthySession can share two browser tabs across many JavaScript pages. Set max_pages=2 and use asyncio.gather(): extra fetches wait for a free tab, then reuse it. One browser, bounded tabs, parallel scraping. ⚡
5
223
🕷️ Scrapling tip: find_similar() missing rows with unique IDs? Add data-id to ignore_attributes so those values do not affect matching. Start from one known item, find the similar rows, then add the seed back. Demo: 0 matches becomes 2 🎯
2
154
🕷️ Scrapling tip: the adaptive feature stores matches by domain + identifier, not full URL. Saving .price on many product pages can overwrite earlier matches. Use a product ID in identifier to keep each saved match separate 🎯
2
201
🕷️ Scrapling tip: page_setup can rewrite API requests before the page sends them. Use page.route() to raise a supported page size from 10 to 100 and load more items per call. The API must allow the larger limit. ⚡
2
168
🕷️ Scrapling tip: Network history spans page loads. Use search(after_id=cursor) to read only new saved API replies, then advance the cursor with reply.meta["network_id"]. No more reading page 1's replies again on page 2. ⚡
1
5
217
🕷️ Scrapling tip: page_action lets you read nested frames that response.css() cannot see. Get the frame's HTML with frame.content(), then parse it with Selector. Reuse the loaded browser page without a second HTTP fetch. 🎯
9
443
🕷️ Scrapling tip: The adaptive feature has a match cutoff: percentage=80. If a price vanishes, the closest match may be "Sold out". This demo rejects that match. Tune the cutoff on real pages; it is a similarity score, not confidence. 🎯
4
193
🕷️ Scrapling tip: Two prices in one tag? Pass ignore_tags=("del", "s") to get_all_text() to skip the crossed-out price. It keeps the text right after </del>, even the lone "$" in this example. You get "$29.95" in one call. 🎯
4
229
🕷️ Scrapling tip: Paginating a JSON API with POST? Keep the spider's duplicate filter on. It checks the JSON body as well as the URL. Send page 1, page 2, then page 2 again to the same endpoint. Only two requests reach the server. 🎯
7
277
🕷️ Scrapling tip: You can use a regex inside XPath. Bind the EXSLT regex namespace and use re:test() to match a SKU on each row. One query gets Lamp and Desk and skips the Flash sale row. 🎯
3
260
🕷️ Scrapling tip: Two GraphQL calls hit /graphql. Which reply has the price? Turn on record_requests=True. Each saved reply carries its POST body in meta["request_body"]. Match the operationName and read the price without sending the query again 🎯
3
317
🕷️ Scrapling tip: RemoteSession(new_context=False) uses an open browser's login state. Connect to a browser with remote debugging enabled, then scrape without exporting cookies. When the session ends, your existing tabs stay open 🎯
3
283
Replying to @AISuperDomain
Thanks for writing about us!
Here’s a Chinese version of our docs: scrapling.readthedocs.io/zh-…
24
Replying to @rammcodes
Thanks for writing about us!
Here’s the full docs: scrapling.readthedocs.io/en/…
2
361
Replying to @clayul
Thanks for writing about us!
Here’s the full docs in Spanish:
scrapling.readthedocs.io/es/…
2
230
















