You know is very interested to see how much the bot landscape has changed over the last few years and the level of sophistication that it has evolved to. Back when I started all you had to do was clear your cookies and get rid of your flash cookies. Fast forward a few years later on and you needed to have residential/mobile proxies, change some user agents, and rate limit your requests to avoid getting blocked. Now you have to turn so many knobs just to get something working. - Thousands of proxy providers to pick from that sell you "high" quality IP bandwidth by the multiple dollars per GB. - Anti-detect browsers that promise you "privacy", "stealth" and "anti-detect" abilities to bypass any site's protection. - Open-source solutions that modify your puppeteer/playwright - Open-source solutions that modify your http2/tls fingerprints - Open-source solutions for collecting generic fingerprints. The list is truly endless: github.com/TheGP/untidetect-… The truth of the matter is that botting is fine and dandy when it works, no one questions it how it works until it breaks and people start bitching and complaining online. At the high-end of the botting/scraping world you have scraping companies such as Zyte, BrightData, Scrapingbee, Nimbleway, Browserless all offering api solutions to essentially bypass antibot solutions that impede scrapers from accessing a website's data. The dirty little secret is that none of the open-source and paid solutions work 100% of the time. They all do break and will break at some point. The difference is that with paid solutions you have someone else at the end of the line that will solve that problem for you. The paid solutions provider might outsource it to another vendor or use a supplementary service if they can't solve it themselves, because they do not want to lose a contract worth tens of thousands of dollars they get from deep-pocket start ups. So if you are new to the botting world, please do not rely on the developer creating open-source botting solutions on github for your botting needs. Then complain and bitch when shit breaks. These developers will eventually end up creating a paid-solution, abandon their open-source solution, and/or join a company tackling anti bot problems where they get paid a salary. This tale is as old as time, check out puppeteer-extra-stealth if you don't believe me. You can only succeed in the botting world if you have a lot of time or a lot of money because stopping bots requires 3 guided principles - Increasing the legislative powers to introduce more laws making botting and scraping illegal - Increasing complexity by introducing harder to reverse scripts/programs that prevents the botter from successfully mimicing a human. Because at the end of the day, all botters and scrapers just want to appear as human traffic not botters. - Last but not least, give all the authority to internet overlords such as Cloudflare and Google with their bullshit privacy pass schemes they cook up every couple of years. These 3 guided principles all have their weakness and strengths but I believe that in the next couple of years we are going to see different parties all argue why one is better than the other two. All I know for sure is that I will be deep in the trenches waiting to see what lies ahead.

Here is some shit I learned about reversing antibots(web not mobile) for some years. As someone that came from years of coding in Python to literally not coding in Python, you must learn Javascript there is no ifs or buts. You just have to. You are going to start reading a lot of Javascript and not descriptive Javascript with because antibots that are not callous to their security will make it hard for you to reverse. The very dumb antibots will still use the default open-source Javascript deobfuscators, the dumb ones will improve upon it, the smart ones will create their own obfuscation scheme, and the top ones will create their own VM. I know the cool thing to do is to use ChatGPT for everything. However, for the sake of whatever it is that you deeply love, please learn to parse using an AST library such as Babel. You can thank me later. Now these antibots companies are going to protect their clients sites by having your browser self-snitched all of your browser properties,keyboard and mouse movements, etc. and sends it in a payload. The data inside the payload is sometimes referred as fingerprints, signals, entroy componets etc. They usually send it in a POST request, sometimes they add it as HTTP headers, or a series of GET requests. Cookies will be phased out so I see less antibots delivering their payloads through that. The very dumb ones will send you a clear JSON payload with beautiful descriptive key names. The dumb ones will encode their payload using the default base64 alpha. The ones that think they are smart will have a series of loops that simply shuffle the bytes around, like reverse the bytes, swap the bytes next to each other, etc. The smart ones will have the decency of using an encryption function like a MurmurHash3, xtea encryption, AES, etc. The top ones will serve their payload encoded using their own packing mechanism, then encrypt it using their own encryption and then finally use a custom base 64 alpha on every request. Not to mention the encryption keys, seeds, ivs etc all change frequently. For hiding strings antibots will use a lot of xoring bytes in their scripts. The basic gist is that if they give you strA and you give them strB but you give them the final xoring result from strA ^ strB then they can get strB from the final xoring result. Some antibots put too much on this as they do not go beyond this for protecting their scripts. You will see a lot of booleans turned into numbers by shifting bytes(<<, >>, >>>). The strings will usually be encoded using UTF-8. Now if an antibot likes to deploy advanced fuckery they will encode their strings using UTF-8 encoding and apply a xoring byte to each byte. Most antibots usually end up having their fingerprints appeared very closely to each other, in the same order, meaning the fingerprints are not shuffled, and with a descriptive key describing what each data point is. The average ones will usually encode their keys using a base64 key with the default base64 alphabet. The smart ones will say nah we are going to send everything as bytes, and every signal will be shuffled in a different order, and their fields will also be shuffled. Some fingerprints are things that make you scratch your head until you run the same piece of Javascript code in two browsers and you get different results such as: JSON.stringify(Object.keys(console)) You will get different results in Chrome and Firefox. Don't believe me? Try it yourself! Those are the things you gotta mostly look for, the discrepancies between the browsers to catch the lazy botters when they lie about their environment. The really lazy antibots will still be trying to look for deprecated browser apis. I'm not sure if that's intentional or because they downsized their whole development department.