Do not use puppeteer and playwright for scraping. Or any other browser automation framework for that matter. Why?

We only use the CDP to obtain the page source and to get the absolute coordinates for an arbitrary CSS selector. That's all what is needed for efficient scraping.

1. To obtain the page source of the browser's current page. Implemented in page_source.js
2. To get the absolute coordinates for an arbitrary CSS selector. Implemented in coords.js

Mouse movements and typing is handled by pyautogui or other means, but not with JavaScript or with the CDP! Reason: Browser based mouse and keyboard emulation is very easy detectable!

1. Analyzing key strokes: TypeNet: Deep Learning Keystroke Biometrics
2. Research how to mimic human mouse movements: BeCAPTCHA-Mouse: Synthetic Mouse Trajectories and Improved Bot Detection

The bot challenge that can be found here bot.incolumitas.com/#botChallenge will be solved in the following quick tutorial.

The example code can be found in example.py.

I am using an Ubuntu 18.04 system with Python3 (with pipenv) and a recent Node version.

The browser google-chrome must be installed.

Clone the repo:

git clone https://github.com/NikolaiT/stealthy-scraping-tools
cd stealthy-scraping-tools

Activate an environment with:

pew new -p python3 sst

pew workon sst

Then install pyautogui:

pip install pyautogui

Install node modules:

npm install chrome-remote-interface

And then run the bot with:

python example.py

The Dockerfile is based on Ubuntu 20.04.

The Dockerfile uses xvfb from the python module PyVirtualDisplay.

I use pyautogui for mouse and keyboard automation.

I use fluxbox as a tiny window manager and x11vnc (Virtual Network Computing server program) as a means to inspect the docker image and see what is going on.

Build the Dockerfile:

docker build -t sst:0.0.1 .

Hint: Avoid chrome in docker crashing: stephen-fox/chrome-docker#8

1. Option 1: Run chrome with --disable-dev-shm-usage
2. Option 2: Set /dev/shm size to a reasonable amount `docker run -it --shm-size=1g` replacing 1g with whatever amount you want.

The docker option --shm-size=2g is really important:

docker run --cap-add=NET_ADMIN --network="host" --shm-size=2g sst:0.0.1

• Look at Kernel/OS level mouse/keyboard control commands (Ditch pyautogui)
• Use the math from ghost-cursor
• Create a set of typign recordings and use it to derive rules for bot writing