D4Vinci/Scrapling

This page introduces an external open-source repository. It is not an HDATF product.

What it does

Scrapling is a Python web scraping framework whose parser relocates elements when pages change. It offers fetchers that bypass anti-bot systems and a spider framework for concurrent crawls with pause, resume and proxy rotation.

How it helps ATF

Its adaptive selectors and resumable crawls could be a reference for how LabChin gathers searched sources from the web. Its anti-bot bypass features come with site terms and legal questions.

License

BSD-3-Clause Permissive. Commercial use is allowed; keep the notice and do not use the authors' names for endorsement.

More in this category

• pinchtab/pinchtab
  A standalone HTTP server in Go that gives AI agents control over Chrome, with headless browser instances, a dashboard, an HTTP API and an MCP server. It is designed first for local, single-user use.
• koala73/worldmonitor
  A real-time dashboard that gathers curated news feeds into AI-written briefs and tracks geopolitical, infrastructure and market signals, with 3D globe and flat map views. Its AI can run locally with Ollama.
• lightpanda-io/browser
  Lightpanda is a headless browser written from scratch in Zig for AI agents and automation. It is not a fork of Chromium or WebKit.
• pathwaycom/pathway
  A Python ETL framework for batch and stream processing, real-time analytics, LLM pipelines and RAG. Python code runs on a Rust engine that computes incrementally, and the same code serves local development, batch jobs and streaming.
• calesthio/Crucix
  A locally run intelligence dashboard that pulls 27 open-source intelligence feeds, such as flight tracking, radiation monitoring, conflict data and market prices, every 15 minutes. With an LLM connected, it sends alerts to Telegram and Discord and answers commands.

Only repositories in the ranked Trendshift lists are included, and the lists are used only to find candidates. We do not copy their ranks. Descriptions, licenses and star counts come from each GitHub repository. The notes are our own reading. We have not tested these projects, and a place on a trending list does not prove quality.