Loading search…

Explain with AI
Open in ChatGPT(opens in a new tab)Open in Claude(opens in a new tab)Open in Perplexity(opens in a new tab)Open in Duck.ai(opens in a new tab)

Opens your own AI account with the prompt ready. comic-dl sends nothing itself.

Edit this page

Architecture

Last updated

Follow a URL from scraper selection to a verified archive — pipeline, module map, concurrency, and security.

On this page

A guided tour of comic-dl’s internals: what each module does, how a URL becomes an archive, and why the concurrency and security rules exist. Read this when you first open the source; the API reference indexes the contracts.

The tour: follow one URL end to end

  1. Entry — cli/__init__.py parses arguments, normalizes and routes the URL, and orchestrates the rest.
  2. Routing — scrapers/registry.py resolves the domain to a scraper. If no domain scraper matches and the generic fallback is enabled, generic.py picks it up.
  3. Metadata — the scraper returns PostMetadata (chapter) or SeriesMetadata (series) through BaseScraper.scrape() / scrape_series(). Everything is fetched through _timeout_get / _open_stream in base.py — the SSRF guard lives there.
  4. Safety — every outbound URL passes validate_request_url; redirect hops are re-validated on both scrape and download paths.
  5. Download — downloader.py streams to .part files with exponential backoff, per-host throttling (rate.py), a shared retry cooldown, and size caps (cli/sizing.py).
  6. Verify — magic-byte check per file; corrupt or over-size files are discarded. Duplicate pages (SHA-256) are dropped later.
  7. Archive — archiver.py writes to .tmp, verifies, then atomically renames. comicinfo.py embeds ComicInfo.xml.
  8. Library — library.py records series/chapters in SQLite, best-effort; the DB never blocks downloads.

Source map

src/comic_dl/
  __main__.py            console entry point
  cli/__init__.py        argument parsing, URL routing, orchestration
  cli/library.py         list/info/latest/remove subcommands
  cli/selection.py       interactive chapter-selection prompt
  cli/sizing.py          size caps, download-size estimates, disk checks
  config.py              config file + platform directory resolution
  platform.py            thin OS seam (system(), downloads_dir(), default_editor())
  utils.py               URL normalization, sanitization, SSRF guard,
                         image magic-byte verification
  models.py              data contracts (ImageItem, PostMetadata, ChapterInfo,
                         SourceInfo, ScrapedChapter, SeriesMetadata)
  errors.py              error types and exit codes (0/1/2/130)
  downloader.py          streaming/retry download engine + DownloadPipeline
  archiver.py            archive creation (.cbz/.zip/.cbt; atomic writes)
  comicinfo.py           ComicInfo.xml generator
  library.py             local SQLite cache of series/chapters
  ui.py                  rich console output (banner, progress, tables)
  http.py                shared curl_cffi session/request helpers
  cookies.py             persistent cookie jar (SQLite, RFC 6265)
  cache.py               scrape-response cache (TTL, ETag revalidation)
  rate.py                per-site request throttling (token bucket)
  cf.py, antibot.py      Cloudflare detection + solver routing
  webview*.py            system-webview / headless challenge solver
  scrapers/
    registry.py          entry-point discovery, URL → scraper routing
    base.py              shared helpers (meta extraction, validated fetch)
    generic.py           fallback HTML scraper
    madara.py            shared Madara-theme framework scraper
    refresh.py           stale-image refresh registration
    sites/               one module per built-in site

The left column is the core pipeline; the right column is everything hostile (HTTP, cookies, rate limits, Cloudflare). scrapers/ sits in the middle, turning page HTML into the model contracts the pipeline consumes.

Modules that need care

  • scrapers/base.py — every fetch path threads here. Never add a helper that bypasses validate_request_url.
  • rate.py and the shared cooldown in downloader.py — politeness is load-bearing. New code rides them, never bypasses them.
  • archiver.py — atomic writes (.tmp + rename); in-place writes corrupt archives on interrupt.
  • errors.py + ui.py — user-facing text flows only through ui.py helpers; error types map to exit codes; message text never leaks raw exception args.
  • webview*.py — subprocess solver; the parent/child protocol lives in webview_constants.py.

Concurrency model

Three independent bounds, each clamped in cli/__init__.py:

  • --concurrency — parallel image downloads within a chapter (max 32).
  • --chapter-parallel — concurrent chapters of a series (max 8, default 1).
  • --parallel — URLs in flight across a batch (max 16, default 5).

Above the caps the value clamps, not rejects; below 1 is a hard error. A shared cooldown pauses in-flight downloads on a retryable error, so N retries don’t become a thundering herd.

Security posture

The threat model is “the user hands us URLs from untrusted sites”:

  • SSRF guard — non-http(s) schemes and hosts resolving to loopback/private/link-local/metadata addresses are rejected. Redirect hops are re-validated on both scrape and download paths (capped at MAX_REDIRECTS = 5). See Security Testing for the known DNS-rebinding (TOCTOU) limitation.
  • Untrusted XML — ComicInfo.xml parses with defusedxml.
  • SQL injection — all Library queries use bound parameters.
  • Path traversal — filenames sanitized and path-contained before writes.
  • Concurrency bounds — values above the cap clamp; below 1 errors.

Touch any of these and the corresponding security test must change in the same PR.

Interrupt handling

SIGINT/SIGTERM use two-stage cooperative shutdown: a STOP_REQUESTED flag is set and checked at item boundaries; a second press within 2 seconds force-exits via os._exit(130). The handler only sets the flag or force-exits — never async teardown.

Platform seam

Per-OS behavior lives mostly in platformdirs (via config.py) and stdlib. platform.py centralizes the handful of conventions used at multiple call sites: system() (the canonical platform name), downloads_dir() (reads the Windows Shell Folder value so OneDrive-redirected homes work), and default_editor(). Keep it small — cross-platform stdlib behavior stays where it is.

Versioning and packaging

pyproject.toml is the single hand-edited version source. Builds regenerate src/comic_dl/_version.py (hatch version hook for wheel/sdist, scripts/write-version.py for PyInstaller). The file is committed so fresh checkouts work pre-build; the drift guard (tests/test_version.py, write-version.py --check) fails the gate on an unregenerated bump.

Extension points

Sources are pluggable through the comic_dl.sources entry-point group; installed plugins appear in --list-sources and can override built-ins with priority > 0. To add a built-in source, create scrapers/sites/<site>.py implementing BaseScraper, decorate with @register_scraper(domain=..., capabilities=...), and run uv run python scripts/update-sites-docs.py to refresh the supported-sites tables. See writing a plugin.