KG Pipeline Dashboard

Source

Configuration

Display Name

Root URL

Crawl Depth

Rate Limit

Max Concurrent Requests

Auto-Refresh Cron (optional)

Include Extensions (comma-separated, repo/dir only)

Exclude Dirs (comma-separated)

Include URL Patterns (one per line, web only)

Exclude URL Patterns (one per line)

Sitemap URLs (one per line)

Respect robots.txt JS Rendering Extract PDFs NPI Enrichment

Run Ingestion

Phases to run

All

Dry Run (validate only, no writes)

Run History

Read-only — write operations blocked