mirror of
https://gh-proxy.org/https://github.com/klinsmaya/deer-flow.git
synced 2026-11-08 17:26:58 +08:00
1f74082987
* feat(community): add Crawl4AI web_fetch provider Crawl4AI is a self-hosted, no-API-key web fetcher: it runs headless Chromium and returns server-cleaned "fit" markdown directly via its POST /md endpoint, so no client-side readability extraction is needed. It sits alongside the existing self-hosted Browserless provider. - deerflow.community.crawl4ai: async Crawl4AiClient + web_fetch_tool (reads base_url/timeout_s/token/filter from config; "Error:" string convention; 4096-char cap), mirroring the browserless provider - tests: 17 unit cases (success, HTTP error, success:false, empty, timeout, request error, token header, truncation, config reads) - config.example.yaml: commented web_fetch example - doctor: register as a no-key (free) web_fetch provider - setup wizard: add to WEB_FETCH_PROVIDERS (no API key) - docs: README, CONTRIBUTING, CONFIGURATION, AGENTS provider lists Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * fix(community): address Crawl4AI provider review feedback - timeout: robust _coerce_timeout (bool / non-numeric -> default) mirroring jina, so 'timeout: off' no longer becomes 0.0 and times out every request - read web_fetch config once per invocation and pass values into the client, so a concurrent hot-reload can't split base_url from filter - rename config key timeout_s -> timeout to match jina/infoquest (the default providers); update config.example.yaml + setup wizard - validate + normalize the markdown filter against {fit,raw,bm25,llm}; unknown values fall back to fit with a warning instead of an opaque HTTP 400 - client: a non-JSON 200 body (reverse proxy / auth wall) now reports the content-type + snippet instead of a generic JSONDecodeError - tests: 22 cases (added non-JSON-200, _coerce_timeout, _coerce_filter, invalid-filter fallback, read-config-once) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> Co-authored-by: DanielWalnut <[email protected]> Co-authored-by: Willem Jiang <[email protected]>