mirror of
https://gh-proxy.org/https://github.com/klinsmaya/LobsterAI.git
synced 2026-11-10 10:16:44 +08:00
The scoring heuristic for GB18030 vs UTF-8 could false-positive on longer CJK strings because UTF-8 CJK bytes reinterpreted as GB18030 also produce CJK characters with comparable scores, defeating the margin. Since genuine GB18030 CJK byte sequences are almost never valid UTF-8 (strict mode), a successful UTF-8 strict decode means the body IS UTF-8. Remove the scoring heuristic and always return utf8Decoded when strict decode passes. GB18030 fallback still applies when UTF-8 decode fails.