mirror of
https://gh-proxy.org/https://github.com/klinsmaya/multica.git
synced 2026-11-08 01:16:45 +08:00
Danila Katalshov
ec6a7b1285
* fix(skills): skip binary supporting files in runtime-local skill sync collectLocalSkillFiles read every supporting file into SkillFileData.Content as a Go string. That string crosses the daemon/server hop as JSON, and encoding/json rewrites each invalid UTF-8 byte to U+FFFD, so writeSkillFiles recreated a file whose bytes differ from the original. A .docx, .xlsx or .pdf carried in a runtime-local skill bundle therefore arrived in the prepared task environment corrupt and would not open. The archive/URL importer already refuses this class of file for the mirror-image reason: those bytes cannot go into a PG TEXT column. Its extension blacklist moves to internal/skill as IsLikelyBinaryFilePath so both ingestion paths share one definition, and runtime-local collection now skips the same files with a log breadcrumb, matching the importer's behaviour. The skip does not depend on includeContent: discovery reports the bundle with content omitted and sync uploads it with content included, so a skip in only one pass would leave the two disagreeing about which files make up the bundle. Fixes #7143 * fix(skills): catch binary supporting files by content, not just extension collectLocalSkillFiles only skipped files whose extension matched a fixed blacklist, so a binary file with an unlisted or missing extension (.safetensors, .parquet, a stray no-extension blob) still went through SkillFileData.Content and came back corrupted on the other side. Read the file and require utf8.Valid on both the includeContent=false (discovery) and includeContent=true (sync) passes so they keep agreeing on the bundle; IsLikelyBinaryFilePath stays as the cheap extension-based first pass. Adds regression cases for an unlisted binary extension, a no-extension binary file, and a valid-UTF-8 text file that must still pass through untouched. * fix(skills): require NUL-free content, check extension before reading utf8.Valid alone is not a sufficient round-trip guarantee: a NUL byte is legal UTF-8, but the server-side import path strips every 0x00 via sanitizeNullBytes (server/internal/handler/skill_create.go), so a file that is valid UTF-8 but contains an embedded NUL still comes back different from what went in. UTF-16LE text made of ASCII characters is a realistic example. Require content to be both valid UTF-8 and NUL-free, with embedded_nul as its own skip reason distinct from invalid_utf8. Also reorders the extension check ahead of the read: IsLikelyBinaryFilePath is genuinely the cheap first pass now, so a known-binary extension never pays for the os.ReadFile it doesn't need. Adds regression cases for an unlisted extension and a no-extension file, each containing valid UTF-8 with an embedded NUL.
41 lines
1.4 KiB
Go
41 lines
1.4 KiB
Go
package skill
|
|
|
|
import (
|
|
"path/filepath"
|
|
"strings"
|
|
)
|
|
|
|
// IsLikelyBinaryFilePath reports whether the file's extension indicates a
|
|
// non-text payload. Conservative blacklist — extensions not on the list are
|
|
// assumed text and pass through.
|
|
//
|
|
// Both skill ingestion paths need this, for the same reason from two
|
|
// directions: an archive/URL import cannot put these bytes in a PG TEXT
|
|
// column (SQLSTATE 22021), and runtime-local sync cannot carry them through
|
|
// SkillFileData.Content, because encoding/json rewrites every invalid UTF-8
|
|
// byte to U+FFFD on the daemon/server hop. Either way the bytes that come
|
|
// back out are not the bytes that went in, so both paths skip the file
|
|
// instead of storing a payload they would later hand back corrupted.
|
|
func IsLikelyBinaryFilePath(path string) bool {
|
|
ext := strings.ToLower(filepath.Ext(path))
|
|
switch ext {
|
|
case
|
|
// images
|
|
".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp", ".tiff", ".ico", ".heic",
|
|
// fonts
|
|
".ttf", ".otf", ".woff", ".woff2", ".eot",
|
|
// archives
|
|
".zip", ".gz", ".tar", ".bz2", ".7z", ".rar",
|
|
// documents (binary office)
|
|
".pdf", ".docx", ".xlsx", ".pptx", ".doc", ".xls", ".ppt",
|
|
// media
|
|
".mp3", ".mp4", ".wav", ".avi", ".mov", ".webm", ".m4a", ".flac",
|
|
// compiled / executable
|
|
".exe", ".dll", ".so", ".dylib", ".class", ".jar", ".wasm",
|
|
// db / cache
|
|
".db", ".sqlite", ".sqlite3", ".pyc":
|
|
return true
|
|
}
|
|
return false
|
|
}
|