mirror of
https://gh-proxy.org/https://github.com/klinsmaya/multica.git
synced 2026-11-09 01:46:46 +08:00
* fix(daemon): give Hermes conversations a session that survives the task Hermes keeps every ACP session in `<HERMES_HOME>/state.db`. Because the daemon takes over HERMES_HOME to inject bound skills, that database lived inside the per-task overlay, so a conversation lasted exactly one task. The failure was silent, which is what made it expensive. `session/resume` against a database that does not hold the session does not fail: Hermes creates a fresh session and answers with an ordinary success frame, and ACP's ResumeSessionResponse has no `sessionId` field to contradict it. The daemon therefore believed it had resumed, stored the same dead id again, and every turn restarted from zero with nothing in the logs (GH #6806). Three changes, in the order they matter: - The backend now reads `_meta.hermes.sessionProvenance.acpSessionId`, the only place Hermes reports the session it actually served. A turn that landed on a different session continues under the new id and carries the existing ResumeContinuityNotice, so the agent can read the transcript back instead of pretending it remembers. - `state.db` is linked to a per-conversation store at `<profile dir>/hermes-sessions/<agent>/<hermes profile>/<conversation>/`. Sharding per conversation is what makes this safe where #6693 left it out of scope: tasks of one conversation are serial, so a shard has a single writer, while two issues never share a database. SQLite resolves the link when deriving the WAL paths, so the journal sidecars live in the store and a task teardown cannot take the write-ahead log with it. Windows without symlink privileges keeps the database task-local rather than copying it. - `gateResumeToReusedWorkdir` no longer treats a matching workdir as proof the session is reachable. The two keys came apart in the local_directory flow, where workdir reuse is disabled but envWorkDir still equals PriorWorkDir, so the gate forwarded an id that could never resolve. Stores are GC'd on their own retention (`MULTICA_GC_HERMES_SESSION_TTL`, default 14d, matching the Codex session store) and held against the GC for the duration of a task. Co-authored-by: multica-agent <[email protected]> * fix(daemon): never destroy a Hermes session database on a failed mount Review of #6845 found three ways the new session store could lose or misreport a conversation. All three are on failure and reclaim paths that the original tests did not reach. - **The degradation path was destructive.** The mount migrated the task-local state.db into the store and deleted it, and only then tried to create the symlink. On a host that cannot symlink, that turned a conversation which previously survived via workdir reuse into an empty one — a regression on exactly the hosts the fallback exists to protect. The link is now created under a staging name first, and nothing destructive runs until it exists; a host that cannot link is left byte for byte as it was found. - **The migration was not atomic.** It copied state.db and its journal sidecars straight into the store, so a failure after the main database landed left a store that looked migrated. The next attempt skipped the migration, deleted the source, and took the un-checkpointed tail of the conversation with it. It now stages the whole family and publishes with one rename, reusing the memory store's staging helpers (renamed to drop their `memory` prefix now that both stores share them). - **"Mounted" was read as "resumable".** A store the GC reclaimed between turns, a switched Hermes profile, an operator's `rm`, or a dangling link from an older overlay all mount cleanly onto nothing, and the resume gate forwarded a dead session id into them — the same silent restart this change set out to fix. The mount now reports two separate facts, and the gate reads whether the store actually holds a transcript, not whether the plumbing succeeded. A zero-length database counts as no history. Tests cover each path with injected failures rather than host-dependent skips: a link failure preserves the database and its WAL and still mounts once linking works again, a migration that dies between the database and its WAL publishes nothing and loses nothing on retry, and a mounted-but- empty store (fresh, reclaimed, dangling, zero-length) reports no history. Verified non-vacuous — reverting each fix reproduces the reviewed symptom. Co-authored-by: multica-agent <[email protected]> * fix(daemon): make the Hermes session store agree with itself on "empty" Review of the previous commit found a certain data-loss branch left between the migration and its publish step. The two used different definitions of an occupied store: - the migration asks hermesStoreHasSessionDB — a zero-length state.db, or journal sidecars with no database, carry no transcript, so it proceeds; - the shared promote asked hermesStorePopulated — any directory entry at all — so the same remnants read as "a competitor published first". With a non-empty source and a zero-length state.db in the store, the staging copy completed, os.Remove(storeDir) failed on the non-empty directory, promote reported a lost race, the migration returned success, and the caller deleted the source database and its un-checkpointed WAL. The link then pointed at a database with nothing in it. Occupancy is now the caller's predicate: the memory store keeps "any entry", the session store passes its own "holds a real database". The session publish clears the remnants it has already classified as nothing before the atomic directory rename, and a store it cannot clear fails closed — the migration returns the error and the source stays put — rather than passing for a lost race. The regression test drives the whole matrix through prepareHermesHome: a zero-length database, orphan sidecars, a real database that must never be overwritten, and an unclearable store that must fail closed with the source intact. It fails on the previous commit with the reviewed symptom (HistoryPresent=false after the source was deleted). Co-authored-by: multica-agent <[email protected]> * fix(daemon): never delete a competitor's freshly published Hermes transcript publishHermesSessionStaging decided the store held no history, then deleted the state.db family by path name. Between those two steps a competing task could publish its real transcript, and the stale decision would remove the winner's database — after the winner had already reported success and discarded its source. The check, the remnant cleanup and the promote now hold one process-wide mutex. In-process exclusion is the real thing here, not an approximation: the daemon is the store tree's only writer (even the GC guard is in-memory refcounts on the Daemon), so two publishers can only ever be two goroutines of this process. A deterministic barrier test parks a publisher inside the critical section and proves a competitor cannot enter the window. The family cleanup also stops using RemoveAll: regular files and symlinks are unlinked (never followed), while a directory or any other irregular type under a state.db name is content this code cannot identify, so it fails closed and the caller keeps its source. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Bohan-J <[email protected]> Co-authored-by: multica-agent <[email protected]> Co-authored-by: Steve Jobs (Multica Agent) <[email protected]>
615 lines
24 KiB
Go
615 lines
24 KiB
Go
package execenv
|
|
|
|
import (
|
|
"fmt"
|
|
"os"
|
|
"path/filepath"
|
|
"strings"
|
|
"sync"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
// TestHermesMemoryProfileSegment covers the store segment derived from a
|
|
// resolved source home: named profiles and the default home stay readable (an
|
|
// operator has to be able to find the directory), an out-of-tree custom home
|
|
// falls back to a hash of its path.
|
|
func TestHermesMemoryProfileSegment(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
if got := hermesMemoryProfileSegment(""); got != "default" {
|
|
t.Fatalf("empty source home segment = %q, want default", got)
|
|
}
|
|
if got := hermesMemoryProfileSegment(platformDefaultHermesHome()); got != "default" {
|
|
t.Fatalf("platform default segment = %q, want default", got)
|
|
}
|
|
// A profile of the native root keeps its bare, readable name.
|
|
native := filepath.Join(platformDefaultHermesHome(), "profiles", "research")
|
|
if got := hermesMemoryProfileSegment(native); got != "research" {
|
|
t.Fatalf("native named profile segment = %q, want research", got)
|
|
}
|
|
// Same profile name under a different root must not share a memory line.
|
|
root := t.TempDir()
|
|
foreign := hermesMemoryProfileSegment(filepath.Join(root, "profiles", "research"))
|
|
if foreign == "research" {
|
|
t.Fatalf("foreign-root profile collided with the native one on %q", foreign)
|
|
}
|
|
if !strings.HasPrefix(foreign, "research_") {
|
|
t.Fatalf("foreign-root profile segment = %q, want a research_<hash> form", foreign)
|
|
}
|
|
|
|
custom := hermesMemoryProfileSegment(filepath.Join(root, "custom-home"))
|
|
if !strings.HasPrefix(custom, "h_") {
|
|
t.Fatalf("custom home segment = %q, want an h_ hash", custom)
|
|
}
|
|
// Same basename in a different location must not collide.
|
|
other := hermesMemoryProfileSegment(filepath.Join(root, "nested", "custom-home"))
|
|
if custom == other {
|
|
t.Fatalf("distinct custom homes collided on %q", custom)
|
|
}
|
|
}
|
|
|
|
// TestHermesMemoryStorePathLayout pins the on-disk layout the documented
|
|
// one-off import depends on: <profile dir>/hermes-state/<agent>/<profile>.
|
|
func TestHermesMemoryStorePathLayout(t *testing.T) {
|
|
home := t.TempDir()
|
|
t.Setenv("HOME", home)
|
|
t.Setenv("USERPROFILE", home)
|
|
|
|
agent := "11111111-2222-3333-4444-555555555555"
|
|
got := HermesMemoryStorePath("", agent, filepath.Join(platformDefaultHermesHome(), "profiles", "research"))
|
|
want := filepath.Join(home, ".multica", hermesMemoryStoreRoot, agent, "research")
|
|
if got != want {
|
|
t.Fatalf("store path = %q, want %q", got, want)
|
|
}
|
|
}
|
|
|
|
// TestHermesMemoryStorePathDisabled covers the task without an agent to key the
|
|
// store on: memory has to stay task-local rather than land in a shared segment.
|
|
func TestHermesMemoryStorePathDisabled(t *testing.T) {
|
|
home := t.TempDir()
|
|
t.Setenv("HOME", home)
|
|
t.Setenv("USERPROFILE", home)
|
|
|
|
if got := HermesMemoryStorePath("", "", ""); got != "" {
|
|
t.Fatalf("store path without an agent = %q, want empty", got)
|
|
}
|
|
}
|
|
|
|
// TestPrepareHermesHomeMemoryStorePersistsAcrossTasks is the regression test for
|
|
// #6638: an agent with skills bound must keep its memory between tasks. Each
|
|
// task gets a fresh overlay home, so the memory can only survive if memories/
|
|
// resolves to the shared per-agent store.
|
|
func TestPrepareHermesHomeMemoryStorePersistsAcrossTasks(t *testing.T) {
|
|
t.Parallel()
|
|
sharedHome := t.TempDir()
|
|
store := filepath.Join(t.TempDir(), "hermes-state", "agent-1", "default")
|
|
skills := []SkillContextForEnv{{Name: "deploy", Content: "# Deploy"}}
|
|
|
|
firstTask := filepath.Join(t.TempDir(), "hermes-home")
|
|
if _, err := prepareHermesHome(firstTask, sharedHome, false, skills, nil, store, "", testLogger()); err != nil {
|
|
t.Fatalf("prepare first task: %v", err)
|
|
}
|
|
// Hermes writes memory back into <HERMES_HOME>/memories during the run.
|
|
mustWrite(t, filepath.Join(firstTask, "memories", "MEMORY.md"), "prefers tabs")
|
|
|
|
secondTask := filepath.Join(t.TempDir(), "hermes-home")
|
|
if _, err := prepareHermesHome(secondTask, sharedHome, false, skills, nil, store, "", testLogger()); err != nil {
|
|
t.Fatalf("prepare second task: %v", err)
|
|
}
|
|
got, err := os.ReadFile(filepath.Join(secondTask, "memories", "MEMORY.md"))
|
|
if err != nil {
|
|
t.Fatalf("second task lost the agent's memory: %v", err)
|
|
}
|
|
if string(got) != "prefers tabs" {
|
|
t.Fatalf("memory content = %q, want %q", got, "prefers tabs")
|
|
}
|
|
|
|
// The link must point at the store, not hold a copy — otherwise the next
|
|
// task's writes would diverge from it.
|
|
fi, err := os.Lstat(filepath.Join(secondTask, "memories"))
|
|
if err != nil {
|
|
t.Fatalf("lstat memories: %v", err)
|
|
}
|
|
if fi.Mode()&os.ModeSymlink == 0 {
|
|
t.Fatalf("memories is not a link (mode %v)", fi.Mode())
|
|
}
|
|
}
|
|
|
|
// TestPrepareHermesHomeMemoryStoreIsolatesAgents guards the isolation promise
|
|
// the store makes: one agent's memory must never be visible to another.
|
|
func TestPrepareHermesHomeMemoryStoreIsolatesAgents(t *testing.T) {
|
|
t.Parallel()
|
|
sharedHome := t.TempDir()
|
|
stateRoot := t.TempDir()
|
|
skills := []SkillContextForEnv{{Name: "deploy", Content: "# Deploy"}}
|
|
|
|
homeA := filepath.Join(t.TempDir(), "hermes-home")
|
|
if _, err := prepareHermesHome(homeA, sharedHome, false, skills, nil, filepath.Join(stateRoot, "agent-a", "default"), "", testLogger()); err != nil {
|
|
t.Fatalf("prepare agent A: %v", err)
|
|
}
|
|
mustWrite(t, filepath.Join(homeA, "memories", "MEMORY.md"), "agent A secret")
|
|
|
|
homeB := filepath.Join(t.TempDir(), "hermes-home")
|
|
if _, err := prepareHermesHome(homeB, sharedHome, false, skills, nil, filepath.Join(stateRoot, "agent-b", "default"), "", testLogger()); err != nil {
|
|
t.Fatalf("prepare agent B: %v", err)
|
|
}
|
|
if _, err := os.Stat(filepath.Join(homeB, "memories", "MEMORY.md")); !os.IsNotExist(err) {
|
|
t.Fatalf("agent B can see agent A's memory (err = %v)", err)
|
|
}
|
|
}
|
|
|
|
// TestPrepareHermesHomeWithoutStore verifies the no-store path: with nothing to
|
|
// key a store on, the overlay keeps a plain task-local memories dir and never
|
|
// leaves a dangling link behind.
|
|
func TestPrepareHermesHomeWithoutStore(t *testing.T) {
|
|
t.Parallel()
|
|
sharedHome := t.TempDir()
|
|
hermesHome := filepath.Join(t.TempDir(), "hermes-home")
|
|
skills := []SkillContextForEnv{{Name: "deploy", Content: "# Deploy"}}
|
|
|
|
if _, err := prepareHermesHome(hermesHome, sharedHome, false, skills, nil, "", "", testLogger()); err != nil {
|
|
t.Fatalf("prepare: %v", err)
|
|
}
|
|
fi, err := os.Lstat(filepath.Join(hermesHome, "memories"))
|
|
if err != nil {
|
|
t.Fatalf("lstat memories: %v", err)
|
|
}
|
|
if fi.Mode()&os.ModeSymlink != 0 {
|
|
t.Fatalf("memories should be a real dir without a store, got a link")
|
|
}
|
|
if !fi.IsDir() {
|
|
t.Fatalf("memories is not a directory (mode %v)", fi.Mode())
|
|
}
|
|
}
|
|
|
|
// TestPrepareHermesHomeWithoutStoreDetachesExistingStoreLink is the regression
|
|
// test for a reused workdir whose next task has no store (no agent to key on, or
|
|
// an unresolvable profile dir). The overlay still carries the link to the
|
|
// persistent store from the previous run, and MkdirAll would follow it and
|
|
// silently succeed — leaving the task writing to a store the daemon no longer
|
|
// marks active, and the GC free to reclaim it mid-task.
|
|
func TestPrepareHermesHomeWithoutStoreDetachesExistingStoreLink(t *testing.T) {
|
|
t.Parallel()
|
|
sharedHome := t.TempDir()
|
|
store := filepath.Join(t.TempDir(), "hermes-state", "agent-1", "default")
|
|
hermesHome := filepath.Join(t.TempDir(), "hermes-home")
|
|
skills := []SkillContextForEnv{{Name: "deploy", Content: "# Deploy"}}
|
|
|
|
// Run once with the store mounted.
|
|
if _, err := prepareHermesHome(hermesHome, sharedHome, false, skills, nil, store, "", testLogger()); err != nil {
|
|
t.Fatalf("prepare with store: %v", err)
|
|
}
|
|
mustWrite(t, filepath.Join(hermesHome, "memories", "MEMORY.md"), "persistent memory")
|
|
|
|
// The same overlay is reused by a task that resolves no store.
|
|
if _, err := prepareHermesHome(hermesHome, sharedHome, false, skills, nil, "", "", testLogger()); err != nil {
|
|
t.Fatalf("prepare without store: %v", err)
|
|
}
|
|
|
|
fi, err := os.Lstat(filepath.Join(hermesHome, "memories"))
|
|
if err != nil {
|
|
t.Fatalf("lstat memories: %v", err)
|
|
}
|
|
if fi.Mode()&os.ModeSymlink != 0 {
|
|
t.Fatalf("the store link is still in place; the task still writes to the persistent store")
|
|
}
|
|
if _, err := os.Stat(filepath.Join(hermesHome, "memories", "MEMORY.md")); !os.IsNotExist(err) {
|
|
t.Fatalf("a task without a store should get an empty memories dir (err = %v)", err)
|
|
}
|
|
// The store itself must survive, so the agent's next task with a store
|
|
// resolved still finds its memory.
|
|
if _, err := os.Stat(filepath.Join(store, "MEMORY.md")); err != nil {
|
|
t.Fatalf("detaching destroyed the persistent store: %v", err)
|
|
}
|
|
}
|
|
|
|
// TestPrepareHermesHomeWithoutStoreKeepsTaskLocalMemories checks the other reuse
|
|
// case: with no store, an existing real memories dir is this task's own memory
|
|
// and must be preserved across reuse.
|
|
func TestPrepareHermesHomeWithoutStoreKeepsTaskLocalMemories(t *testing.T) {
|
|
t.Parallel()
|
|
sharedHome := t.TempDir()
|
|
hermesHome := filepath.Join(t.TempDir(), "hermes-home")
|
|
skills := []SkillContextForEnv{{Name: "deploy", Content: "# Deploy"}}
|
|
|
|
if _, err := prepareHermesHome(hermesHome, sharedHome, false, skills, nil, "", "", testLogger()); err != nil {
|
|
t.Fatalf("first prepare: %v", err)
|
|
}
|
|
mustWrite(t, filepath.Join(hermesHome, "memories", "MEMORY.md"), "task memory")
|
|
|
|
if _, err := prepareHermesHome(hermesHome, sharedHome, false, skills, nil, "", "", testLogger()); err != nil {
|
|
t.Fatalf("reuse prepare: %v", err)
|
|
}
|
|
if _, err := os.Stat(filepath.Join(hermesHome, "memories", "MEMORY.md")); err != nil {
|
|
t.Fatalf("reuse dropped the task's own memory: %v", err)
|
|
}
|
|
}
|
|
|
|
// TestMigrateHermesTaskMemoriesFailureKeepsSource is the regression test for
|
|
// migration data loss: when an entry cannot be carried over, the source must
|
|
// survive and the overlay must fail, because the caller deletes the source
|
|
// directory as soon as migration reports success.
|
|
func TestMigrateHermesTaskMemoriesFailureKeepsSource(t *testing.T) {
|
|
t.Parallel()
|
|
taskDir := t.TempDir()
|
|
storeDir := t.TempDir()
|
|
|
|
mustWrite(t, filepath.Join(taskDir, "MEMORY.md"), "first")
|
|
mustWrite(t, filepath.Join(taskDir, "USER.md"), "second")
|
|
// Make one entry uncopyable the way a cross-filesystem move fails: a
|
|
// directory whose contents cannot be read.
|
|
blocked := filepath.Join(taskDir, "blocked")
|
|
if err := os.MkdirAll(blocked, 0o700); err != nil {
|
|
t.Fatalf("create blocked dir: %v", err)
|
|
}
|
|
mustWrite(t, filepath.Join(blocked, "note.md"), "unreadable")
|
|
if err := os.Chmod(blocked, 0o000); err != nil {
|
|
t.Fatalf("chmod blocked dir: %v", err)
|
|
}
|
|
t.Cleanup(func() { _ = os.Chmod(blocked, 0o700) })
|
|
|
|
err := migrateHermesTaskMemories(taskDir, storeDir, testLogger())
|
|
if err == nil {
|
|
t.Fatalf("migration reported success despite an unreadable entry; the caller would delete the source")
|
|
}
|
|
|
|
// Source intact.
|
|
for _, name := range []string{"MEMORY.md", "USER.md"} {
|
|
if _, statErr := os.Stat(filepath.Join(taskDir, name)); statErr != nil {
|
|
t.Fatalf("source %s was lost on a failed migration: %v", name, statErr)
|
|
}
|
|
}
|
|
// Store rolled back — a half-populated store would read as accumulated
|
|
// memory on the next run and block the retry.
|
|
left, readErr := os.ReadDir(storeDir)
|
|
if readErr != nil {
|
|
t.Fatalf("read store: %v", readErr)
|
|
}
|
|
if len(left) != 0 {
|
|
t.Fatalf("failed migration left %d entries in the store, want 0", len(left))
|
|
}
|
|
}
|
|
|
|
// TestMountHermesMemoriesUnreadableStoreKeepsSource covers the entry-point I/O
|
|
// error: an unreadable store must not read as "nothing to migrate", because the
|
|
// caller deletes the source directory the moment migration reports success.
|
|
func TestMountHermesMemoriesUnreadableStoreKeepsSource(t *testing.T) {
|
|
t.Parallel()
|
|
if os.Geteuid() == 0 {
|
|
t.Skip("root ignores directory permissions")
|
|
}
|
|
hermesHome := t.TempDir()
|
|
memories := filepath.Join(hermesHome, "memories")
|
|
mustWrite(t, filepath.Join(memories, "MEMORY.md"), "irreplaceable")
|
|
|
|
storeDir := filepath.Join(t.TempDir(), "store")
|
|
if err := os.MkdirAll(storeDir, 0o700); err != nil {
|
|
t.Fatalf("create store: %v", err)
|
|
}
|
|
if err := os.Chmod(storeDir, 0o000); err != nil {
|
|
t.Fatalf("chmod store: %v", err)
|
|
}
|
|
t.Cleanup(func() { _ = os.Chmod(storeDir, 0o700) })
|
|
|
|
if err := mountHermesMemories(hermesHome, storeDir, testLogger()); err == nil {
|
|
t.Fatalf("mount reported success against an unreadable store; the source would be deleted")
|
|
}
|
|
if _, err := os.Stat(filepath.Join(memories, "MEMORY.md")); err != nil {
|
|
t.Fatalf("source memory was lost after a failed mount: %v", err)
|
|
}
|
|
}
|
|
|
|
// TestMigrateHermesTaskMemoriesRefusesUnsupportedEntries checks that an entry
|
|
// the migration will not copy fails closed instead of being skipped — a skip
|
|
// plus the caller's source deletion is a silent loss.
|
|
func TestMigrateHermesTaskMemoriesRefusesUnsupportedEntries(t *testing.T) {
|
|
t.Parallel()
|
|
taskDir := t.TempDir()
|
|
storeDir := filepath.Join(t.TempDir(), "store")
|
|
|
|
mustWrite(t, filepath.Join(taskDir, "MEMORY.md"), "regular")
|
|
if err := os.Symlink(filepath.Join(taskDir, "MEMORY.md"), filepath.Join(taskDir, "LINK.md")); err != nil {
|
|
t.Skipf("symlinks unavailable: %v", err)
|
|
}
|
|
|
|
if err := migrateHermesTaskMemories(taskDir, storeDir, testLogger()); err == nil {
|
|
t.Fatalf("migration skipped a symlink and reported success")
|
|
}
|
|
if _, err := os.Stat(filepath.Join(taskDir, "MEMORY.md")); err != nil {
|
|
t.Fatalf("source was lost on a refused migration: %v", err)
|
|
}
|
|
if _, err := os.Stat(storeDir); !os.IsNotExist(err) {
|
|
t.Fatalf("refused migration left a store behind (err = %v)", err)
|
|
}
|
|
// A nested symlink must be refused too, not silently skipped.
|
|
nested := t.TempDir()
|
|
mustWrite(t, filepath.Join(nested, "notes", "note.md"), "regular")
|
|
if err := os.Symlink(filepath.Join(nested, "notes", "note.md"), filepath.Join(nested, "notes", "link.md")); err != nil {
|
|
t.Skipf("symlinks unavailable: %v", err)
|
|
}
|
|
if err := migrateHermesTaskMemories(nested, filepath.Join(t.TempDir(), "store"), testLogger()); err == nil {
|
|
t.Fatalf("migration skipped a nested symlink and reported success")
|
|
}
|
|
}
|
|
|
|
// TestMigrateHermesTaskMemoriesConcurrentFirstWriterWins covers two tasks of the
|
|
// same agent migrating at once, which is reachable in the window right after a
|
|
// daemon upgrade. Exactly one store must be published, and it must be one task's
|
|
// complete tree — never a mix of both.
|
|
func TestMigrateHermesTaskMemoriesConcurrentFirstWriterWins(t *testing.T) {
|
|
t.Parallel()
|
|
storeDir := filepath.Join(t.TempDir(), "agent", "default")
|
|
|
|
const tasks = 4
|
|
sources := make([]string, tasks)
|
|
for i := range sources {
|
|
sources[i] = t.TempDir()
|
|
// Same filenames, distinct contents: a store that interleaves the two
|
|
// would be detectable as a mixed marker set.
|
|
mustWrite(t, filepath.Join(sources[i], "MEMORY.md"), fmt.Sprintf("task-%d", i))
|
|
mustWrite(t, filepath.Join(sources[i], "USER.md"), fmt.Sprintf("task-%d", i))
|
|
}
|
|
|
|
var wg sync.WaitGroup
|
|
errs := make([]error, tasks)
|
|
start := make(chan struct{})
|
|
for i := range sources {
|
|
wg.Add(1)
|
|
go func(i int) {
|
|
defer wg.Done()
|
|
<-start
|
|
errs[i] = migrateHermesTaskMemories(sources[i], storeDir, testLogger())
|
|
}(i)
|
|
}
|
|
close(start)
|
|
wg.Wait()
|
|
|
|
for i, err := range errs {
|
|
if err != nil {
|
|
t.Fatalf("concurrent migration %d failed: %v", i, err)
|
|
}
|
|
}
|
|
|
|
memory, err := os.ReadFile(filepath.Join(storeDir, "MEMORY.md"))
|
|
if err != nil {
|
|
t.Fatalf("store is missing MEMORY.md after concurrent migration: %v", err)
|
|
}
|
|
user, err := os.ReadFile(filepath.Join(storeDir, "USER.md"))
|
|
if err != nil {
|
|
t.Fatalf("store is missing USER.md after concurrent migration: %v", err)
|
|
}
|
|
if string(memory) != string(user) {
|
|
t.Fatalf("store interleaved two tasks: MEMORY.md=%q USER.md=%q", memory, user)
|
|
}
|
|
|
|
// Every source survives: a task that lost the race must not have had its
|
|
// directory reported as migrated.
|
|
for i, src := range sources {
|
|
if _, err := os.Stat(filepath.Join(src, "MEMORY.md")); err != nil {
|
|
t.Fatalf("source %d was consumed by the migration: %v", i, err)
|
|
}
|
|
}
|
|
// No staging directories left behind next to the store.
|
|
siblings, err := os.ReadDir(filepath.Dir(storeDir))
|
|
if err != nil {
|
|
t.Fatalf("read store parent: %v", err)
|
|
}
|
|
if len(siblings) != 1 {
|
|
t.Fatalf("expected only the store next to itself, got %d entries", len(siblings))
|
|
}
|
|
}
|
|
|
|
// TestPromoteHermesMemoryStagingClassifiesRemoveFailures covers the publish
|
|
// latch. Losing the race must be positively confirmed, never inferred from any
|
|
// os.Remove failure: the caller deletes the task's source directory on
|
|
// (false, nil), so a permission or I/O error there is a data-loss path.
|
|
func TestPromoteHermesMemoryStagingClassifiesRemoveFailures(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
// A genuinely non-empty store means another task published first.
|
|
t.Run("populated store loses the race", func(t *testing.T) {
|
|
t.Parallel()
|
|
parent := t.TempDir()
|
|
storeDir := filepath.Join(parent, "default")
|
|
mustWrite(t, filepath.Join(storeDir, "MEMORY.md"), "winner")
|
|
staging := filepath.Join(parent, ".default.migrating-x")
|
|
mustWrite(t, filepath.Join(staging, "MEMORY.md"), "loser")
|
|
|
|
promoted, err := promoteHermesStoreStaging(staging, storeDir, hermesStorePopulated)
|
|
if err != nil {
|
|
t.Fatalf("a populated store should be a lost race, not an error: %v", err)
|
|
}
|
|
if promoted {
|
|
t.Fatalf("promote clobbered a store another task had published")
|
|
}
|
|
got, readErr := os.ReadFile(filepath.Join(storeDir, "MEMORY.md"))
|
|
if readErr != nil || string(got) != "winner" {
|
|
t.Fatalf("winner's store was modified: content=%q err=%v", got, readErr)
|
|
}
|
|
})
|
|
|
|
// An empty store that cannot be removed is an I/O failure, not a race.
|
|
t.Run("unremovable empty store fails closed", func(t *testing.T) {
|
|
if os.Geteuid() == 0 {
|
|
t.Skip("root ignores directory permissions")
|
|
}
|
|
parent := t.TempDir()
|
|
storeDir := filepath.Join(parent, "default")
|
|
if err := os.MkdirAll(storeDir, 0o700); err != nil {
|
|
t.Fatalf("create store: %v", err)
|
|
}
|
|
staging := filepath.Join(parent, ".default.migrating-x")
|
|
mustWrite(t, filepath.Join(staging, "MEMORY.md"), "irreplaceable")
|
|
|
|
// Removing a directory needs write permission on its parent.
|
|
if err := os.Chmod(parent, 0o500); err != nil {
|
|
t.Fatalf("chmod parent: %v", err)
|
|
}
|
|
t.Cleanup(func() { _ = os.Chmod(parent, 0o700) })
|
|
|
|
promoted, err := promoteHermesStoreStaging(staging, storeDir, hermesStorePopulated)
|
|
if promoted {
|
|
t.Fatalf("promote reported success against an unremovable store")
|
|
}
|
|
if err == nil {
|
|
t.Fatalf("a permission error was reported as a lost race; the caller would delete the source")
|
|
}
|
|
})
|
|
}
|
|
|
|
// TestMigrateHermesTaskMemoriesCopiesRatherThanMoves pins the cross-filesystem
|
|
// property: migration must not depend on rename, and must carry nested dirs.
|
|
func TestMigrateHermesTaskMemoriesCopiesRatherThanMoves(t *testing.T) {
|
|
t.Parallel()
|
|
taskDir := t.TempDir()
|
|
storeDir := t.TempDir()
|
|
|
|
mustWrite(t, filepath.Join(taskDir, "MEMORY.md"), "top level")
|
|
mustWrite(t, filepath.Join(taskDir, "notes", "deep", "note.md"), "nested")
|
|
|
|
if err := migrateHermesTaskMemories(taskDir, storeDir, testLogger()); err != nil {
|
|
t.Fatalf("migrate: %v", err)
|
|
}
|
|
for _, rel := range []string{"MEMORY.md", filepath.Join("notes", "deep", "note.md")} {
|
|
if _, err := os.Stat(filepath.Join(storeDir, rel)); err != nil {
|
|
t.Fatalf("store is missing %s: %v", rel, err)
|
|
}
|
|
// Copy, not move: the source stays until the caller removes the dir.
|
|
if _, err := os.Stat(filepath.Join(taskDir, rel)); err != nil {
|
|
t.Fatalf("migration moved %s instead of copying it: %v", rel, err)
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestPrepareHermesHomeMigratesTaskLocalMemories covers the upgrade path: a
|
|
// workdir reused from a pre-store daemon still holds a real memories/ dir, whose
|
|
// contents must move into the (empty) store rather than be dropped.
|
|
func TestPrepareHermesHomeMigratesTaskLocalMemories(t *testing.T) {
|
|
t.Parallel()
|
|
sharedHome := t.TempDir()
|
|
hermesHome := filepath.Join(t.TempDir(), "hermes-home")
|
|
skills := []SkillContextForEnv{{Name: "deploy", Content: "# Deploy"}}
|
|
|
|
// Simulate the old layout: a task-local memories dir with accumulated state.
|
|
if _, err := prepareHermesHome(hermesHome, sharedHome, false, skills, nil, "", "", testLogger()); err != nil {
|
|
t.Fatalf("prepare pre-store overlay: %v", err)
|
|
}
|
|
mustWrite(t, filepath.Join(hermesHome, "memories", "MEMORY.md"), "carried over")
|
|
|
|
store := filepath.Join(t.TempDir(), "hermes-state", "agent-1", "default")
|
|
if _, err := prepareHermesHome(hermesHome, sharedHome, false, skills, nil, store, "", testLogger()); err != nil {
|
|
t.Fatalf("prepare with store: %v", err)
|
|
}
|
|
got, err := os.ReadFile(filepath.Join(store, "MEMORY.md"))
|
|
if err != nil {
|
|
t.Fatalf("task-local memory was not migrated into the store: %v", err)
|
|
}
|
|
if string(got) != "carried over" {
|
|
t.Fatalf("migrated content = %q, want %q", got, "carried over")
|
|
}
|
|
}
|
|
|
|
// TestPrepareHermesHomeMigrationKeepsExistingStore is the other half of the
|
|
// upgrade path: an agent that already accumulated memory must never have it
|
|
// overwritten by a stale task-local dir.
|
|
func TestPrepareHermesHomeMigrationKeepsExistingStore(t *testing.T) {
|
|
t.Parallel()
|
|
sharedHome := t.TempDir()
|
|
hermesHome := filepath.Join(t.TempDir(), "hermes-home")
|
|
skills := []SkillContextForEnv{{Name: "deploy", Content: "# Deploy"}}
|
|
|
|
if _, err := prepareHermesHome(hermesHome, sharedHome, false, skills, nil, "", "", testLogger()); err != nil {
|
|
t.Fatalf("prepare pre-store overlay: %v", err)
|
|
}
|
|
mustWrite(t, filepath.Join(hermesHome, "memories", "MEMORY.md"), "stale task copy")
|
|
|
|
store := filepath.Join(t.TempDir(), "hermes-state", "agent-1", "default")
|
|
mustWrite(t, filepath.Join(store, "MEMORY.md"), "authoritative agent memory")
|
|
|
|
if _, err := prepareHermesHome(hermesHome, sharedHome, false, skills, nil, store, "", testLogger()); err != nil {
|
|
t.Fatalf("prepare with store: %v", err)
|
|
}
|
|
got, err := os.ReadFile(filepath.Join(store, "MEMORY.md"))
|
|
if err != nil {
|
|
t.Fatalf("read store memory: %v", err)
|
|
}
|
|
if string(got) != "authoritative agent memory" {
|
|
t.Fatalf("store memory was overwritten by the task-local copy: %q", got)
|
|
}
|
|
}
|
|
|
|
// TestPruneHermesMemoryStores covers the GC contract: stores idle past retention
|
|
// are reclaimed, recently-used ones are kept, and a store a live task holds is
|
|
// never removed.
|
|
func TestPruneHermesMemoryStores(t *testing.T) {
|
|
home := t.TempDir()
|
|
t.Setenv("HOME", home)
|
|
t.Setenv("USERPROFILE", home)
|
|
|
|
root := filepath.Join(home, ".multica", hermesMemoryStoreRoot)
|
|
idle := filepath.Join(root, "agent-idle", "default")
|
|
fresh := filepath.Join(root, "agent-fresh", "default")
|
|
held := filepath.Join(root, "agent-held", "default")
|
|
for _, dir := range []string{idle, fresh, held} {
|
|
mustWrite(t, filepath.Join(dir, "MEMORY.md"), "remembered")
|
|
}
|
|
|
|
now := time.Now()
|
|
old := now.Add(-30 * 24 * time.Hour)
|
|
for _, dir := range []string{idle, held} {
|
|
if err := os.Chtimes(filepath.Join(dir, "MEMORY.md"), old, old); err != nil {
|
|
t.Fatalf("age store: %v", err)
|
|
}
|
|
if err := os.Chtimes(dir, old, old); err != nil {
|
|
t.Fatalf("age store dir: %v", err)
|
|
}
|
|
}
|
|
|
|
reserve := func(storeDir string) (func(), bool) {
|
|
if storeDir == held {
|
|
return nil, false // a live task holds it
|
|
}
|
|
return func() {}, true
|
|
}
|
|
|
|
removed, freed := PruneHermesMemoryStores("", 14*24*time.Hour, now, reserve, testLogger())
|
|
if removed != 1 {
|
|
t.Fatalf("removed = %d, want 1", removed)
|
|
}
|
|
if freed <= 0 {
|
|
t.Fatalf("bytesFreed = %d, want > 0", freed)
|
|
}
|
|
if _, err := os.Stat(idle); !os.IsNotExist(err) {
|
|
t.Fatalf("idle store survived the prune (err = %v)", err)
|
|
}
|
|
if _, err := os.Stat(fresh); err != nil {
|
|
t.Fatalf("recently used store was reclaimed: %v", err)
|
|
}
|
|
if _, err := os.Stat(held); err != nil {
|
|
t.Fatalf("store held by a live task was reclaimed: %v", err)
|
|
}
|
|
}
|
|
|
|
// TestPruneHermesMemoryStoresDisabled documents that retention <= 0 turns the
|
|
// pruner off entirely, matching the Codex store knob.
|
|
func TestPruneHermesMemoryStoresDisabled(t *testing.T) {
|
|
home := t.TempDir()
|
|
t.Setenv("HOME", home)
|
|
t.Setenv("USERPROFILE", home)
|
|
|
|
store := filepath.Join(home, ".multica", hermesMemoryStoreRoot, "agent-1", "default")
|
|
mustWrite(t, filepath.Join(store, "MEMORY.md"), "remembered")
|
|
old := time.Now().Add(-365 * 24 * time.Hour)
|
|
if err := os.Chtimes(store, old, old); err != nil {
|
|
t.Fatalf("age store: %v", err)
|
|
}
|
|
|
|
if removed, _ := PruneHermesMemoryStores("", 0, time.Now(), nil, testLogger()); removed != 0 {
|
|
t.Fatalf("removed = %d with pruning disabled, want 0", removed)
|
|
}
|
|
if _, err := os.Stat(store); err != nil {
|
|
t.Fatalf("store was reclaimed with pruning disabled: %v", err)
|
|
}
|
|
}
|