Something sounds very wrong with your setup or how you use it.

Is your CLAUDE.md barren?

Try moving memory files into the project:

    (In your project's .claude/settings.local.json)

    { ...
      "plansDirectory": "./plans/wip",
      "autoMemoryDirectory": "/Users/foo/project/.claude/memory"
    }

(Memory path has to be absolute)

I did this because memory (and plans) should show up in git status so that they are more visible, but then I noticed the agent started reading/setting them more.