I accidentally turned LLM memory into program analysis

gist

Jordy Zomer wanted LLM agents to keep a live model of a vulnerability investigation, not a pile of retrieved snippets. He built Lemmalog, a Datalog engine that treats observations as facts, derives conclusions with rules, and retracts dependents when a fact is disproven. On LongMemEval and LoCoMo it is competitive with dedicated memory systems, strongest on knowledge updates and unsupported-premise rejection, while sending the reader a fraction of the transcript. Most gains came from extraction, identity, and retrieval, not a smarter evaluator; conversational benchmarks are not yet a real multi-hour exploit hunt.

ideas

• Memory is two problems. Retrieval answers which past information is relevant; maintained state answers what is currently true given everything learned so far.
• Split fuzzy extraction from deterministic derivation. The LLM turns source, debugger output, and notes into facts; Datalog derives, retracts, and explains the consequences.
• Provenance makes forgetting mechanical. Tracking derivation support lets dependent conclusions vanish when a fact is disproven, while independently supported conclusions remain.
• The front-end is the bottleneck. Entity identity, date encoding, retrieval aliases, and reader instructions moved the scores more than a smarter evaluator; missing extraction cannot be derived away.
• Competitive, not solved. Lemmalog leads the cited field on LongMemEval knowledge update, trails PropMem overall, uses far less query context, and still needs a long-running vulnerability investigation as the real test.

quotes