Federated AI Agent Memory: Moving Distilled Knowledge Between Teacher and Student Machines Across Engineering Teams

Over the previous installments of this series—from Giving Local AI Assistants Long-Term Memory and Learning While You Sleep: Out-of-Band Memory Dreaming to Hardening Claude Code’s Memory: Mistake Ledgers, Per-Prompt Context, and Verifying the Dreamer Itself and Hardening Local AI Agents: Procedural Skills, Mistake Ledgers, and Real-Time Verification—we solved single-machine amnesia. An AI coding assistant running locally can now archive sessions, curate Markdown knowledge graphs, synthesize reusable SKILL.md runbooks, and maintain a negative constraint ledger (Meta/lessons.md) while you sleep.

Yet once a workstation’s agent accumulates months of battle-tested debugging invariants, a glaring bottleneck remains: all of that hard-won intelligence is trapped on one machine.

Consider what happens in two very common scenarios:

1. Multi-Machine Individual Workflow: You solve a subtle Kubernetes image-pull bug, a Kafka partition fan-out trap, or an ORM migration edge case on your primary development workstation. Later, you switch to a secondary workstation running a different workspace—or even a completely different AI coding harness—and find yourself explaining the same architectural invariant all over again.
2. Team-Wide Knowledge Federation: A principal engineer or tech lead spends hours coaching their coding agent through tricky distributed-systems invariants and repository conventions. Today, every other engineer on the team starts with an uninitiated assistant that repeats the exact same mistakes. What if the senior engineer’s machine could act as a Teacher, broadcasting sanitized, generalized lessons to every team member’s Student workstation automatically?

Why Naive File Syncing Fails

The tempting shortcut is to sync .claude/, .agents/, or the Obsidian Meta/ directory directly using rsync, Syncthing, or a shared Git repository. In practice, naive file syncing breaks down immediately across machines and teams:

• Secret and PII Leakage: Raw session transcripts, operational notes, and local configs contain API tokens, internal URLs, private RFC1918 IP ranges, proprietary company names, and local usernames. Moving raw memory across trust boundaries is a security incident waiting to happen.
• Hardcoded Workstation Assumptions: A memory file that references /home/alice/projects/payments-core or a machine-specific mount path actively misleads a Student agent running under /Users/bob/src.
• Heterogeneous Agent Harnesses: One machine may run Claude Code, another Google Antigravity (agy), and a teammate’s machine Cursor or Aider. Their directory layouts, frontmatter expectations, and skill paths differ.
• Destructive Overwrites: Copying whole Markdown files from machine A to machine B wipes out machine B’s local discoveries unless both sides merge with optimistic concurrency control and semantic deduplication.

To solve this cleanly, we need a privacy-preserving Teacher–Student knowledge distillation pipeline—a clean separation between local raw memory and a portable, redacted curriculum that can traverse any network boundary and be assimilated into any agent harness.

1. Architectural Overview: The 3-Stage Teacher–Student Pipeline

Rather than tightly coupling two agent directories, the pipeline uses a vendor-neutral Intermediate Representation (teacher-curriculum-v1) split across three decoupled stages:

Each stage has a single responsibility:

1. Stage 1 (teacher_export.py) runs on the Teacher machine—either on-demand or automatically chained via systemd immediately after the nightly dream cycle. It harvests 9 local knowledge sources, strips secrets and environment-specific terms, generalizes project incidents into reusable engineering invariants, and emits an immutable bundle (teacher-curriculum-YYYY-MM-DD-<hash>.json plus a human-readable .md digest).
2. Stage 2 (Transport Layer) moves that bundle across trust boundaries using whichever one-way or broadcast mechanism fits your network topology—SSH/SCP bastion, GitLab repository, GitLab API snippet, encrypted Signal attachment, webhook, or P2P synced inbox.
3. Stage 3 (student_assimilate.py) runs on one or many Student machines on a timer (e.g., every 30 minutes), on webhook/Signal arrival, or right before the Student’s own nightly dream cycle. It deduplicates against existing memory (SequenceMatcher >= 0.85), tags provenance ([teacher:<bundle_id>]), quarantines incoming executable skills for optional review, and merges everything atomically with SHA-256 Optimistic Concurrency Control (OCC).

2. Stage 1 Deep-Dive: Harvesting, Redacting, and Generalizing (teacher_export.py)

A high-signal curriculum cannot rely on a single file. On the Teacher workstation, important engineering wisdom is scattered across episodic summaries, negative constraints, architectural write-ups, and reusable runbooks.

The 9 Knowledge Sources

teacher_export.py harvests across nine complementary sources on the Teacher host:

1. Negative Constraint Ledger (Meta/lessons.md): Explicit “never do X again” rules and user corrections.
2. Long-Term Declarative Memory (Meta/claude-memory.md / Meta/antigravity-memory.md): Stable technical discoveries and system invariants.
3. Global Behavioral Directives (CLAUDE.md / Meta/agent-best-practices.md): Standing engineering discipline, verification standards, and workflow rules.
4. Structured Knowledge Notes (Knowledge/*.md): Deep topic notes such as transactional-outbox checklists, cache consistency patterns, or container runtime quirks.
5. Global Procedural Skills (~/.claude/skills/*/SKILL.md and ~/.agents/skills/*/SKILL.md): Repeatable multi-step operational runbooks.
6. Dream Journal (Meta/dream-journal.md): Consolidated syntheses from recent nightly REM cycles.
7. Session Compaction Summaries: High-signal turn summaries extracted from local CLI session logs.
8. Per-Project Memory Stores: Repository-scoped notes stored under project memory directories.
9. Workflow Profile (Meta/user-profile.md): Non-sensitive developer preferences (e.g., preferred tooling, PR conventions, Linux/JVM conventions) while excluding personal identity fields.

Defense-in-Depth Redaction & Generalization

Raw notes almost always leak context that must never leave the source machine or cross team boundaries. Stage 1 applies a two-tier sanitization pipeline before anything is written to the bundle:

1. Deterministic Regex Scrubbing: Strips Bearer tokens, API keys, PEM private key blocks, database connection strings, email addresses, phone numbers, internal RFC1918/CGNAT IPs, and /home/<user> or /Users/<user> filesystem prefixes.
2. Custom Policy Redaction (~/.config/agent-transfer/config.json): Organizations and individuals can declare custom vocabulary lists and regexes to erase proprietary product names, internal domains, customer codes, or repository prefixes:

{
  "redaction": {
    "redact_terms": [
      "AcmePayments",
      "internal-core-gateway",
      "corp.internal"
    ],
    "redact_regexes": [
      "\\bACME-[0-9]+\\b",
      "\\b[a-z]+-prod-db-[0-9]+\\b"
    ],
    "replacement": "[REDACTED]"
  }
}

3. Semantic Generalization (--engine auto|llm|heuristic): When running with LLM distillation enabled (auto or llm), the exporter rewrites narrow incident logs into portable architectural invariants. For example:
   • Raw workstation incident: "Service order-router-eu kept stale cache on pod-2 when Kafka topic payments.evict used group order-router-group in acme-prod."
   • Distilled curriculum invariant: "Multi-pod stale cache bugs depend on topology. Active/active services need per-pod invalidation (broadcast topic with a unique consumer group per pod, or a non-durable, non-shared JMS subscription). A shared or durable subscription makes pods compete and recreates the bug."

Real-world invariants exported through this stage cover cross-cutting lessons any backend or platform team benefits from:

• Pinning a dependency in a Gradle subprojects { constraints } block does not propagate to external repos consuming a published version catalog.
• Migrating synchronous cross-service REST writes to a transactional outbox requires MANDATORY transaction propagation, business-ID keying, and idempotent consumers—while keeping synchronous reads read-only REST.
• In Jedis 7.x, legacy JedisPooled and setex patterns give way to RedisClient.builder() with ConnectionPoolConfig and SetParams.ex(ttl) (portable across Redis and Valkey).
• On ubi9/openjdk-*-runtime container images, run-java.sh runs as CMD (not ENTRYPOINT) and ignores JAVA_ARGS; pass program arguments explicitly and JVM flags via JAVA_OPTS_APPEND.
• With dnf5, upgrade packages live in /var/lib/dnf/offline separately from /var/cache/libdnf5, so dnf clean all preserves downloaded upgrade RPMs while dnf system-upgrade clean deletes them.

The Portable Curriculum Schema (teacher-curriculum-v1)

Both teacher_export.py and student_assimilate.py communicate exclusively through a strict, versioned JSON envelope:

{
  "schema": "teacher-curriculum-v1",
  "bundle_id": "777ec864",
  "generated_at": "2026-11-13T11:50:00+02:00",
  "teacher": {
    "agent": "claude",
    "engine": "llm"
  },
  "lessons": [
    "Compare normalized identifiers before equality checks: an 8-char BIC vs 11-char BIC silently disables String.equals guards."
  ],
  "invariants": [
    "Admission and gating decisions requiring strict consistency must read authoritative DB state under a lock, not an eventually-consistent cache."
  ],
  "facts": [
    "Hibernate @NotFound(action=IGNORE) handles dangling references on reads, but still requires explicit handling across write and accept paths."
  ],
  "knowledge_notes": [
    {
      "title": "Replacing synchronous cross-service REST writes with a transactional outbox",
      "markdown": "..."
    }
  ],
  "skills": [
    {
      "name": "crc-recovery-checklist",
      "description": "Recover local OpenShift (CRC) clusters after disk pressure or stale internal registry tokens.",
      "instructions_markdown": "..."
    }
  ]
}

On-Demand vs. Nightly Chained Execution

Depending on how fast knowledge needs to move, teams and solo engineers can trigger Stage 1 in two ways:

• On-Demand: Immediately after solving a nasty production or build bug, run python3 ~/scripts/teacher_export.py --transport auto --force to publish a fresh bundle right away.
• Automated Nightly Chaining: Running teacher_export.py --install-systemd --dream-unit claude-dream.service wires a user systemd service with ExecStartPost (or After=/WantedBy=) directly to the Teacher’s nightly dream cycle. Every time the Teacher consolidates its own memory at 02:30 AM, it automatically redacts, generalizes, and publishes the delta bundle before you start work the next morning.

3. Stage 2 Deep-Dive: The Communication & Transport Matrix

No single file-transfer protocol fits every network or organizational security posture. A personal setup behind a firewall has different constraints than a 15-person engineering squad working behind corporate proxies or across air-gapped networks.

By decoupling Stage 1 and Stage 3 behind a common transport interface, all of the following communication channels work seamlessly—and multiple transports can remain enabled simultaneously in config.json.

Option 1: SSH / SCP Relay Dropzone (scp_relay) — Best for Zero-Cloud Automation

If you control a neutral Linux host reachable by both Teacher and Student machines (a cloud VPS, an internal team bastion, or an always-on jump server), SCP over Ed25519 SSH keys is the most reliable zero-cloud transport:

1. Generate a dedicated keypair on the Teacher (and Student) machine:ssh-keygen -t ed25519 -f ~/.ssh/agent_transfer_ed25519 -N "" -C "agent-transfer" ssh-copy-id -i ~/.ssh/agent_transfer_ed25519.pub -p 2222 transfer@relay.example.com
2. Restrict the relay configuration in ~/.config/agent-transfer/config.json (chmod 600):{ "transports": { "scp_relay": { "enabled": true, "host": "relay.example.com", "port": 2222, "user": "transfer", "identity_file": "~/.ssh/agent_transfer_ed25519", "remote_dir": "/srv/agent-transfer/curriculum" } } }
3. Why it excels: Both teacher_export.py and student_assimilate.py invoke ssh/scp with -o BatchMode=yes and explicit IdentityFile pinning. Uploads write non-destructively into a timestamped/hashed dropzone, and any authorized Student machine (yours or a teammate’s read-only SSH key restricted via rrsync or authorized_keys command constraints) can poll and ingest bundles unattended.

Option 2: Self-Hosted GitLab Repository (gitlab_repo) — Best for Team Review & Auditability

For engineering teams that already collaborate on a self-hosted GitLab instance, a dedicated repository (e.g., engineering/agent-curriculum) turns agent knowledge sharing into a standard Git workflow:

• Teacher Push: teacher_export.py commits the sanitized .json and .md curriculum bundle to a feature branch or directly to main.
• Optional Merge Request Gate: Instead of pushing straight to main, a team can push bundles to incoming/<engineer-name> and open a GitLab Merge Request. Teammates—or CI secret-detection jobs (gitleaks, trufflehog)—review the diff of the .md digest before merging it into the team’s canonical knowledge stream.
• Student Pull: Every team member’s student_assimilate.py runs git pull --ff-only on a 30-minute timer and assimilates newly merged bundles automatically.

Option 3: Self-Hosted GitLab API Snippets (gitlab_snippet) — Best for Lightweight Gitless REST Push

When spinning up a full Git checkout on every machine feels too heavy, GitLab Snippets offer a single-endpoint REST store (PUT /api/v4/snippets/:id and GET /api/v4/snippets/:id/raw) authenticated with a scoped PRIVATE-TOKEN:

{
  "transports": {
    "gitlab_snippet": {
      "enabled": true,
      "base_url": "https://gitlab.internal.example.com",
      "snippet_id": "42",
      "token_env": "GITLAB_TRANSFER_TOKEN"
    }
  }
}

• Zero local git state: Ideal for ephemeral CI runners, containers, or locked-down workstations where outbound HTTPS to internal GitLab is allowed but SSH port 22/2222 is blocked.

Option 4: Encrypted Messaging & ChatOps (signal, Matrix, Mattermost / Slack Webhooks)

Sometimes knowledge transfer starts interactively—dropping a bundle directly into a chat channel or receiving real-time notifications:

• Signal (signal-cli JSON-RPC): teacher_export.py can attach the teacher-curriculum-*.json bundle directly to an end-to-end encrypted Signal message (either self-messaging across linked devices or a dedicated team channel). On the receiving node, a persistent daemon watches incoming attachments, automatically detects "schema": "teacher-curriculum-v1", stages the JSON into the inbox, and triggers assimilation immediately—plus supports an instant assimilate chat command.
• Matrix / Mattermost / Slack Webhooks: For teams living in Mattermost or Slack, posting the human-readable .md digest alongside a link or attachment to the .json bundle lets engineers see what the collective AI just learned in real time (e.g., “3 new Hibernate & Kafka invariants published by Alice’s agent”).

Option 5: Shared Inbox Filesystems & P2P Sync (inbox via Syncthing, Tailscale, NFS, or S3/MinIO)

Whenever 00-Inbox/teacher/ is backed by a synchronized folder or shared mount, zero-network-code transports work out of the box:

• Syncthing (P2P Mesh): End-to-end encrypted peer-to-peer folder sync across laptops and workstations with zero central server.
• Tailscale Drive / Taildrop / WireGuard NFS: Mount an internal shared folder across a private overlay network.
• S3 / MinIO Object Bucket: Upload bundles with aws s3 cp or mc cp into a versioned team bucket with lifecycle retention and IAM per-team read/write policies.

Transport Comparison Matrix

4. Stage 3 Deep-Dive: Teacher–Student Combinations & Team Topologies

Because teacher-curriculum-v1 is agent-agnostic, the Teacher and Student do not need to run the same AI coding assistant. You choose the Teacher source and the Student target independently.

All Supported Agent Combinations

Team Collaboration Topologies

Beyond syncing one engineer’s machines, this exact architecture scales to whole teams:

1. 1-to-1 Personal Bridge: Your primary coding machine exports nightly (teacher_export.py) to an SSH relay or Syncthing inbox; your secondary machine pulls (student_assimilate.py) before you open a terminal.
2. 1-to-N Tech Lead Broadcast: A principal or staff engineer maintains a curated Teacher agent. Whenever the lead discovers a repo-specific gotcha—say, why an ORM @NotFound mapping breaks write paths or why a local cluster’s ImagePullBackOff stems from stale default-dockercfg-* secrets—their machine exports a bundle. Every developer on the squad runs student-assimilate.timer every 30 minutes and inherits that guardrail automatically.
3. N-to-N Collective Squad Hub (via GitLab): Every team member runs teacher_export.py pointing to a shared team-agent-curriculum GitLab repository using per-engineer branches (contrib/alice, contrib/bob). An automated MR checks for redaction compliance, merges net-new lessons into main, and every teammate’s Student daemon pulls the collective wisdom of the entire engineering org. One engineer fixes an obscure bug once; nobody on the team hits it again.

5. Student Safety Mechanisms: Deduplication, Quarantine, and OCC

Ingesting external knowledge automatically into an AI agent’s long-term memory carries real risks: duplicate bloat, circular feedback loops, poisoned instructions, and race conditions against live sessions. student_assimilate.py enforces four strict guardrails before writing a single line.

1. Watermark Tracking & Fuzzy Deduplication (>= 85%)

Every imported bundle is recorded by bundle_id and target harness (claude, agy) in Meta/states/student-assimilation-state.json. Even when a Teacher runs --force to emit a fresh snapshot containing both old and net-new discoveries, the Student evaluates every incoming lesson, invariant, and fact against existing memory using normalized difflib.SequenceMatcher similarity (DUP_RATIO = 0.85).

When running assimilation on a freshly updated bundle in production, the output cleanly reports the delta:

$ python3 ~/scripts/obsidian_memory.py assimilate --fetch scp-relay
Assimilated 1 curriculum bundle(s) into targets=['agy']:
  [agy] +9 lesson(s), +26 fact(s), +6 note(s), +0 skill(s) installed, +0 draft(s) queued, 3 duplicate(s) skipped

Notice that 3 duplicate(s) were caught and skipped automatically while 9 net-new lessons, 26 facts, and 6 architectural knowledge notes were merged cleanly.

2. Provenance Tagging

Every assimilated invariant and lesson carries an explicit provenance suffix such as [teacher:777ec864]. If a rule later turns out to be outdated or overly broad, you know in seconds which bundle introduced it and which Teacher exported it.

3. Skill Quarantine & Review Gate (review vs. install vs. skip)

Prose facts are low-risk, whereas executable SKILL.md runbooks contain shell commands the agent will follow. Accepting unreviewed executable runbooks blindly across a team is a supply-chain risk.

To guard against this, student_assimilate.py supports --skill-mode:

• review (Default for untrusted/team transfers): Writes incoming SKILL.md files into a staging directory (00-Inbox/teacher/skills-review/<skill-name>/SKILL.md) where the human developer can inspect them first, then promote approved skills with:python3 ~/scripts/student_assimilate.py --promote-skill <skill-name>
• install: Auto-installs verified skills directly into ~/.agents/skills/ and/or ~/.claude/skills/ (convenient for single-owner trusted workstation bridges).
• skip: Imports declarative facts and negative constraints only, ignoring procedural skills entirely.

4. SHA-256 Optimistic Concurrency Control (OCC) & Dream Pre-Hook

Just like the nightly dreaming engine described in Learning While You Sleep: Out-of-Band Memory Dreaming, Student writes acquire an exclusive fcntl.flock lock, hash the target Markdown files before and after staging, re-merge if an interactive agent edited the file mid-flight, and append an audit entry directly into Meta/dream-journal.md.

Even better: chaining student_assimilate directly as an ExecStartPre hook ahead of the Student machine’s nightly dream service ensures incoming Teacher lessons are always merged before the Student runs its own REM deduplication and verification cycle.

6. Putting It All Together: Quick-Start Deployment Checklist

Whether you are bridging two personal development machines or rolling out federated agent memory across a 10-person backend squad:

1. Configure shared policy & redaction (~/.config/agent-transfer/config.json, mode 0600):
   • Define redact_terms and redact_regexes for internal company names, repos, and ticket prefixes.
   • Enable one or more transports (scp_relay, gitlab_repo, gitlab_snippet, signal, or inbox).
2. Enable Stage 1 on the Teacher machine(s):# Test export safely without pushing python3 ~/scripts/teacher_export.py --engine auto --dry-run # Chain nightly export automatically after the Teacher's dream service python3 ~/scripts/teacher_export.py --install-systemd --dream-unit claude-dream.service
3. Enable Stage 3 on each Student machine:# Choose target harness: claude, agy, both, or auto python3 ~/scripts/student_assimilate.py --student claude --skill-mode review --install-systemd
4. Verify end-to-end delivery:
   • Trigger an immediate push on the Teacher (--force) and run assimilate on the Student.
   • Confirm Meta/states/student-assimilation-state.json, Meta/lessons.md, and Meta/dream-journal.md record the new bundle_id cleanly with zero leaked internal terms.

Conclusion

AI coding assistants become dramatically more valuable the moment memory ceases to be a fragile, single-machine cache. By compiling raw workstation experience into a redacted, generalized teacher-curriculum-v1 contract and decoupling extraction, transport, and assimilation, you turn isolated agents into a federated learning network.

Your primary machine can teach your secondary machine while you are away from your desk. Your team’s senior engineers can share production debugging wisdom with every developer’s agent within 30 minutes—over plain SSH relays, GitLab Merge Requests, encrypted Signal messages, or P2P folders—without ever sacrificing privacy, auditability, or harness independence.