eq-micro-expression-langchain-v-4 · Emotion API v1 shippedAn affective operating system for embodied AI.
Project Theo-SaaS perceives emotion from face, voice and text; reasons about what a speaker actually means; and gates every LLM reply and robot gesture through a deterministic safety harness — entirely on the local network. It is hardware-agnostic middleware that decouples emotional intelligence from robot actuation.
“I'm fine.” — the Pragmatic Context Deficit
Literal words carry a fraction of intent. When a patient says “I'm fine” while AU4 flickers across the brow, a standard language model takes the statement at face value and moves on. Theo-SaaS treats language as an act performed by a speaker in a state — so the same words can, and should, produce different decisions.
A speech act, not a string
Austin and Searle: an utterance is an action with felicity conditions. The Sentinel classifies the act first, then asks whether the reply honours it.
The speaker profile is load-bearing
Register, irony, prosodic contour and disfluency are threaded into the verdict. Same text, different speaker profile, different decision — by design and by test.
Leakage warms the tone; it never accuses
A micro-expression leak is routed as a separate, discounted channel. It can soften a reply. It can never out-vote the visible display or be framed as deception.
Six layers. One gate that everything passes through.
Perception feeds fusion; fusion feeds interpretation; interpretation conditions cognition. Nothing reaches actuation without clearing the harness. Select a layer to see what it does and where it lives in the tree.
The Deterministic Safety Harness
Every other component on this page exists to feed, or be governed by, this layer. Four properties, each implemented as plain Python with fixed thresholds — say less when unsure, and never accuse. The demos below run the real constants from the source.
Byte-identical replay
Same events + same seed + same model cache ⇒ the same action stream, byte for byte. For a system that turns model output into physical motion, replayability is a safety control. 32/32 determinism tests.
apps/determinism/replay.pyapps/determinism/clock.pyKnown-answer self-test
A deployed instance can prove at runtime that it still behaves as certified: GET /safety/selftest runs deterministic KATs across VAD, AU inference, tiering and the Sentinel — including that a celebratory reply to “I lost my job today” is refused.
apps/api/app/services/emotion_safety.py::run_selftestEQ scored on human instruments
The robot's emotional intelligence is a measured subsystem: MSCEIT, GERT (Rasch-weighted), GECo and Trait EI, with measurement-invariance checks and Wilson intervals. ESConv empathy preference 0.460 ≥ 0.45; support-assembly latency p95 ≈ 0.04 ms against a 200 ms HRI budget.
shared/eq_bench/POST /eq/scorecardEmotion API v1 — the whole stack behind eleven routes
Inputs are text, classifier scores or FACS Action-Unit intensities. Outputs are platform-neutral behavior-intent tokens, never joint trajectories — each vendor's bridge maps them to whatever the hardware supports. capabilities reports hardware_required: false. Select a route to see a real request and response.
Per-key policy isolation
Each API key resolves to its own PolicyEngine — use context, attestations and audit trail are never shared across tenants.
Append-only audit
Every allow / deny decision is appended to an in-memory ring and, when EMOTION_API_AUDIT_PATH is set, a durable JSONL file readable via GET /audit.
Graceful degradation
Missing optional stacks report available: false, never an exception. Anomalous inputs force abstention rather than action.
Shipped as a container
deploy/emotion_api/Dockerfile, docker-compose.emotion-api.yml and a GitHub Actions CI workflow. De-branded via EMOTION_API_MODEL_ROOT.
What's new in eq-micro-expression-langchain-v-4
Grounded in the branch diff against eq-micro-expression-v-3. Everything below is on disk; nothing is aspirational.
Emotion API v1 complete
Eleven routes under /api/v1/emotion productising the analysis stack for third-party hardware, with the use-context policy engine, runtime safety supervisor and known-answer self-test.
LangChain in-process cognition layer complete · ADR 038
Eleven-module PEP 562 lazy package: LAN-guarded ChatOpenAI factory that fails closed on tracing env vars, LCEL chains over the emotion prompt builder (parity test-enforced), session memory, fail-open structured output, deterministic EKTC retriever, SSE streaming.
Bidirectional voice loop full-duplex E2E pass
Riva Nemotron cache-aware streaming ASR and Magpie TTS running concurrently with Llama-3.1-8B on one 16 GB RTX 4080 SUPER. Required patching NIM RMIR archives and shrinking Magpie bs8 → bs2 (10.5 → 6.0 GB).
n8n Agent 11 — Empathy Chat bridge import pending
Any workflow or LAN client obtains a full empathetic turn through one webhook, bridged to POST /api/v1/lc/chat with correlation-ID propagation and per-turn observability samples.
Model-root de-branding & containerised delivery complete
msa/model_root.py decouples model discovery from the Reachy chassis; the API ships as its own Docker image with a GitHub Actions CI pipeline.
Governance: ADR 038, ADR 039, three engineering reports
ADR 038 fixes the LangChain ⊕ n8n division of labour and refuses LangSmith. ADR 039 separates the capability of durable cognitive state from any vendor, under five binding privacy conditions. Reports 1–3 inventory completed components with evidence.
The remaining gap is not algorithmic.
It is hardware access, data licensing and external validation — none of which more code will solve.
| Gap | State |
|---|---|
| Physical-robot validation | SDK bridge raises NotImplementedError; all validation is simulator / deterministic-fake |
| Licensed ME corpora | CASME II / SAMM / SMIC / MMEW loaders and gate checker complete; institutional access not obtained |
| Triton edge Sentinel adapter | Deliberate skeleton; separate Jetson engineering effort |
| n8n Agents 10 / 11 | Code and replay-verified contracts done; manual import + GATEWAY_BASE_URL pending |
| Legal / notified-body AI Act review | Technical guardrails implemented and tested; no external review |
| Cognitive Orchestration Engine | Design complete (ADR 039 + plan); implementation not started — §12 |
docs/reports/REPORT_2_EXECUTIVE_SUMMARY.mdCalibration is what makes confidence-gating honest
EfficientNet-B0 (HSEmotion VGGFace2 + AffectNet weights) fine-tuned for happy / sad / neutral. Two interventions took the engine from V1 to the deployed V2: mixed-domain training — 15,000 real AffectNet images added to 86,519 synthetic frames, test set excluded — then post-hoc temperature scaling at T = 0.59, which cut calibration error by 65% without touching a single argmax.
| Metric | V1 baseline | V2 mixed + T |
|---|---|---|
| Macro F1 | 0.781 | 0.916 +17.3% |
| Balanced accuracy | 0.799 | 0.921 +15.3% |
| Expected Calibration Error | 0.102 | 0.036 −64.7% |
| neutral → sad confusion | 35.1% | 5.7% |
| Gate A | — | 7 / 7 pass |
trainer/fer_finetune/temperature_scaling.pytrainer/gate_a_validator.pymemory-bank/decisions/012-…Micro-expressions as a leakage channel, never a vote
FACS / EMFACS Action Units, dependency-free optical-strain spike filtering, ECC-stabilised face alignment, loaders for CASME II / SAMM / SMIC / MMEW and MEGC-standard spotting metrics — delivered with zero edits to any existing file. Toggle Action Units to see which prototype or suppression signature fires.
msa/visual/micro_expression/action_units.pyoptical_flow.pyconcealment.pyeval/harness.pyThe Pragmatic Sentinel vets every candidate reply
A neuro-symbolic gate between probabilistic language production and anything externally visible. Speech-act felicity, indirect acts, Gricean maxims, Horn-scale implicature, presupposition, Sperber–Wilson relevance, Brown–Levinson face-threat weight and affective grounding — each sub-score written to an append-only audit sink. It fails closed.
apps/pragmatics/sentinel.pyapps/pragmatics/revision.pyapps/linguistics/analyzer.pyTell the model the emotion only when you'd bet on it
The defining behaviour of the dialogue layer: the LLM is told the detected emotion only when fused confidence ≥ 0.60. Below that, the prompt states the read is unclear and instructs the model to elicit rather than assume. The system is built to admit uncertainty about feelings rather than perform false empathy.
LangChain complements n8n — it never replaces it
n8n owns system-level orchestration across 11 agents (ingest → label → promote → reconcile → train → evaluate → deploy → privacy → observability → gesture → empathy-chat). apps/llm/lc/ owns the single in-process LLM turn: LCEL chains, session memory, structured output, EKTC retrieval. Bridges run both ways.
ESConv support planner
Eight strategies under Hill's Helping-Skills stages (Exploration → Insight → Action). Advances when distress is not rising; regresses to Exploration on a distress spike. EKTC commonsense knowledge is tool-called only when network regime and somatic state warrant it — zero prompt cost otherwise.
Voice, full-duplex, one GPU
Nemotron cache-aware streaming ASR + Magpie TTS + Llama-3.1-8B co-resident on a single 16 GB RTX 4080 SUPER. Verified end to end: mic WAV → verbatim transcript → emotion=sad/0.82 → empathetic reply carrying [SAD_ACK][LISTEN] → 8.64 s of spoken audio. Voice and text share one session store. No robot required.
apps/llm/lc/chat_model.pyapps/llm/esconv/planner.pyapps/llm/ektc/apps/llm/asr_riva.pyapps/llm/tts_riva.pymemory-bank/decisions/038-…Behavior intents, not joint trajectories
The API emits tokens such as comfort, celebrate, listening. A vendor's RobotBridge adapter maps them to whatever the chassis supports; unknown tokens are skipped. LLM replies carry bracketed gesture keywords — all 14 bridged and tested — parsed, passed through a five-tier confidence modulator, dispatched, and stripped from the speech the user hears.
Two deployment profiles, one contract
Mini Lite runs on an RTX 4080 workstation via ONNX Runtime CUDA; the Jetson path converts ONNX → TensorRT FP16 with trtexec, ships a health server and a Gate B validator, and rolls out shadow → canary → production with automatic rollback.
| Gate | Scope | Threshold |
|---|---|---|
| A-val | Synthetic validation, gates ONNX export | F1 ≥ 0.84 · bal. acc ≥ 0.85 · ECE ≤ 0.12 |
| A-deploy | Real-world test tier | F1 ≥ 0.75 — current engine 0.916 |
| B (Mini Lite) | Runtime on RTX 4080 host | ≥ 60 FPS · p50 ≤ 40 ms |
| B (Jetson) | TensorRT FP16 on Xavier NX | ≥ 25 FPS · p50 ≤ 120 ms · ≤ 2.5 GB |
| C | Mini Lite end-to-end | perception → reply → gesture within HRI budget |
| HRI | Resonance window | ≤ 200 ms end-to-end · Godspeed Animacy ≥ 3.5 |
What the robot actually does with a prediction
A reading of sad at 0.81 and one at 0.46 produce the same words and a visibly different robot. Confidence sets gesture amplitude, gesture duration and speech rate from one table; the per-emotion system prompt sets what may be said at all. Press play and watch a single turn run from percept to motion.
The voice carries the same constraint as the words
Each emotion's system prompt states a behavioural rule, and that rule shapes prosody as well as content. The anger prompt is the clearest case: do not match their intensity. So the reply slows and flattens rather than rising to meet the user — the opposite of what a naive sentiment-matching system would do.
Gesture keywords are parsed from the reply, dispatched to the bridge, and stripped before the text is spoken. The listener hears sentences; the chassis receives tokens.
| Emotion | Prompt constraint | Default gesture |
|---|---|---|
| happy | Match their energy; stay genuine, not artificially cheerful | THUMBS_UP |
| sad | Validate, don't fix. Listen more than you speak | EMPATHY |
| anger | Do not match their intensity — calm presence is the offer | LISTEN |
| fear | Be an anchor; never amplify urgency | COMFORT |
| surprise | Start curious; pivot once you know if it's good or bad news | OPEN_ARMS |
| neutral | Attentive, ready to adapt if emotions shift | LISTEN |
The Mini Lite has no arms — head pose on three axes, two antennas and a body yaw. Every gesture above is re-interpreted into that vocabulary by the robot bridge, which is why a THUMBS_UP reads as a firm nod with the antennas perked. Trajectories and thresholds shown here are the repository's own; the dialogue is written to illustrate each prompt, and the voice uses browser speech synthesis in place of the Piper backend.
Cognitive Orchestration Engine
Design complete; implementation not started. ADR 039 separates the capability — durable, resumable, thread-keyed cognitive state — from any vendor. Today the honest diagnosis is simple: no object owns a session. Per-session state is built ad hoc in five places with five different lifetimes, so there is nothing to serialise, key or resume.
CognitiveState contract
A Pydantic v2 contract split into a replay-relevant projection (tick-indexed, no wall-clock, no RNG-derived values) and an observational envelope (timestamps, audit ids, provenance). Only the projection may influence an action. Dialogue text is referenced, never carried.
shared/contracts/cognition/state.py plannedSessionStore under C1–C5
Persistence of affective state is permitted only when C1 consent-gated, C2 TTL-bounded, C3 encrypted at rest, C4 verifiably purgeable — the test asserts plaintext bytes are absent from disk — and C5 no egress. Ethics review 039a must complete before it ships.
apps/cognition/store.py plannedOwnership consolidation
A CognitiveSession façade owns state, store, planner, monitor and Sentinel for one (subject, session), retiring the ad hoc owners. Only then does the continuity claim become true in the present tense.
docs/plans/cognitive-orchestration-engine-plan.mdcryptography dependency for Fernet, the proven physio/store.py storage layout, Pydantic component schemas, and a replay gate that is already a callable API (32/32 tests).apps/determinism/ remains the sole authority for replay.Where emotional intelligence creates real value
Especially valuable where emotional context matters but privacy constraints are non-negotiable — the same on-premise architecture underwrites every deployment below.
Companion Robotics
Loneliness and emotional isolation are pressing challenges in aging populations and mental-health contexts. Affective AI lets a companion robot detect emotional state and respond with appropriate gestures and dialogue — consistent, calibrated support rather than a scripted loop.
- Real-time emotion detection during conversation
- Gesture-aware empathetic responses —
HUG,EMPATHY,LISTEN - Confidence-gated modulation avoids over-expression at low certainty
- Fully on-premise — nothing from the home environment reaches a cloud service
Privacy constraints are non-negotiable in home and care settings. On-premise processing means residents never become training data for an external service.
Healthcare & Therapy
Clinical environments demand rigorous data governance alongside genuine empathetic capability. Affective AI pairs real-time emotional-state detection with locally-held data, calibrated confidence scores, and structured, correlation-ID audit trails.
- Calibrated ECE / Brier score — signals a clinician can trust, not raw softmax
- Audit trail: every decision logged with a
correlation_idandaudit_id - Gate A validation before any model reaches deployment
- Abstain below the confidence floor — no reading is asserted, ever overstated
Healthcare AI must be explainable and auditable. Affective AI provides calibration metrics, structured logs and fail-closed behaviour on a LAN-contained pipeline — the posture clinical deployments require.
Education
Engagement, frustration and confusion are difficult to measure from behaviour alone. Affective AI adds an emotional-intelligence layer to educational robots — detecting when a learner is confused, disengaged or frustrated, and adapting pacing and tone accordingly.
- The neutral class catches baseline disengagement before it compounds
- Confidence-gated feedback — no intervention on a low-certainty read
- Same LAN-only architecture as every other deployment — no separate policy to trust
- Gesture vocabulary tuned for encouragement —
LISTEN,THINK,NOD, never correction
Student data is among the most sensitive in any sector. The on-premise architecture keeps it on the institution's own network rather than routed through a third-party inference service.
Figures above restate claims made elsewhere on this page — zero cloud egress from the safety-harness posture (§4), the 200 ms Gate-HRI budget from actuation (§11) — not new measurements specific to any one deployment.
Built by a linguist who became an engineer
Project Theo-SaaS is designed and built by Russell Bray — three Master's degrees (Linguistics, Business Administration, Data Science), a decade of university-level instruction, and enterprise BI and technical-sales experience. It emerged from Project Reachy (Reachy_Local_08.4.2) and the Reachy Mini platform from Pollen Robotics.
Institutional pathway
An application under the French Statut National d'Étudiant-Entrepreneur (SNEE) and Diplôme d'Étudiant-Entrepreneur (D2E) framework, via Pépite Sorbonne Université, is currently under review, with an expected start in January 2027. Target academic anchor: the Institut des Systèmes Intelligents et de Robotique (ISIR / Sorbonne Université), toward a CIFRE industrial doctorate.
Open-core, hardware-free
No chassis to manufacture, no supply chain to carry. The baseline classifier and dataset formatters are slated for open publication; the harness, edge orchestration and compliance modules are licensed B2B. Publication is a committed next step, not a completed one.
Privacy as structure
All inference and training run on-premise. No raw video, audio or biometric signal leaves the deployment network; the chat-model factory refuses to start if tracing env vars are present. Retention, purge and consent are code paths, not policy PDFs.
Restricted contexts refuse biometric emotion inference unless a medical-or-safety attestation is recorded — the narrow exemption the Regulation allows. Text-only pragmatics is audited but not context-blocked (Art. 3(39) scopes emotion recognition to biometric input).
apps/api/app/services/emotion_policy.py::RESTRICTED_CONTEXTS