PranavMishra17/SoulEngine

What is SoulEngine?

SoulEngine transforms static game NPCs into genuinely evolving entities. Characters remember player interactions, develop personalities over time, speak with their own voices, and take autonomous actions in the game world. A dual-instance Mind lets the Speaker respond instantly while a parallel thinker reasons with tools in the background.

The Five Pillars of SoulEngine NPCs

Features

Unity SDK coming soon!

Multi-Provider LLM, TTS, and STT

Switch providers per-project. Use your own API keys (BYOK — encrypted at rest, never logged).

Flexible Conversation Modes

Memory Architecture

Short-Term Memory (STM): Created at session end from a detective-style LLM summary that captures specific facts, phrases, and names — not emotional atmosphere. Filtered against injection patterns while preserving legitimate player-shared content.

Long-Term Memory (LTM): Synthesized at weekly whisper time. Multiple STM entries are compressed by an LLM into condensed, insight-level observations. Raw entries are removed from STM after promotion — no duplication.

Per-NPC Memory Retention: Configurable salience_threshold per NPC. Low threshold = genius-level recall (2-sentence summaries, promotes more to LTM). High threshold = forgetful character (1-sentence summaries, most memories fade).

Player Identity System

NPCs can be told who the player is before conversation starts:

• Player name, description, role, context
• Bidirectional network: "You know them" vs "You know of them (famous)"
• Relationship persistence: trust, familiarity, sentiment tracked per player

NPC Social Graph

Each NPC has a configurable network of relationships with other NPCs, with tiered familiarity levels controlling what information they share in context:

Full Version History

Every state change creates a versioned snapshot — rollback is always available.

NPC Definition History: Every time you save changes to an NPC's personality, voice, backstory, etc., the previous version is archived. View field-level diffs, revert to any prior version.

Mind State History: Every session end, daily pulse, weekly whisper, and persona shift creates a snapshot of the NPC's runtime mind (mood, STM, LTM, trait modifiers, relationships). View any historical snapshot in the UI. Revert to any prior mind state.

Security

• Core Anchor immutability: Enforced at the cycle logic layer and session integrity check. Modifications are detected and rejected.
• Input sanitization: XSS prevention, injection pattern detection. Quoted content preserved (doesn't strip legitimate player phrases).
• Content moderation: Keyword-based, triggers in-character conversation exit.
• Rate limiting: Per-player per-NPC per-minute.
• Narration stripping: (stage directions) and *actions* stripped from all LLM responses post-processing, both in text and voice modes.
• Game Client API Key: SHA-256 hashed. Required for external game clients (Unity), bypassed for authenticated dashboard users.

MCP Tool System

Three tool types for different decision authorities:

Define tools once in the web UI, assign permissions per NPC, implement handlers in your game client.

NPC Mind (Parallel Dual-Instance Architecture)

Every conversation turn runs two LLM instances in parallel:

How it works:

• Speaker streams the instant reply immediately -- zero latency from Mind, pure voice, no tool overhead.
• Mind runs in parallel, evaluating whether tools are needed and executing an agent loop if so.
• Recall tools (recall_npc, recall_knowledge, recall_memories): results are deferred and injected into the Speaker's prompt on the next turn. No follow-up speech, no added latency.
• MCP/project tools (request_credentials, lock_door, call_guards, etc.): trigger a short follow-up speech in the same turn addressing the action taken.
• Always on. No feature flag -- every turn benefits from the split.

Tool ownership:

• Recall Tools (built-in): recall_npc, recall_knowledge, recall_memories -- Mind fetches context on demand; results deferred to next turn's prompt.
• Conversation Tools (project-defined): warn_player, call_police, etc. -- Mind decides when to invoke them; results produce a brief follow-up response.

Cost control: Mind LLM provider and model are configurable per project (defaults to the project LLM). The slim Speaker context achieves 29-57% token savings vs the previous full-context approach.

Web UI

Full management and testing interface — no build step required.

NPC Editor (9 tabs)

1. Basic Info — Name, description, profile picture, draft/complete status
2. Core Anchor — Backstory, principles, trauma flags
3. Personality — Big Five sliders, preset archetypes, memory retention slider
4. Voice — Provider, voice browser with previews, speed control
5. Knowledge — Depth-level knowledge access assignment per category
6. Schedule — Time-block routines (location + activity)
7. MCP Tools — Conversation and game-event tool permissions
8. Network — NPC social graph with familiarity tiers and mutual/one-sided awareness
9. History — Mind state snapshots + definition version timeline, both with revert buttons

Testing Playground

• 4 conversation modes
• Live NPC State panel: real-time mood bars, memory counts, latest memory, daily pulse
• Cycle trigger panel: run daily pulse / weekly whisper / persona shift from the UI
• World Context panel: project overview, NPC roster, knowledge tiers, available tools
• Player identity configuration per session

Project Settings

• LLM/TTS/STT provider configuration
• Mind LLM provider, model, and timeout configuration (defaults to project LLM)
• Per-project API key management (encrypted)
• Game Client API Key generation and revocation
• Import API keys from another project
• Project limits and timeout configuration

Quick Start

# Clone the repository
git clone https://github.com/PranavMishra17/SoulEngine.git
cd SoulEngine

# Install dependencies
npm install

# Configure environment
cp .env.example .env
# Add your API keys (at least one LLM provider required)

# Start development server
npm run dev

# Open in browser
open http://localhost:3000

Environment Variables

# LLM Providers (at least one required)
GEMINI_API_KEY=your_key
OPENAI_API_KEY=your_key
ANTHROPIC_API_KEY=your_key
GROK_API_KEY=your_key

# Voice Providers
DEEPGRAM_API_KEY=your_key      # Speech-to-text
CARTESIA_API_KEY=your_key      # Text-to-speech (default)
ELEVENLABS_API_KEY=your_key    # Text-to-speech (alternative)

# Configuration
DEFAULT_LLM_PROVIDER=gemini
ENCRYPTION_KEY=your_32_char_key_for_api_storage

# Production (Supabase)
SUPABASE_URL=your_url
SUPABASE_SERVICE_ROLE_KEY=your_key

Project Structure

src/
+-- index.ts              # Server entry point
+-- config.ts             # Environment configuration
+-- providers/
|   +-- llm/              # LLM factory (Gemini, OpenAI, Anthropic, Grok)
|   +-- stt/              # Speech-to-text (Deepgram)
|   +-- tts/              # Text-to-speech (Cartesia, ElevenLabs)
+-- storage/              # Dual-backend storage (local filesystem + Supabase)
+-- core/                 # NPC cognition (memory, personality, cycles, summarizer, mind)
+-- session/              # In-memory session management
+-- mcp/                  # MCP tool registry and execution
+-- voice/                # Multi-modal voice pipeline
+-- security/             # Sanitizer, moderator, rate limiter
+-- routes/               # REST API endpoints
+-- ws/                   # WebSocket voice handler

web/                      # Web UI (vanilla JS SPA, no build step)
+-- index.html            # SPA with all page templates
+-- css/                  # Design system
+-- js/                   # Router, API client, page modules

API Overview

Session & Conversation

Memory Cycles

Mind State History

Projects & NPCs

WebSocket: ws://localhost:3001/ws/voice?session_id=xxx

Tech Stack

Documentation

• System Design — Full architecture, all design decisions, and implementation details
• Chat Interface — Voice and text chat interface, details of VAD, Mind State, MCP tools, Streaming
• Unity SDK — Unity integration plan, scene setup guide, feature mapping
• Add Providers — How to add additional LLM/TTS/STT providers

License

Academic/Research Use Only

Connect with me