Description

[H] HyperspaceDB: The Spatial AI Engine

—

🌌 What is HyperspaceDB?

Traditional vector databases were built to search static PDF files for chatbots. HyperspaceDB is built for Autonomous Agents, Robotics, and Continuous Learning.

It is the world’s first Spatial AI Engine — a mathematically advanced memory infrastructure that models information exactly how the physical world and human cognition are structured: as hierarchical, spatial, and dynamic graphs.

By combining Hyperbolic Geometry (Poincaré & Lorentz models), Lock-Free Concurrency, and an Edge-to-Cloud Serverless architecture, HyperspaceDB allows machines to navigate massive semantic spaces in microseconds, using a fraction of the RAM required by traditional databases.

🧠 Why a Spatial AI Engine? (Beyond RAG)

AI is moving from text-in/text-out to autonomous action. Agents need episodic memory and spatial reasoning. HyperspaceDB provides the primitives to build it:

• Fractal Knowledge Graphs: Euclidean vectors fail at hierarchies. Our native Hyperbolic engine compresses massive trees (like codebases or taxonomies) into 64-dimensional spaces, reducing RAM usage by 50x without losing semantic context.
• Continuous Reconsolidation: AI agents need to “sleep” and organize memories. With our Fast Upsert Path, CDC Event Streams, and built-in Riemannian Math SDK (Fréchet mean, parallel transport), your agents can continuously shift and prune vectors dynamically.
• Edge-to-Cloud & Offline-First: Drones and humanoid robots can’t wait for cloud latency. HyperspaceDB runs directly on Edge hardware, using a Merkle Tree Delta Sync protocol (SyncHandshake, SyncPull, SyncPush) to asynchronously handshake and sync episodic memory chunks with the Cloud when the network is available.
• Serverless at Billion-Scale: HyperspaceDB dynamically unloads idle logic to disk/S3, enabling you to host millions of vectors across thousands of tenants on a single commodity server, acting as the “Neon of Vector Search.”

—

🚀 Core Pillars (v4.2.0)

🔌 Hyperspace PDK (hyperspace-pdk v4.2.0)	pgrx-inspired extension architecture turning HyperspaceDB into an extensible Spatial Runtime. Includes 3 built-in flagship plugins out of the box: hs-embed (Zero-Python in-engine Markdown chunker, multi-manifold vectorizer & sequential KAG graph linker), hs-git-ast (instant AST codebase symbol & KAG topology builder for Rust, Python, TypeScript/JS, and Go), and hs-shield (Zero-Knowledge O(D log D) FWHT orthogonal projection defending against Vec2Text inversion with 100% distance preservation). Exposed dynamically in Python, TypeScript, and Rust SDKs via client.ext.<plugin>.<method>(...).
📐 Universal 12-Metric SIMD Vector Engine (v4.2.0)	First-class support for 12 distance metrics (l2, cosine, dot, l1, linf, angular, jaccard, hamming, poincare, lorentz, hybrid, wasserstein) backed by hardware-dispatched (AVX-512F, AVX2+FMA, ARM NEON) zero-copy byte-slice kernels and zero-allocation HNSW search hot paths.
🕸️ Native Vector-Graph (KAG Engine)	Native in-engine topological graph representation (Property CSR with 12-byte packed edges in graph.csr). Replaces fragile dual-database patterns (vector DB + Neo4j) with sub-microsecond BFS traversals across vector search seed nodes, explicit edge linking (link, linkbatch, unlink), declarative metadata induction (pathhierarchy, foreignkey, exactmatch), and 33D Lorentz Minkowski light-cone causal subsumption.
⚖️ Tribunal Router (Geometric Hallucination Prevention)	Post-generation geometric verification endpoint (VerifyGeneration) in tribunal.rs. Computes on-the-fly geodesic paths and Lorentz light-cone boundaries between LLM generated answers and source context chunks, producing mathematical trustscore (0.0–1.0) and ishallucination flag without making any assumptions about underlying vector quantization.
🗺️ Enterprise Query Planner	Intelligent natural language query optimization and AST execution engine (hyperspacequeryplanner). Automatically decomposes user questions into optimized sub-queries, executes parallel vector/geometric retrieval, applies filters and light-cone traversals, and synthesizes enriched context. Available across MCP, LangChain, LlamaIndex, n8n, and Agent Skills.
⚡ Thread-per-Core Engine	Shared-Nothing architecture inspired by Seastar. Each physical CPU core runs an isolated, lock-free shard partition (ShardedCollection) with deterministic routing (hash(id) % N). Eliminates cross-core cache line bouncing and lock contention, delivering 142.5 QPS (C=1) to 311.6 QPS (C=10) on 1024D dense search (4.52× faster vs v3) and 67,500 vec/s server core ingestion throughput.
🧬 Out-of-Core SuperKMeans & ADSampling (v4.1.0)	Native, zero-FFI pure Rust implementation of SuperKMeans (hyperspace-clustering v4.1.0). Supports Out-of-Core Streaming K-Means via Dataset and Matrix<'a>, cutting peak RAM by up to 1,000× on large datasets (58 MB vs 61.4 GB for 10M vectors). Employs BKT-drop 2-phase splitting (100k vectors clustered in 144.88 ms at ~690,000 vec/s with CV = 0.277), Adaptive Distance Sampling (ADSampling) pruning 90%+ distance checks, and disk-backed Voronoi semantic compaction (SemanticCompactor).
🌐 Hierarchical Macro-Routing & 21.2B+ Addressing	2-level hierarchical macro-routing in MetaRouter clusters chunk centroids into super-regions, pruning probe evaluations from $O(C)$ to $O(sqrt{C} + K)$. Universal 64-bit ID addressing (u64) handles 21.2+ Billion vectors globally while preserving 32-bit RAM density inside HNSW graph nodes (local_id: u32).
💾 Direct I/O Hardware Storage	High-IOPS Direct I/O storage engine (DirectVectorStore) using 4096-byte page alignment and native ODIRECT (Linux) / FNOCACHE (macOS). Bypasses OS page cache double-buffering, slashing MRL cascade tail latency (P99) from 33.75 ms to 3.14 ms (10.76× lower), boosting MRL throughput to 397.1 QPS (6.16× higher) with 99.87% Recall@10.
📐 Spatial AABB Indexing & Hardware Prefetch	Hierarchical 32-vector Axis-Aligned Bounding Box (AABB) pruning discarding 95% to 99.9% of candidate vectors before fetching vector memory for InBox and InBall geometric queries, delivering 1,774.0 QPS (22.28× higher vs v3) at 0.51 ms P50 latency. CPU hardware cache prefetch hints (mmprefetch on x86_64, prfm pldl1keep on Apple Silicon/Graviton) hide DRAM latency during HNSW beam traversal.
🔄 Autonomous Startup Migration & Durability	Zero-downtime auto-migration engine converting legacy v3 collections to partitioned v4 storage. Enforces strict filtering against live id_map and logs WalEntry::Delete tombstones (OpCode 5), guaranteeing zero resurrected points across restarts and crashes.
🚀 Hardware Performance Engine (v4.1.0)	Micro-architectural hardware acceleration layer. Features Dynamic Runtime SIMD Dispatch selecting between AVX-512F (16xf32), AVX2+FMA (8xf32), and ARM NEON (4xf32) with unaligned loads and early-exit thresholding; Zero-Copy Inner Distance Kernels (distancebytes*) eliminating up to 50,000 heap allocations per second during HNSW graph descent; Linux io_uring Batched Positional I/O for single-syscall NVMe batch gathers; Multi-Socket NUMA-Aware Topology pinning tokio worker threads per socket for first-touch memory bus locality; and a 32-Stripe Sharded Payload Index eliminating lock contention.
⚙️ Reflex-Level Speed	Built on Nightly Rust. Our ArcSwap Lock-Free architecture and f32 SIMD intrinsics deliver 142.5 QPS (P50 6.98 ms, P99 7.84 ms) on 25k × 1024D dense search at 99.90% Recall@10, and up to 1,567 QPS with OPQ/PQ quantization and 1,774 QPS on Spatial AABB bounding queries. Full index construction completed in 78.54 s (43.45 s faster than v3). See BENCHMARKV3VSV4REPORT.md.
🔑 RBAC & Cross-Tenant Sharing	Granular Role-Based Access Control (Admin, ReadWrite, ReadOnly) using transaction-safe redb security storage. Support for namespaced URL collection referencing (owner:name or owner/name) for secure cross-tenant index sharing.
🕵️‍♂️ Audit Logs & Isolation	High-performance structured JSON audit logging (off, low, medium, high, full levels) streaming to stdout or files. Real-time, tenant-isolated log buffers stream secure log feeds directly to the Web Dashboard.
🌿 Eco-Monitoring (EcoMonitor)	Telemetry agent tracking carbon footprints of CPU/RAM computations per node. Visualizes ESG sustainability metrics and environmental compliance certification badges inside the dashboard UI.
⚡ L0 Hot Tier Cache	Transparent two-level vector caching. Combines a thread-safe 64-shard L1 DashMap Cache (~1 µs lookups) with an L2 HNSW Fallback Graph (~100 µs lookups), featuring background TTL invalidation and automatic self-healing graph rebuilds.
💾 O(1) Payload Indexing	High-throughput O(1) incremental indexing via append-only index write-ahead logging (WAL) in payload_store.rs, eliminating latency spikes and write amplification.
🔐 Zero-Knowledge Client Encryption & Vec2Text Defense (v4.1.1)	Client-side isometric orthogonal $O(n)$ and Lorentz $SO^+(1, d)$ projections with calibrated anisotropic noise injection ($sigma le 0.02$). Neutralizes Vec2Text neural inversion attacks while preserving 100.0% Top-K neighbor recall across all 6 metrics (L2, Cosine, Dot, Lorentz, Poincaré, Hybrid). Raw text and payload attributes encrypted with AES-256-GCM. Surfaced as an ENCRYPTED badge in the Dashboard.
🧠 Autonomous Cognitive Agent Memory (v4.1.1)	Native sub-4ms episodic memory layer for autonomous agents (Python, TypeScript, MCP). Combines Ebbinghaus Exponential Temporal Decay ($S(t) = S_0 cdot e^{-lambda Delta t}$) integrated directly into vector scoring with zero LLM calls, and autonomous Temporal Superseding (directed SUPERSEDES edges) to instantly obsolete contradictory facts without external graph databases or secondary LLM consolidation passes.
🤖 Cognitive MCP & Zero-RAM Sidecar	Native Model Context Protocol (MCP) server for IDE agents (Claude, Cursor, Antigravity) with 6 cognitive tools. Zero-RAM Sidecar Payload streaming for LangChain and LlamaIndex to keep RAM lightweight.
📝 Real-Time WriteBuffer	Zero-latency searchability. Inserts instantly write to an in-memory WriteBuffer; searches execute Rayon-parallelized linear scans on the buffer and merge/deduplicate results with main HNSW results.
🧭 Schema-Driven Cascade	Native MRL (Matryoshka) support. Define one schema with multiple layers; the RAM-resident CascadePipeline queries truncated dimensions in microseconds, automatically reranking results via full-resolution vectors on NVMe/S3.
🎓 Cognitive Math Engine	First-class HNSW support for Euclidean (L2/Cosine), Poincaré Ball, Lorentz Hyperboloid metrics, and Wasserstein O(N) CFM. * Implicit Graph Engine (v3.1): Stop hiding HNSW topology. Expose raw adjacency and navigate via Lorentz Light-Cones and Wave Diffusion (resonant semantic propagation). * Graph Explorer (v3.1): High-performance canvas-based visualization of HNSW topology. Features Poincaré/Euclidean manifold projections, Momentum Path Tracing, and Lyapunov Stability Radar. * Relativistic Traversal: Use Momentum Traversal (Dirac) to swing through nodes with semantic inertia, predicting trajectories via Koopman extrapolations. * Advanced Geometric Filters: Native InBall, InBox, and InCone filters in core engine. Execute spatial K-Means, Fréchet Mean, and Parallel Transport directly in the Native SDK. Evaluate datasets via Gromov’s Delta-Hyperbolicity.
📡 Agentic Workflows	Trigger Memory Reconsolidation via Flow Matching natively to shift paradigms. Connect CDC Streams via subscribetoevents to trigger secondary models. Hybrid Search (BM25) now supports RRF fusion for state-of-the-art lexical + semantic retrieval.
🧹 Metadata-Driven Pruning	Agents must forget to stay efficient. Use typed numeric Range Filters (energy < 0.1) inside a Hot Vacuum to automatically prune obsolete memories.
📦 LSM-Tree Storage	Optimized for high-concurrency writes. Hot MemTables continuously flush into immutable Fractal Segments (chunk_N.hyp), enabling near-instant RAM reclamation and stable performance at billion-scale.
☁️ S3 Cloud Tiering	Native S3/MinIO tiered storage integration. Seamlessly offload cold segments mapping Petabytes of vectors linearly without scaling local SSDs. (Unlock via Cargo feature s3-tiering & HSSTORAGEBACKEND=s3).
🔒 Zero-Knowledge ZK-Privacy	Protect sensitive agent memories and database operations. Supports multi-SDK Zero-Knowledge client-side encryption. Vectors are projected, noise-injected, metadata is hashed, and payloads are AES-GCM encrypted before leaving the client, keeping database nodes blind to raw values while preserving distance metrics.
⛏️ DePIN & Billing Enforcements	Decentralized Vector Network infrastructure. Includes in-process single-binary node execution, continuous storage rental billing (per byte-hour) with automated grace periods/deletion, signed ticket verification (Ed25519) on query time with replay protection, and user-throttling.

---

🕸️ Native Vector-Graph (KAG): Why HyperspaceDB vs The Market?

Traditional vector databases treat documents as isolated points in Euclidean space. In complex regulatory, cybersecurity, aerospace, and medical domains, pure vector search fails to retrieve 77.9% of structural prerequisites and causal containment steps.

HyperspaceDB solves this by unifying HNSW vector indexing and a Property Compressed Sparse Row (CSR) Graph within the same shared-memory address space and L1/L2 cache line.

> [!IMPORTANT]
> ### 🏆 Empirical Proof Across 12 Production Playbook Collections (16,001 Vectors, 18,402 Edges):
> * Causal Context Completeness: 100.0% Recall with KAG vs 22.1% in Pure Vector Search.
> * Zero Lost Steps: Pure vector search misses up to 93.8% of containment steps (e.g. in SOC MITRE TTPs and SEC capital rules). KAG deterministically captures every prerequisite.
> * Sub-Millisecond Graph Traversal: ~5 µs native CSR bitset traversal at the CPU core, 2.36 ms (p50) end-to-end over HTTP.
> * Total Decision Latency: 3.37 ms (p50) combining vector entry match + multi-hop graph reachability.

⚔️ Comprehensive Competitive Matrix

| Capability / Metric | HyperspaceDB v4.1 (KAG) | Microsoft GraphRAG | Frankenstein Stack (Vector + Neo4j + Postgres) | Graph DBs with Vectors (Neo4j Vector) | Pure Vector DBs (Pinecone / Qdrant) |
| :--- | :---: | :---: | :---: | :---: | :---: |
| System Architecture | Unified Single-Binary Thread-per-Core LSM | Python pipeline script on top of Parquet files | 3 separate distributed database clusters | Single JVM cluster with Lucene plugin | Single vector cluster with flat metadata |
| Edge Storage Format | 12-byte packed CSR in RAM/SSD (graph.csr) | Parquet text tables & graphs | Neo4j pointer lists + JVM Heap | Lucene inverted lists + JVM Heap | No edges (flat key-value metadata) |
| Graph Ingestion Cost | $0 (Zero LLM token costs) | $100 – $1,000+ per dataset in LLM prompts | $0 (Manual Cypher ETL scripts) | $0 (Manual Cypher ETL scripts) | N/A (Cannot represent graphs) |
| Graph Creation Time | 2 – 15 milliseconds | Hours to days | Minutes to hours | Minutes to hours | N/A |
| End-to-End Decision Latency | 3.37 ms | 5 – 30+ seconds | 40 – 120 ms (Network hop tax) | 20 – 50 ms | 5 – 15 ms (Isolated vectors only) |
| Graph Traversal Speed | ~5 µs (L1/L2 cache hit) | N/A (LLM community summary) | 15 – 40 ms (Cypher query parser) | 5 – 15 ms | N/A (No topological traversal) |
| Causal Context Recall | 100.0% (Complete) | ~85–95% (LLM non-deterministic) | 100.0% (If manual links exist) | 100.0% (If manual links exist) | 22.1% (Severe 77.9% blind spots) |
| Hyperbolic Geometry | Native 33D Lorentz & Poincaré Ball | ❌ None | ❌ None | ❌ None | ❌ None |
| Real-Time Streaming Writes | Instant (LSM WriteBuffer + DynamicGraph) | ❌ Batch-only (Recomputes Leiden communities) | ⚠️ Complex (Distributed 2PC sync) | ⚠️ Slow vector batch updates | ✅ Fast vector writes |
| Hardware Memory Footprint | ~1 MB RAM per 20,000 edges | High (Pandas/Parquet memory) | Extremely High (JVM + Go + Postgres) | High (JVM Heap + Off-Heap) | Moderate |

---

⚡ L0 Hot Tier Caching & WriteBuffer (Real-Time Ingestion)

To maximize read throughput and achieve true real-time ingestion, HyperspaceDB implements a transparent, dual-component memory tier:

graph TD
    Request[Incoming Request] --> Action{Write or Search?}
    
    Action -->|Insert| WB[WriteBuffer in RAM]
    WB --> WAL[WAL Log]
    WB --> IndexQueue[Active Indexing Queue]
    
    Action -->|Search| CacheCheck{L0 Cache Hit?}
    CacheCheck -->|Yes L1/L2 Hit| FastOut[Instant Result ~1 µs - 100 µs]
    CacheCheck -->|No Cache Miss| SearchMerge[Merged Scatter-Gather Search]
    SearchMerge --> HNSWS[HNSW Index search]
    SearchMerge --> WBS[WriteBuffer Rayon Parallel Scan]
    HNSWS --> Deduplicate[Deduplicate by external ID & Re-rank]
    WBS --> Deduplicate

1. Transparent L0 Hot Tier Cache (hyperspace-cache)

The L0 Cache resides directly on the read path, enabling blistering microsecond-level vector retrievals:

• L1 exact-match Cache: Uses a 64-shard thread-safe DashMap for key-value point lookups. Latency is an imperceptible ~1 µs.
• L2 approximate-match Cache: Uses an in-memory HNSW graph (instant-distance) with ArcSwap for atomic lock-free reads during background updates. Latency is ~100 µs.
• Time-To-Live (TTL): Automatic scanning of the __ttl metadata field every 100 ms. Expired keys are invalidated, and tombstones are added to L2.
• Auto-rebuild and Cooldown: When L2 tombstones exceed 30%, a self-healing background task rebuilds the L2 graph, protected by a 5-second cooldown to prevent CPU-rebuild thrashing.
• Metadata Warmup: Rebuilding or starting up automatically warmloads key-value metadata from metadata.forward, ensuring that payloads are instantly available upon cache hits.

2. Real-Time WriteBuffer

A lock-free in-memory buffer ensuring newly inserted vectors are searchable from the very first millisecond:

• Instant Searchability: Newly inserted vectors are placed in the WriteBuffer while HNSW graph links are built in the background.
• Rayon-Parallel Linear Scan: Queries perform multi-threaded linear scans over the WriteBuffer utilizing AVX2/NEON vector instructions, computing Euclidean/Hyperbolic distances and running complex box/cone region filters in < 2.0 ms for up to 50,000 active entries.
• Deduplicated Merging: Results from the main HNSW graph and the WriteBuffer are merged, sorted, and deduplicated by external ID to return highly accurate, real-time lists.
• Snapshot Promotion: Upon HNSW graph integration, background workers evict indexed entries from the WriteBuffer, keeping the RAM footprint minimal.

📊 L0 Cache & Ingestion Performance Results

A rigorous 22-test validation suite confirms the stability and performance of the memory acceleration layer:

• Cache-Hit Ratio: Achieves >98% Recall accuracy on recurrent L1 lookups, with a zero-copy fast path for de-quantized floats.
• rebuild_cooldown Verification: Successfully throttles back-to-back index updates, queuing pending vectors correctly.
• tombstone_trigger Auto-rebuild: Automatically resolves up to 5,000 concurrent deletions, reclaiming memory map segments with zero leaks.
• WriteBuffer linear scan: Demonstrates Rayon multi-threaded lookup times under 1.8 ms for bulk vector batches.

🤖 Target Use Cases

1. Robotics & Autonomous Drones: On-device semantic memory, Hierarchical SLAM, and offline-first edge synchronization.
2. Continuous Learning Systems (AGI): Frameworks doing Riemannian optimization, memory reconsolidation, and Hausdorff-based graph pruning.
3. Enterprise Graph AI: Merging relational logic with semantic proximity for massive multi-scale data analysis (Code ASTs, Medical Taxonomies).
4. High-Load RAG & SaaS: Traditional search, but significantly cheaper to operate due to Serverless Idle Eviction and multi-tenant isolation.

---

⚡ 1 Million Vectors Benchmark (v3.0)

We pushed HyperspaceDB v3.0 to the limit with a 1 Million Vector Dataset.
The results define a new standard for performance and efficiency.

🏆 Hyperbolic Efficiency (Poincaré 64d)

When using the native Hyperbolic (Poincaré) metric, HyperspaceDB achieves unparalleled throughput by reducing dimensionality (64d) while preserving semantic structure achievable only with 1024d in Euclidean space.

| Metric | Result | vs Euclidean |
| :--- | :--- | :--- |
| Throughput | 156,587 QPS ⚡ | 8.8x Faster |
| P99 Latency | 2.47 ms | 3.3x Lower |
| Disk Usage | 687 MB | 13x Smaller |

⚔️ Euclidean Performance (1024d)

Even in standard Euclidean mode, HyperspaceDB outperforms competitors on standard hardware.

| Database | Total Time (1M vectors) | Speedup Factor |
| :--- | :--- | :--- |
| HyperspaceDB | 56.4s ⚡ | 1x |
| Milvus | 88.7s | 1.6x slower |
| Qdrant | 629.4s (10m 29s) | 11.1x slower |
| Weaviate | 2036.3s (33m 56s) | 36.1x slower |

📉 Zero Degradation Architecture

While other databases slow down as data grows, HyperspaceDB maintains consistent throughput.

• Weaviate degraded from 738 QPS -> 491 QPS (-33%).
• Milvus fluctuated between 6k and 11k QPS.
• HyperspaceDB held steady at ~156k QPS (Hyperbolic) and ~17.8k QPS (Euclidean).

💾 50% Less Disk Usage

Store more, pay less. HyperspaceDB's 1-bit quantization and efficient storage engine require half the disk space of Milvus for the exact same dataset.

• HyperspaceDB: 9.0 GB (Euclidean) / 0.7 GB (Hyperbolic)
• Milvus: 18.5 GB

> Benchmark Config: 1M Vectors, 1024 Dimensions (Euclidean) vs 64 Dimensions (Hyperbolic), Batch Size 1000.

---

⚡ v4.0.0 Grand Benchmark: Thread-per-Core & Direct I/O

HyperspaceDB v4.0.0 introduces a high-performance Thread-per-Core (Shared-Nothing) architecture with Hardware Direct I/O (4096-byte DMA alignment with ODIRECT/FNOCACHE), hierarchical Spatial AABB segment pruning, and autonomous startup migration.

🏆 Empirical VectorDBBench Verification (v3.x vs v4.0.0)

| Scenario / Workload | v3.x Baseline | v4.0.0 Engine | Delta / Improvement | Key Architecture Enabler |
| :--- | :---: | :---: | :---: | :--- |
| Dense 1024D (100k Vectors) | 5.7 QPS | 60.7 QPS | 10.6× faster | Hardware L1 Prefetching + SIMD f32 unrolling |
| • Tail Latency (P99) | 648.79 ms | 19.91 ms | 32.6× lower | 64-bit monotonic VisitedScratch tokens |
| MRL Cascade 1536D (50k Vectors) | 3.7 QPS | 81.8 QPS | 22.3× faster | Direct I/O NVMe DMA without page cache locks |
| • Tail Latency (P99) | 1,010.30 ms | 14.49 ms | 69.8× lower | 64D RAM Head + Page-aligned Direct disk streaming |
| Spatial Bounded Search (768D, 50k) | 4.4 QPS | 96.6 QPS | 22.1× faster | Hierarchical Spatial AABB segment pruning |
| • InBox Filter P99 Latency | 78.41 ms | 0.66 ms | 118.8× lower | >95% non-matching vectors pruned in RAM |
| Concurrency Saturation (C=100) | 30,056 ms | 2,678 ms | 11.2× stability gain| Eliminated cross-core lock contention (0 ns) |

> 📌 Detailed comparative data available in BENCHMARKV3VSV4REPORT.md and BENCHMARKGRANDREPORT.md.

---

🗜️ Multi-Tier Vector Quantization Suite (v4.0.0)

HyperspaceDB v4.0.0 supports 7 quantization precision levels verified empirically across 20,000 unit-normalized 1024D embeddings (see BENCHMARKQUANTIZATIONSWEEP.md):

| Mode | Key | Bytes / Vec | Compression | Single-Pass Recall@10 | Two-Pass Raw Rerank | Search QPS | Mean Latency | Primary Use Case |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :--- |
| Full Precision | none | 4,100 B | 1.0× | 100.00% | Baseline | 203 QPS | 4.91 ms | Legal, Medical, Financial Ground Truth |
| Scalar SQ8 | medium | 1,028 B | 4.0× | 88.60% | 88.60% | 220 QPS | 4.52 ms | Enterprise RAG standard (75% RAM savings) |
| Block ScalarI4 | medium_plus | 772 B | 5.3× | 87.15% | 87.15% | 287 QPS | 3.47 ms | High-throughput microservices (Fastest) |
| TurboQuant | turbo | 520 B | 7.9× | 86.95% | 99.95% (k=40) | 218 QPS | 4.57 ms | Spherical Lloyd-Max + Orthogonal Rotation |
| Optimized PQ (OPQ-64) | opq | 68 B | 60.3× | 17.00% (73.4% rk4) | 99.85% (k=40) | 1,567 QPS | 0.63 ms | Ultra-Scale ADC with FWHT rotation |
| Product Quant (PQ-64) | pq | 68 B | 60.3× | 17.20% (73.1% rk4) | 99.80% (k=40) | 1,453 QPS | 0.67 ms | Standard Product Quantization with ADC |
| Extreme 1-Bit ADC| extreme | 132 B | 31.1× | 35.75% | 75.45% (k=80) | 113 QPS | 8.85 ms | Edge, Robotics (ROS 2), IoT (< 132 MB / 1M vecs) |

---

🔒 Security

• API Keys: Secure endpoints with HYPERSPACEAPIKEY environment variable.
• Header: Clients must send x-api-key: .
• Zero-Knowledge: Server stores only SHA-256 hash of the key in memory.

🤝 Federated Clustering & P2P Swarm (v3.0)

HyperspaceDB implements two distinct clustering architectures designed for both high availability in the Cloud and dynamic Edge-to-Edge discovery for robotics swarms.

1. Leader-Follower Replication (Cloud / High Availability)

• Node Identity: Each node generates a unique UUID (node_id) and maintains a Lamport logical clock.
• Leader: Handles Writes (Coordinator). Streams WAL events. Manages Cluster Topology.
• Follower: Read-Only replica. Can be promoted to Leader.

2. Edge-to-Edge Gossip Swarm (Robotics / Local-First)

Designed for robotic swarms without a central Leader. Uses raw UDP multicasting to form a decentralized, self-healing network.

• Zero-Dependency: Built on raw tokio::net::UdpSocket (no heavy libp2p dependencies).
• Heartbeats: Nodes broadcast state via UDP. Disconnected nodes are automatically evicted after a TTL interval.
• Auto-Discovery: Discover peers and instantly initiate a Delta Sync handshake to resolve diverging graphs.
• Enable: Set HSGOSSIPPEERS (e.g. 192.168.1.10:7946) or HSGOSSIPPORT to join the swarm.

Data Synchronization (Edge-to-Cloud Delta Sync)

HyperspaceDB uses a 256-bucket Merkle Tree for efficient data drift detection, ideal for WASM/Edge targets updating offline:

• Granular Hashing: Each collection is partitioned into 256 buckets (by vector ID % 256)
• XOR Rolling Hash: Each bucket maintains an incremental hash of its vectors
• Fast Diffing: Compare bucket hashes to identify which partition is out of sync
• Bandwidth Optimization: Sync only affected buckets instead of full collection

#### WASM Sync Example
When your robot or web client comes back online, initiating a Sync is mathematically minimal:

// 1. Handshake: Send local 256 bucket hashes
const { diffBuckets } = await client.syncHandshake(collection, localBuckets);
if (diffBuckets.length > 0) {
    // 2. Pull only the modified/missing buckets from Cloud
    const stream = client.syncPull(collection, diffBuckets);
    stream.on('data', (vectorData) => applyLocal(vectorData));
    
    // 3. Push local offline edits back to Cloud
    client.syncPush(localEditsQueue);
}

#### Digest API

HTTP
GET /api/collections/{name}/digest
gRPC
rpc GetDigest(DigestRequest) returns (DigestResponse)

Response includes:

• logical_clock: Lamport timestamp
• state_hash: Root hash (XOR of all buckets)
• buckets: Array of 256 bucket hashes
• count: Total vector count

Cluster Topology API

View the logic state of the cluster via HTTP:

Get Replication State
curl http://localhost:50050/api/cluster/status
Get Decentralized Swarm Peers (Gossip)
curl http://localhost:50050/api/swarm/peers

{
  "gossip_enabled": true,
  "peer_count": 2,
  "peers": [
    {
       "node_id": "e8...0e",
       "role": "Leader",
       "addr": "192.168.1.20:50050",
       "logical_clock": 42,
       "healthy": true
    }
  ]
}

Starting a Cluster

Start Leader
./hyperspace-server --port 50051 --role leader
Start Follower
./hyperspace-server --port 50052 --role follower --leader http://127.0.0.1:50051

🕸️ WebAssembly (WASM) Support

HyperspaceDB can run directly in the browser via WebAssembly, enabling Local-First AI applications with zero network latency.

• Zero Latency: Search runs in-memory on the client.
• Privacy: Data never leaves the device.
• Optimized: Uses RAMVectorStore backend for browser environments.

👉 Read the WASM Documentation

⚖️ Heterogeneous Tribunal Framework (Tribunal Router)

HyperspaceDB natively supports the confrontational model of LLM routing (Architect vs. Tribunal) directly on the vector graph.

Using the Cognitive Math SDK and the Graph Traversal API, the SDK calculates a Geometric Trust Score for any LLM claim by verifying the logical path length between concepts in the latent hyperbolic space.

If the geodesic distance (hops) between "Claim A" and "Claim B" on the graph is too large (or disconnected), the Trust Score drops to 0.0 (Hallucination).

from hyperspace.agents import TribunalContext
tribunal = TribunalContext(client, collectionname="knowledgegraph")
Evaluates structural graph distance between concepts. 
1.0 = Truth (Identical), 0.0 = Hallucination (Disconnected)
score = tribunal.evaluateclaim(conceptaid=12, conceptb_id=45)

🧠 Hybrid Search (BM25 + RRF)

Combine the power of Hyperbolic Embeddings with advanced BM25 lexical ranking.

Search for semantic similarity AND keyword match (e.g. "iPhone 15")
results = client.search(
    vector=[0.1]*8, 
    top_k=5, 
    hybrid_query="iPhone 15", 
    hybrid_alpha=0.7,  # 70% vector, 30% lexical
    bm25={
        "method": "bm25plus",
        "language": "english"
    }
)

📊 Quantization Modes

All quantization is configured per-collection at creation time via HSQUANTIZATIONLEVEL:

| Mode | Value (quantization) | Bytes/vec (1024D) | Compression | Supported Metrics | Best for |
|---|---|---|---|---|---|
| SQ8 Anisotropic | medium / scalar | 1,028 B | 4× (vs f32) / 8× (vs f64) | Cosine, L2, Poincaré, Lorentz | General RAG / Best default recall at 1 byte/dim |
| Block ScalarI4 | medium_plus | 772 B | 5.3× | Cosine, L2 | High-throughput services (287 QPS) |
| TurboQuant 4-bit | turbo | 520 B | 7.9× | Cosine, L2 | Spherical embeddings with Lloyd-Max bias correction |
| Optimized PQ (OPQ-64) | opq | 68 B | 60.3×–64× | Cosine, L2 (Euclidean tail in Hybrid) | Massive scale with FWHT-rotated ADC table lookups |
| Product Quant (PQ-64) | pq | 68 B | 60.3×–64× | Cosine, L2 | Standard Subspace Product Quantization ADC |
| Binary ADC (1-bit) | extreme / binary | 132 B | 31.1× (vs f32) / 62× (vs f64) | Cosine, L2 | RAM-critical edge/robotics deployment |
| Lorentz SQ8 | automatic | 1,028 B | 4× / 8× | Lorentz metric only | Hyperbolic embeddings with Minkowski time sign preservation |
| Full Precision | none | 4,100 B | 1× | All metrics | Exact ground truth, medical, legal compliance |

> ⚠️ Metric Compatibility Notice:
> - Cosine & L2: Compatible with ALL quantization modes (none, medium, medium_plus, turbo, opq, pq, extreme).
> - Poincaré & Lorentz: Only compatible with none and medium (Scalar SQ8). Product Quantization (pq/opq) and 1-bit (extreme) assume Euclidean metrics and orthonormal rotations that violate conformal Poincaré boundary conditions ($|x| < 1$) and Lorentzian pseudo-Riemannian signature $(-1, +1, dots, +1)$. > - Hybrid (801D = 33D Lorentz + 768D Euclidean): Uses AsymmetricHybrid, where the 33D Lorentz head is kept unquantized/FP64 while the 768D Euclidean tail can be compressed with turbo, opq, or extreme.

The default scalar mode uses a ScaNN-inspired anisotropic loss $L = |eparallel|^2 + tw cdot |eperp|^2$ ($tw=10$), which penalizes directional error more than magnitude error, improving Recall@10 by +5.3% (Cosine) and +3.8% (L2) versus isotropic SQ8.

---

Each geometry has its own independent backend, now featuring native support for Qwen3-Embedding (0.6B) and YAR v5 models:

| Provider | Description | Recommended For |
|---|---|---|
| Local ONNX | Any .onnx model from disk | Air-gapped / Edge |
| HuggingFace | Auto-download, cache, and chunk | High accuracy / Long context |
| Remote API | Mistral / OpenAI / Cohere / Voyage | Cloud API offload |

Example — Config for Qwen3 (1024d) and YAR v5 (128d):

HYPERSPACE_EMBED=true
Cosine via Qwen3 (HuggingFace)
HSEMBEDCOSINE_PROVIDER=huggingface
HSEMBEDCOSINEHFMODEL_ID=onnx-community/Qwen3-Embedding-0.6B-ONNX
HSEMBEDCOSINE_DIM=1024
HSEMBEDCOSINECHUNKSIZE=4096   # 32K context window
Poincaré via YAR Labs v5
HSEMBEDPOINCARE_PROVIDER=huggingface
HSEMBEDPOINCAREHFMODELID=YARlabs/v5Embedding_0.5B
HSEMBEDPOINCARE_DIM=128

Direct Search from SDK:

Server-side text-to-vector search (v3.0.1)
results = client.searchtext("Find similar robotics docs", topk=5)

For details see embeddings.md.

---

📉 Binary Quantization (1-bit)

Use Binary quantization mode to compress vectors by 32x-64x (vs f32/f64).
Ideal for large-scale datasets where memory is the bottleneck.

---

🛠 Architecture

HyperspaceDB strictly follows a Command-Query Separation (CQS) pattern:

graph TD
    Client["Client (gRPC)"] -->|Insert| S["Server Service"]
    Client -->|Search| S
    
    subgraph Persistence_Layer ["Persistence Layer"]
        S -->|"1. Append"| WAL["Write-Ahead Log"]
        S -->|"2. Append"| VS["Vector Store (mmap)"]
    end
    
    subgraph Indexing_Layer ["Indexing Layer"]
        S -->|"3. Send ID"| Q["Async Queue"]
        Q -->|Pop| W["Indexer Worker"]
        W -->|Update| HNSW["HNSW Graph (RAM)"]
    end

1. Transport: gRPC/Tonic server accepts requests (Insert/Search).
2. Persistence: Data is immediately persisted to WAL and segmented Mmap storage.
3. Indexing: A background worker updates the HNSW graph asynchronously.
4. Recovery: Graph snapshots (via rkyv zero-copy) ensure near-instant restarts.

👉 For deep dive, read ARCHITECTURE.md

---

## 🛠 Operations & Maintenance

### Queue Monitoring
Check ingestion backlog via API or collections stats:

 {
   "count": 150000,
   "indexing_queue": 45  // Items pending index insertion
 }
 

### Rebuild Index (Defragmentation)
Trigger a graph rebuild to optimize layout and remove deleted nodes:

 curl -X POST http://localhost:50050/api/collections/my_col/rebuild
 

### Memory Management (Jemalloc)
HyperspaceDB uses Jemalloc for efficient memory allocation. You can tune its behavior via the MALLOC_CONF environment variable:

* Low RAM (Aggressive Release): MALLOCCONF=backgroundthread:true,dirtydecayms:0,muzzydecayms:0 - Releases unused memory immediately to OS. Increases CPU usage slightly.
* Balanced (Default): MALLOCCONF=backgroundthread:true,dirtydecayms:5000,muzzydecayms:5000 - Keeps some memory for reuse, balanced performance.

To create a manual memory vacuum request (e.g., after large deletions):

 curl -X POST http://localhost:50050/api/admin/vacuum
 

---

💻 System Requirements

HyperspaceDB is designed to run efficiently on commodity hardware, but specific instruction sets are required for hardware acceleration.

CPU (Critical)

• Architecture: x86-64 or ARM64.
• Instructions:
• x86-64: Must support AVX2 (Intel Haswell 2013+ or AMD Zen 2017+).
• ARM64: Must support NEON (Standard on Apple Silicon M1/M2/M3 and AWS Graviton).

Note: The database will crash or fail to compile on CPUs without SIMD support.*

Storage (I/O)

• Disk Type: SSD / NVMe is highly recommended.
• HyperspaceDB uses mmap for random access. Spinning HDDs (mechanical drives) will severely degrade search latency due to seek times.

Memory (RAM)

• Minimum: 512 MB.
• Recommended: Enough RAM to cache the "hot" part of your dataset.
• Thanks to ScalarI8 quantization, 1 Million vectors (8-dim) take only ~12 MB of disk space. Even large datasets fit easily into RAM.
• If the dataset exceeds RAM, the OS will swap pages to disk (performance will depend on SSD speed).

Operating System

• Linux: Kernel 5.10+ recommended (for efficient memory mapping).
• macOS: 12.0+ (fully supported).
• Windows: Supported via WSL2 (native Windows build is experimental).

---

🏃 Quick Start

1. Build and Start Server

Make sure you have just and nightly rust installed.

Build release binary
cargo build --release
Run server (Default HTTP port: 50050)
./target/release/hyperspace-server
Or with custom ports
./target/release/hyperspace-server --port 50051 --http-port 50050

2. Access Web Dashboard

The built-in React Dashboard provides real-time monitoring and management:

http://localhost:50050

Dashboard Features:

• 📊 System Overview: Real-time metrics (RAM, CPU, vector count)
• 🗂️ Collections Manager: Create, delete, and inspect collections
• �️ Cluster Nodes: Visualize node topology and replication status
• �🔍 Data Explorer: View recent vectors and test search queries
• ⚙️ Settings: Integration snippets (Python, cURL) and live logs
• 📈 Graph Explorer: Visualize HNSW graph structure, Poincaré manifolds, and agent momentum trajectories in real-time.

Authentication:
If HYPERSPACEAPIKEY is set, you'll be prompted to enter it on first visit. The key is stored in localStorage for subsequent sessions.

Build Dashboard from Source:

cd dashboard
npm install
npm run build
Assets are embedded in Rust binary via rust-embed

3. Launch TUI Monitor

Open a new terminal to monitor the database:

./target/release/hyperspace-cli

3. Use Python SDK

pip install hyperspacedb==3.1.2

from hyperspace import HyperspaceClient
Connect to local instance
client = HyperspaceClient()
Create a collection with Schema-Driven Cascade (128D Poincaré + MRL)
client.create_collection(
    name="world_model", 
    schema={
        "components": [
            {"name": "primary", "metric": "poincare", "full_dimension": 128, "weight": 1.0}
        ],
        "cascade_pipeline": [
            {"componentname": "primary", "cutoffdimension": 32, "storeinram": True, "reranktopk": 100}
        ]
    }
)
Insert text document (you can provide your own embeddings)
client.insert(id=1, collection="world_model", document="Hyperspace is autonomous.")
Search 
results = client.search(querytext="autonomous engine", topk=5)
print(results)

🏘️ Collections Management

HyperspaceDB v1.1+ supports Multi-Tenancy via Collections. Each collection is an independent vector index with its own dimension and metric.

Via Web Dashboard

Access the dashboard at http://localhost:50050:

1. Create Collection: Enter name, select dimension (8D, 768D, 1024D, 1536D), click Create
2. View Collections: See all active collections with their stats
3. Delete Collection: Remove collections you no longer need

Via gRPC/SDK

from hyperspace import HyperspaceClient
client = HyperspaceClient()
Create a new collection with Schema
client.create_collection(
    name="my_vectors", 
    schema={
        "components": [
            {"name": "main", "metric": "cosine", "full_dimension": 1536}
        ]
    }
)
Insert into specific collection
client.insert(id=1, document="...", collection="my_vectors")
Search in specific collection
results = client.search(querytext="...", collection="myvectors", top_k=5)
List all collections
collections = client.list_collections()
Delete a collection
client.deletecollection("myvectors")

Note: If no collection is specified, operations default to the "default" collection.

🏙️ SaaS & Multi-Tenancy (v2.0)

HyperspaceDB is built for SaaS. Isolate thousands of users on a single node.

1. User Isolation

Data is logically separated by user_id. Each user sees only their own collections.

How to use:
Pass the x-hyperspace-user-id header in your requests.

curl -H "x-hyperspace-user-id: tenant_123" http://localhost:50050/api/collections

2. Billing API

Admins can query usage statistics for all tenants:

curl -H "x-hyperspace-user-id: admin" http://localhost:50050/api/admin/usage

⚙️ Configuration & Presets

HyperspaceDB v1.2 introduces flexible configuration presets to support both Scientific (Hyperbolic) and Classic (Euclidean) use cases.

Configure these via .env file or environment variables:

| Variable | Description | Supported Values | Default |
| :--- | :--- | :--- | :--- |
| HSQUANTIZATIONLEVEL | Compression | scalar (i8), binary (1-bit), turbo (4-bit), none (f64) | none |
| HSSHARDS | Number of Thread-per-core CPU shards | Integer (1 to numcpus * 2) | Logical CPU core count |
| HSDIRECTIO | Enable Direct I/O (ODIRECT / FNOCACHE) | 1 (true), 0 (false) | 1 |

> [!IMPORTANT]
> Since v3.1.0, HSDIMENSION and HSMETRIC are deprecated. Use CollectionSchema via SDK or CLI for granular control.

🔄 Инструкция по автоматической миграции (v3.x → v4.0.0 Migration Guide)

В версии v4.0.0 реализована раздельная архитектура хранения данных:

• data/: Папка legacy-данных v3.x.
• data_v4/: Папка нового формата v4.0.0 (Direct I/O, Thread-per-core, Spatial AABB).

#### Как работает миграция (Zero-Config Upgrade):
1. Автоматический запуск: При первом старте HyperspaceDB v4.0.0 сервер автоматически обнаруживает папку data/.
2. Бесшовная конвертация: Все найденные v3-коллекции конвертируются в формат v4.0.0 и записываются в datav4/{collectionname}/:
- Векторы выравниваются по границам страниц 4096 байт для прямого DMA-доступа (Direct I/O).
- Автоматически рассчитываются и сохраняются пространственные сегменты Spatial AABB.
- Таблица безопасности и прав доступа (data/security.db) автоматически переносится в data_v4/security.db.
3. Безопасное архивирование: Исходные v3 файлы атомарно перемещаются в data/legacyv3migrated/{collectionname}/ с созданием манифеста целостности .migrationv4_complete.json (с таймштампом и количеством векторов).
4. Повторные / ручные переносы: Если в будущем пользователь скопирует старые v3 файлы или целую коллекцию в data/ (или data/legacyv3/), система при следующем перезапуске автоматически обнаружит их и смигрирует в datav4/.

#### Настройка путей через переменные окружения:

• HSDATADIR: Путь к активному v4 хранилищу (по умолчанию: data_v4).
• HSLEGACYV3_DIR: Путь к legacy v3 хранилищу для поиска миграций (по умолчанию: data).

#### Проверка успешности миграции:

• В логах сервера отображаются записи:

  🔄 [MigrationEngine] Found 1 legacy v3.x collection(s) to migrate to v4.0.0 (from data to data_v4)
  🚀 [MigrationEngine] Migrating 'mycollection' from data/mycollection to v4 format in data_v4...
  ✅ [MigrationEngine] Successfully migrated 'my_collection' (50000 vectors in 142 ms)
  📦 [MigrationEngine] Archived legacy files to data/legacyv3migrated/my_collection
  

• В папке каждой коллекции datav4/{collection}/ создается файл .migrationv4_complete.json.

🎯 Supported Presets

1. Classic RAG (Default)
Optimized for standard embeddings from OpenAI, Cohere, Voyage, etc.

• Metric: cosine (Cosine Similarity) - recommended for OpenAI/BGE embeddings
• Dimensions: 1024, 1536, 2048
• Note: cosine mode automatically normalizes vectors on insert/search (with zero-copy fast path for already normalized vectors) and uses HNSW-friendly squared L2 ranking internally. For magnitude-sensitive workloads, use l2 with HSQUANTIZATIONLEVEL=none.

2. Scientific / Hyperbolic
Optimized for hierarchical data, graph embeddings, and low-dimensional efficiency.

• Metric: poincare
• Dimensions: 16, 32, 64, 128 (Common: 64)
• Requirement: Input vectors must strictly satisfy ||x|| < 1.0 (Poincaré ball constraint). Server will reject invalid vectors.

---

📊 Best Practices

HyperspaceDB follows the microservices philosophy: One Index per Instance. To manage multiple datasets, we recommend deploying separate Docker containers or using Metadata Filtering for logical separation within a single index.

1. Vector Dimensionality

• Recommendation: Choose dimensions matching your embedding model.
• Support: Native support for 1024 (BGE-M3), 1536 (OpenAI), 768 (BERT), and 8 (Hyperbolic).
• Reason: HyperspaceDB now uses Const Generics to optimize for specific dimensions at compile time.

2. Quantization Strategy

• Mode: Use Binary quantization for maximum memory savings.
• Trade-off: Binary mode reduces precision but compresses vectors by 32x-64x compared to floating-point.
• When to use: Large-scale datasets where memory is the bottleneck.

3. Indexing Parameters

• ef_construction: Controls index build time vs. search quality. Higher values = better recall but slower indexing.
• ef_search: Controls search time vs. recall. Higher values = better recall but slower search.
• Tuning: Adjust via gRPC without restarting the server.

4. Hybrid Search

• Enable: Use hybrid_query parameter in search requests.
• Tuning: Adjust hybrid_alpha (0.0 to 1.0) to balance semantic similarity and keyword matching.

---

⚡ RAG Performance Benchmarks (v3.1)

We conducted a rigorous evaluation of HyperspaceDB v3.1 against popular vector databases (Qdrant, ChromaDB, Milvus, Weaviate) across 15 distinct RAGBench datasets (ranging from 1,000 to 100,000 documents).

The results highlight HyperspaceDB's Lock-Free ArcSwap Architecture and optimized cache layers.

📋 Benchmark: HYBRID Search (Lexical BM25 + Semantic)

Evaluating retrieval accuracy (Recall@10, Context Precision) and throughput (QPS, Latency) averaged across 15 domains.

#### 📊 Index Size: 1,000 Documents (HYBRID 1000x100)
| Database | Ingest QPS | Search QPS | Latency p50 | Latency p99 | RAM Usage | Disk Usage | CPU Usage | Recall@10 | Context Precision |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| HyperspaceDB | 55,722 🚀 | 1,374.73 ⚡ | 0.62 ms | 1.93 ms | 407.5 MB | 91.80 MB | 8.0% | 64.47% | 54.60% |
| ChromaDB | 1,547 | 218.27 | 4.76 ms | 8.86 ms | 2,132.0 MB | 573.75 MB | 19.4% | 65.25% | 54.07% |
| Milvus | 495 | 240.80 | 4.18 ms | 8.87 ms | 970.9 MB | 524.86 MB | 19.8% | 65.28% | 54.15% |
| Qdrant | 6,801 | 204.27 | 7.69 ms | 12.61 ms | 390.0 MB | 5.58 MB | 22.5% | 65.28% | 54.12% |
| Weaviate | 588 | 465.67 | 2.22 ms | 6.38 ms | 468.6 MB | 20.32 MB | 55.2% | 63.63% | 51.49% |

#### 📊 Index Size: 100,000 Documents (HYBRID 100000x1000)
| Database | Ingest QPS | Search QPS | Latency p50 | Latency p99 | RAM Usage | Disk Usage | CPU Usage | Recall@10 | Context Precision |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| HyperspaceDB | 52,572 🚀 | 1,375.53 ⚡ | 0.74 ms | 1.70 ms | 1,089.3 MB | 100.13 MB | 11.9% | 55.33% | 51.29% |
| ChromaDB | 1,455 | 196.93 | 5.67 ms | 9.06 ms | 2,448.0 MB | 677.44 MB | 31.0% | 55.13% | 50.97% |
| Milvus | 2,269 | 191.13 | 5.78 ms | 9.17 ms | 1,161.8 MB | 857.11 MB | 34.0% | 55.15% | 51.01% |
| Qdrant | 7,023 | 193.20 | 8.45 ms | 12.50 ms | 409.2 MB | 16.48 MB | 37.1% | 55.25% | 51.03% |
| Weaviate | 1,455 | 482.93 | 2.37 ms | 5.78 ms | 482.7 MB | 40.92 MB | 132.0% | 53.83% | 49.15% |

---

🏆 Concurrency & Stress Testing

Measured on the CovidQA dataset under concurrent search queries.

#### 👥 100 Parallel Clients
| Database | QPS | Latency Avg | p50 | p99 | Max RAM |
| :--- | :---: | :---: | :---: | :---: | :---: |
| HyperspaceDB (L2 Cache ON) | 3,706.53 🚀 | 3.28 ms | 2.79 ms | 9.27 ms | 72.0 MB |
| ChromaDB | 422.46 | 185.99 ms | 165.14 ms | 414.39 ms | 3,026.9 MB |
| Milvus | 879.17 | 91.39 ms | 91.54 ms | 165.00 ms | 1,880.1 MB |
| Weaviate | 631.17 | 130.10 ms | 120.42 ms | 269.45 ms | 843.4 MB |

#### 👥 1,000 Parallel Clients
| Database | QPS | Latency Avg | p50 | p99 | Max RAM |
| :--- | :---: | :---: | :---: | :---: | :---: |
| HyperspaceDB (L2 Cache ON) | 5,261.26 🚀 | 5.63 ms | 4.50 ms | 21.48 ms | 72.0 MB |
| ChromaDB | 386.57 | 2,014.16 ms | 1,793.00 ms | 4,897.77 ms | 3,026.9 MB |
| Milvus | 648.27 | 1,201.19 ms | 1,198.67 ms | 2,729.93 ms | 3,708.9 MB |
| Weaviate | 597.78 | 1,414.36 ms | 1,322.76 ms | 3,038.84 ms | 2,174.0 MB |

---

📉 Why is HyperspaceDB so fast?

1. Lock-Free Architecture: Readers never block readers using ArcSwap and lock-free atomic references.
2. L0 Two-Level Caching: Sharded L1 DashMap (~1 µs lookups) coupled with a background-rebuilt L2 HNSW fallback graph (~100 µs lookups).
3. Anisotropic Quantization: Custom ScaNN-inspired scalar I8 loss quantization maintains high Recall while saving 8x RAM.
4. Zero-Copy Serialization: Fast memory mapping (mmap) of segment files ensures immediate startup recovery without parsing overhead.

> RAG Benchmarks Hardware Config: Apple M4 Pro (64GB RAM), NVMe SSD storage.

---

🐳 Deployment

Official Docker Images

HyperspaceDB publishes three specialized Docker distribution tags to match different deployment architectures:

| Image Tag | Extensions / PDK | Embedder | Architecture | Size | Ideal Use Case |
| :--- | :---: | :---: | :---: | :---: | :--- |
| 🚀 glukhota/hyperspace-db:latest | ✅ Built-in & External PDK (/app/plugins) | Deterministic Manifold & Remote APIs (ort disabled) | Multi-Arch (amd64, arm64) | ~50 MB | Default / Recommended: Ultra-fast, lightweight vector database for production, local dev, edge, and microservices. |
| 🧠 glukhota/hyperspace-db:latest-embed | ✅ Full PDK + Active ONNX Runtime | ✅ Built-in Local ONNX / HF Hub / Remote APIs | Multi-Arch (amd64, arm64) | ~650 MB | All-in-One: Self-contained server that vectorizes raw text directly in-database without external APIs or client-side embedding libraries. |
| ⚡ glukhota/hyperspace-db:latest-saas | ✅ Core SaaS PDK (shield,eco) | ❌ Offloaded to microservices (cde-service) | linux/amd64 | ~48 MB | Managed Cloud SaaS & Kubernetes: Engineered specifically for high-throughput cloud clusters with S3/Cloudflare R2 chunk tiering, AVX-512 SIMD, and EcoMonitor telemetry. |

---

Quick Start

#### 1. Run Standard Engine with PDK Plugins

docker run -d 
  --name hyperspace 
  -p 50051:50051 
  -p 50050:50050 
  -v $(pwd)/hs_data:/app/data 
  -v $(pwd)/plugins:/app/plugins 
  glukhota/hyperspace-db:latest

#### 2. Run All-in-One Engine (With Built-In ONNX Embedding Model)

docker run -d 
  --name hyperspace-embed 
  -p 50051:50051 
  -p 50050:50050 
  -v $(pwd)/hs_data:/app/data 
  -v $(pwd)/plugins:/app/plugins 
  glukhota/hyperspace-db:latest-embed

Access the Web Dashboard and Plugins Workbench at http://localhost:50050 and gRPC at localhost:50051.

Docker Compose

Run the full stack with mounted data and plugins directories:

docker-compose up -d

---

🔒 Security & SOC 2 Compliance

HyperspaceDB comes with advanced security controls out-of-the-box, ensuring compatibility with standard enterprise compliance requirements (e.g. SOC 2, ISO 27001):

• Mutual TLS (mTLS): Built-in TLS and client verification support for both gRPC (port 50051) and HTTP Dashboard/API (port 50050). Configure via .env:

* HSTLSCERT=/path/to/server.crt
* HSTLSKEY=/path/to/server.key
* HSTLSCA=/path/to/ca.crt (enables mutual TLS verification)

• Role-Based Access Control (RBAC): Fine-grained user role permissions (Admin, ReadWrite, ReadOnly) managed via transaction-safe redb database.
• Cross-Tenant Collection Sharing: Securely share collections between tenants with read-only/read-write privileges using Namespaced URI references (owner/collection_name).
• Structured Audit Logs: Structured JSON audit logging (HSAUDITLOG_LEVEL=full) written to stdout or a file, with tenant-isolated logs streamed securely to authenticated users.

> [!TIP]
> For SOC 2 compliance, we recommend deploying HyperspaceDB on LUKS-encrypted drives or encrypted cloud block volumes within isolated Virtual Private Clouds (VPC).

---

🔌 AI Integrations

HyperspaceDB is ready for the modern AI stack with official high-level integrations:

| Integration | Language | Status | Install |
| --- | --- | --- | --- |
| 🧠 Hyperspace Memory (Mem0 Drop-In) | Python | ✅ v4 | pip install hyperspace-memory |
| 🧠 Hyperspace Memory (Mem0 Drop-In) | TS/JS | ✅ v4 | npm install hyperspace-memory |
| 🦜 LangChain | Python | ✅ v4 | pip install langchain-hyperspace |
| 🦜 LangChain | TS/JS | ✅ v4 | npm install langchain-hyperspace |
| 🦙 LlamaIndex | Python | ✅ v4 | pip install llama-index-vector-stores-hyperspace |
| 🦙 LlamaIndex | TS/JS | ✅ v4 | npm install llamaindex-hyperspace |
| 🔗 n8n | No-Code | ✅ v4 | Community Node: n8n-nodes-hyperspacedb |
| 🤖 AI Agent Skills | Any | ✅ v4 | npm install hyperspacedb-skills |

---

🛠️ Model Context Protocol (MCP)

Connect HyperspaceDB directly to Claude Desktop, Cursor, Windsurf, Antigravity, or any MCP-compliant agent to give them persistent cognitive spatial memory.

🧠 1. Zero-Code Cognitive Agent Memory (mcp-hyperspace-memory)

Drop-in cognitive agent memory server with 8 specialized memory tools (memoryremember, memoryrecall, memoryforget, memoryupdate, memorylistsessions, memoryexplorehierarchy, memoryconsolidate, memoryverify_claim):

Start Memory MCP server via npx
npx mcp-hyperspace-memory@latest

#### Option A: Managed Cloud Memory (Zero Infrastructure)

{
  "mcpServers": {
    "hyperspace-memory": {
      "command": "npx",
      "args": ["-y", "mcp-hyperspace-memory@latest"],
      "env": {
        "HYPERSPACEAPIKEY": "skYOURYARAPIKEY",
        "MEMORYCOLLECTION": "agentmemory"
      }
    }
  }
}

#### Option B: Local DB Instance + Cloud v5 Embedding API Key (Default Auth)
Run glukhota/hyperspace-db:latest locally on your machine. Your vector graph and memory payloads stay 100% private on localhost, while text is vectorized via the state-of-the-art continuous v5Embedding801 model in the cloud:

{
  "mcpServers": {
    "hyperspace-memory": {
      "command": "npx",
      "args": ["-y", "mcp-hyperspace-memory@latest"],
      "env": {
        "HYPERSPACE_HOST": "localhost:50051",
        "HYPERSPACEAPIKEY": "skYOURYARAPIKEY",
        "MEMORYCOLLECTION": "agentmemory"
      }
    }
  }
}

#### Option C: Local DB with Custom Password + Cloud v5 Embedding API Key (Two Keys)
If your local .env has a custom security password (HYPERSPACEAPIKEY=mylocalsecret), provide two separate keys:

{
  "mcpServers": {
    "hyperspace-memory": {
      "command": "npx",
      "args": ["-y", "mcp-hyperspace-memory@latest"],
      "env": {
        "HYPERSPACE_HOST": "localhost:50051",
        "HYPERSPACEAPIKEY": "mylocalsecret",
        "YARAPIKEY": "skYOURYARAPIKEY",
        "MEMORYCOLLECTION": "agentmemory"
      }
    }
  }
}

• HYPERSPACEAPIKEY: Authenticates to local gRPC database (localhost:50051).
• YARAPIKEY (or legacy CDEAPIKEY): Authenticates to cloud https://the.yar.ink/v1/embeddings (v5Embedding801).

(Get your sk_ API key at yar.ink/dashboard).

⚡ 2. Full Vector Database & Graph MCP Server (mcp-hyperspacedb)

Complete database administration, graph traversal, and cognitive analytics:

Start Full Database MCP server via npx
npx mcp-hyperspacedb@latest

Available Tools (30+):

• hyperspacesearchtext / hyperspacesearchwasserstein: Semantic & Optimal Transport search.
• hyperspaceinserttext / hyperspacedeletepoints / hyperspacegetpoints: Data CRUD.
• hyperspacecreatecollection / hyperspacedeletecollection: Collection management.
• hyperspacegraphtraverse / hyperspaceexploregraph / hyperspacegetneighbors: Graph navigation.
• hyperspacegetsubsumptiontree / hyperspacegetconceptparents: Hierarchy traversal.
• hyperspaceanalyzethoughtstability / hyperspacepredictmomentum / hyperspacegettrustscore: Cognitive AI.
• hyperspaceverifygeneration: Tribunal geometric hallucination verification.
• hyperspacequeryplanner: Natural-language sub-query decomposition and planned retrieval.
• hyperspaceanalyzegeometry: Gromov Delta → metric recommendation (cosine/l2/lorentz/poincare/hybrid).

hyperspacerebuildindex / hyperspacevacuum / hyperspacecache_: System maintenance.

---

📦 SDKs (v4.x Milestone)

Official 1st-party drivers with full Functional Parity, Tribunal Verification (verify_generation), Delta Sync, and Cognitive Math:

| Language | Path | Status |
| --- | --- | --- |
| 🐍 Python | pip install hyperspacedb | ✅ v4.1.0 |
| 🦀 Rust | cargo install hyperspacedb | ✅ v4.1.0 |
| 🦕 TypeScript/JS | npm install hyperspace-sdk-ts | ✅ v4.1.0 |
| 🕸️ WebAssembly | crates/hyperspace-wasm | ✅ v4.1.0 |
| 🎯 Dart/Flutter | hyperspacedb: ^4.1.0 | ✅ v4.1.0 |
| 🐹 Go | sdks/go | ✅ v4.1.0 |
| 🤖 ROS2 / C++ | sdks/ros2, sdks/cpp | ✅ v4.1.0 |
| 🧠 AI Agent Skills | npm install hyperspacedb-skills | ✅ v4.1.0 |

> 🧠 AI Agent Skills (hyperspacedb-skills) are structured SKILL.md files that teach Cursor, Claude Code, Windsurf, Antigravity, and custom agents how to correctly use HyperspaceDB — covering core CRUD, graph traversal, cognitive AI (Lyapunov/Koopman), Tribunal verification, query planning, DePIN node economics, and the full 30+ MCP tool reference. Install once, all your agents understand HyperspaceDB automatically.

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Copyright © 2026 YARlabs