[Bastion-RAG 1 – Sentinel] – Metadata Filtering

When designing “Metadata Filtering,” defining its conceptual boundaries was initially challenging. Unlike “Prompt Injection,” which presents explicit threat signatures, mapping metadata constraints within an enterprise pipeline is inherently abstract. However, breaking the framework down to core software engineering principles clarified the architecture and justified the necessity of this layer.

When a user payload enters the Bastion-RAG boundary, the architecture must enforce immediate validation. While standard RAG implementations focus on semantic or vector proximity, an enterprise-grade governance ecosystem requires stricter controls. Before computing similarity, the framework must verify the request’s Identity, Structural Format, and Freshness.

Even with a cryptographically segregated multi-tenancy storage (Vault), isolation fails if the downstream execution path can access unauthorized metadata fields. The metadata validation engine (Validator) within the validators/metadata/ package functions as a low-latency, fail-closed gatekeeper. It analyzes incoming request envelopes before downstream compute engines run, blocking malformed or manipulated payloads at the perimeter.

This post analyzes our compile-time memory optimizations designed to eliminate runtime overhead, alongside the specifications for our four-stage validation matrix.

URL Site > https://github.com/zafrem/bastion-sentinel

Series Name: Bastion – Project Security RAG

• [Bastion-RAG] Project Security RAG
• [Bastion-RAG 0] Get help from AI (Architecture Design)
• [Bastion-RAG 1 – Sentinel]
  • Prompt Injection Defense
  • Metadata Filtering – Here!
• [Bastion-RAG 2 – Vault]
  • Multi-tenancy
  • Deterministic De-identification
• [Bastion-RAG 3 – Navigator]
  • Hybrid Reranking
  • Logical Partitioning
• [Bastion-RAG 4 – Archor]
  • Embedding Noise Injection
  • Embedding Model Bias Verification
• [Bastion-RAG 5 – Tracker]
  • Data Lineage Tracking
  • Honey-token Injection
• [Bastion-RAG Demo]

1. Ingress Threat Modeling: Countering Vulnerabilities at the Gate

The ingress boundary of a RAG pipeline is a high-value target for adversaries attempting to map index structures or trigger buffer anomalies. The structural validation matrix is explicitly engineered to neutralize four core threat categories:

1.1 Missing Context (Unauthenticated Request Infiltration)

This vulnerability occurs when a payload bypasses perimeter authentication checks and enters the pipeline without a tenant_id or a verifiable user_id. If an un-scoped query reaches downstream vector engines, the retriever executes a global index scan across the shared database instance, leading to cross-tenant data exposure.

1.2 Privilege Escalation (System-Level Key Injection)

An adversary injects reserved system identifiers—such as system, admin, or root—into metadata fields. This attempts to manipulate Open Policy Agent (OPA) validation engines or relational query routers into granting administrative access, enabling unauthorized access to corporate records.

1.3 Replay Attacks (Session Re-use Exploitation)

An attacker intercepts an authorized request payload and replays it against the ingress endpoint. This bypasses active authorization lifecycles, exploits stale cache states, or exhausts system concurrency limits.

1.4 Amplification Abuse (Buffer and Parsing Exhaustion)

Adversaries embed large text payloads inside metadata fields. This structural attack starves the internal JSON parser’s memory pool or triggers CPU utilization spikes via regular expression backtracking (ReDoS), resulting in a denial of service (DoS) for the application node.

2. Validator Component Design and Compile-Time Memory Optimization

An ingress validation gateway must operate under an ultra-strict processing budget, ensuring it introduces zero perceptible lag to user interactions. To constrain runtime evaluation costs to less than 0.1 milliseconds, Bastion-RAG isolates regular expression compilation entirely to the application bootstrap phase.

Go

// validators/metadata/validator.go

type Validator struct {
    cfg      config.MetadataValidationConfig
    patterns map[string]*regexp.Regexp  // field name → pre-compiled pattern
}

func New(cfg config.MetadataValidationConfig) (*Validator, error) {
    v := &Validator{
        cfg:      cfg,
        patterns: make(map[string]*regexp.Regexp),
    }
    for field, rule := range cfg.FieldRules {
        if rule.Pattern == "" {
            continue  // Well-defined schemas like context_id rely on native type parsers (e.g., UUID)
        }
        re, err := regexp.Compile(rule.Pattern)
        if err != nil {
            return nil, fmt.Errorf("invalid metadata regex pattern for field %s: %w", field, err)
        }
        v.patterns[field] = re
    }
    return v, nil
}

By executing regexp.Compile inside the New() constructor, all regex trees are permanently anchored into memory within a thread-safe map[string]*regexp.Regexp map. When a live request stream floods the gateway, the engine bypasses runtime allocation and parsing completely. Fields mapped to deterministic standards (such as a standard context_id following UUIDv4) bypass the regex evaluation layer entirely, routing instead through native string byte-matching logic for maximum throughput.

3. Four-Stage Validation Matrix and Fail-Closed Circuit Breaking

To guarantee pipeline integrity, every incoming metadata envelope must pass sequentially through four synchronous verification layers. Following strict zero-trust design paradigms, a single exception or failed assertion across any layer triggers an immediate, absolute execution halt.

Incoming Request Envelope (Metadata + Query)
    │
    ▼
[Layer 1: Required Field Availability] ← Instantly drops payload if tenant_id/user_id is missing (0.01ms)
    │
    ▼
[Layer 2: Format & Length Boundary]    ← Blocks anomalous sizes (4KB max) and enforces alpha-numeric maps
    │
    ▼
[Layer 3: Business Logic Window]       ← Runs BR-001 (Authoritative Timestamp) & BR-002 (Reserved Words)
    │
    ▼
[Layer 4: Access Isolation Mapping]    ← Cryptographically binds verified tokens to Navigator search filters
    │
    ▼
  [PASSED] (Safe handoff to downstream Vault and Navigator processes)

Layer 1: Required Field Availability

The engine scans top-level keys to confirm the presence of routing prerequisites. If either tenant_id or user_id is absent, execution terminates immediately. This fast-fail mechanism at the pipeline perimeter prevents downstream compute nodes from processing un-routed payloads.

Layer 2: Format and Length Boundary

This layer enforces size and format boundaries to prevent buffer manipulation and query amplification:

• Volumetric Constraints: The raw query string is limited to a maximum of 10,000 runes, and the structural metadata block is restricted to a maximum of 4KB.
• Alphanumeric Enforcement: To prevent injection mutations, incoming string identifiers are validated against pre-compiled character sets (^[a-z0-9-]+$). Fields designated as UUIDs undergo strict parsing to prevent downstream type anomalies.

Layer 3: Business Logic and Freshness Window

This layer applies state assertions to mitigate token manipulation and session re-use vectors:

• Freshness Verification (BR-001): The gateway extracts the client-supplied timestamp and measures its variance against the system clock. If the temporal deviation exceeds ±1 hour, the transaction is classified as a replayed or tampered packet and is dropped. The server clock serves as the single source of truth.
• Reserved Word Auditing (BR-002): The user_id string is checked against system reservation arrays. If an unauthenticated payload contains identifiers such as system, admin, or root, the validator flags it as a privilege escalation attempt and triggers security isolation workflows.

Layer 4: Access Isolation Mapping

Only payloads that pass all validation steps receive security clearance. If an exception occurs, the system strips internal schema metrics from the response to prevent exposing system details, returning a generic error summary instead (e.g., "timestamp: outside allowed window", "user_id: reserved identifier not allowed"). The transaction status is set to BLOCKED, the execution loop terminates, and an anomaly log is sent asynchronously to the central Tracker platform.

4. The Core Paradigm: Pre-Filtering Isolation vs. Post-Filtering Evacuation

The core design difference of this validation layer is the enforcement of pre-filtering isolation rather than post-search data truncation.

Many standard RAG architectures execute similarity searches globally across an entire collection. They load all vector matches into memory and then apply an application-level filter to discard records that do not match the user’s tenant_id or authorization status. This design pattern presents a security risk; because unprivileged data is read and processed by the database instance prior to filtration, the system is susceptible to metadata side-channel leaks and timing attacks.

[ Antipattern: Post-Filtering Extraction ]
User Query ──▶ Global Vector DB Scan ──▶ Aggregate Candidates ──▶ Strip Violations via Memory (Vulnerable ❌)

[ Bastion-RAG Pattern: Pre-Filtering Isolation ]
User Query ──▶ [Ingress Invalidation] ──▶ Inject Strict Crypto Meta-Filters ──▶ HNSW Restricted Scan (Secure 🎯)

The Bastion-RAG architecture prohibits post-retrieval trimming through the following mechanism:

• Identifiers validated at the ingress gate are bound to the gRPC request context payload via an immutable contract.
• When the query reaches the Python-based Navigator module, these validated identities are translated into structural database filters before search initialization.
• The database engine (Qdrant) applies these conditions directly to the root level of the Hierarchical Navigable Small World (HNSW) graph traversal.
• As a result, unauthorized partitions and alternate tenant segments are excluded from the graph search space from the initial step, ensuring data isolation at the compute layer.

5. Architectural Symmetry: Evaluating the Egress Path (Sentinel-OUT)

A frequent point of debate during system design was whether a symmetrical verification layer is necessary when text flows from the LLM back to the user, given that strict metadata partitioning is already enforced during data ingestion.

The necessity arises from the non-deterministic nature of generative models. While metadata pre-filtering limits what data the pipeline retrieves from the database, it cannot control how an LLM processes or transforms that data internally. Therefore, the egress gate (Sentinel-OUT) does not replicate database metadata filtering; instead, it executes distinct Content Guardrail Validations to mitigate generative risks:

• PII Re-emergence Prevention: During ingestion (Vault Phase-1), raw identifiers (e.g., "Hong Gildong") are mapped to tokens (e.g., [PERSON_a3f2c1]). The LLM processes only these anonymized tokens. However, during text synthesis, model weights or indirect patterns within source texts can occasionally cause the LLM to reconstruct or deduce real identities (e.g., generating "The status of customer Hong Gildong is..."). Sentinel-OUT parses the egress stream, detects if original PII signatures have re-emerged, and replaces them with a [REDACTED] format.
• Exfiltration Containment: If an indirect prompt injection attack occurs—where system commands embedded in documents hijack the model’s instructions—the LLM may attempt to output system blueprints, operational paths, environmental configurations, or API tokens (sk-...) within the response. Sentinel-OUT monitors the outgoing text for infrastructure signatures and terminates the connection if corporate asset signatures are detected.

6. Epilogue: Forging Real-World Resilience Through Collective Intent

Implementing the validation and filtering layers of Bastion-RAG demonstrates why software engineering principles must govern AI deployment. Standard boilerplate implementations or basic single-direction HTTP proxies often fail to meet production latency budgets while leaving output vectors vulnerable to exploitation.

By establishing strict technological boundaries—including compile-time regular expression mapping, authoritative temporal validation, and strict pre-filtration—this ingress architecture isolates enterprise data streams while maintaining an execution footprint of under 0.1 milliseconds.

For Chief Information Security Officers (CISOs) and AI Architects operating in regulated data environments, this architecture provides a practical framework for implementing zero-trust data protection across LLM initiatives without sacrificing pipeline velocity or operational scalability.