Trust-aware middleware layer for LLM-powered coding agents that quantifies generation reliability using composite risk scoring and automated human-in-the-loop decision routing.
Team of 3 (with Anisha Apte and Shruti Bhamidipati). Columbia COMS E6156
Coding agents will hand you a diff with total confidence whether the diff is right or catastrophically wrong. The interesting question isn't "can an LLM write code" (it clearly can), it's whether you can tell, before applying it, which outputs deserve your trust. I wanted a system that answers that with a number instead of a vibe.
There's a second failure mode upstream of generation: prompts that are too ambiguous to answer well in the first place. An agent that guesses at an underspecified request produces plausible code for the wrong problem, which is worse than producing nothing.
TAOL sits between a developer and their local LLM (Ollama), intercepting every prompt and every generated output. Rather than trusting or distrusting the model wholesale, it computes a Composite Trust Score from four post-execution risk signals (Static Analysis Gate, Function Overlap, CodeBLEU, and Dependency Volatility) and routes each output into one of three decision zones: auto-apply, human review, or defer with clarification questions.
The pipeline runs in six phases. An Intent Gate pre-screens prompts with a Random Forest classifier over 24 handcrafted semantic features. A Context Enricher does RAG-based augmentation via BM25 search and git diffs, but only for borderline prompts, so the common case stays fast. An Ollama Proxy streams generation behind a circuit breaker. Then the Trust Calibrator, Decision Engine, and Handover Manager score, route, and explain the result.
The weights weren't guessed. I learned them with an ensemble of Logistic Regression, Random Forest, and Gradient Boosting under grid search against 500 SWE-bench Verified tasks drawn from 12 real Python repositories (Django, Flask, scikit-learn, sympy and others), each with a gold patch to compare against. CodeBLEU came out dominant at weight 0.5465, with Function Overlap at 0.270, Dependency Volatility at 0.094, and Static Analysis at 0.089.
prompt
│
├─▶ 1. INTENT GATE Random Forest, 24 semantic features
│ │ 99.2% acc · defer if ambiguous
│ ▼
├─▶ 2. CONTEXT ENRICHER BM25 + git diffs (borderline prompts only)
│ ▼
├─▶ 3. OLLAMA PROXY streamed generation, circuit breaker
│ ▼
├─▶ 4. TRUST CALIBRATOR SAG · FO · CB · DV
│ ▼ 4 post-execution risk signals
├─▶ 5. DECISION ENGINE CTS = Σ(learned weights × signals)
│ ▼
└─▶ 6. HANDOVER MANAGER
│
├── auto_apply high trust
├── human_review briefing + LLM explanation
└── defer_to_human clarification questionsThe Intent Gate hits 99.2% accuracy with an AUC of 0.999, and holds 99.4% on held-out data. A five-experiment ablation study confirmed CodeBLEU as the dominant discriminator between safe and risky patches: the signal that actually separates them, not just the one that correlates.
Beyond the scoring itself: live codebase indexing through tree-sitter AST parsing with watchdog-based incremental reindexing, and a session-adaptive Human-Over-The-Loop modifier that shifts thresholds by up to ±0.15 based on whether the developer has been accepting or rejecting suggestions.
It ships three ways: a drop-in FastAPI proxy server, an interactive Rich CLI, or an embeddable Python library, with 144 unit tests and reproducibility scripts for the full evaluation.
The ablation study was the most valuable part of the project and I almost skipped it. Learning the weights told me what predicted risk; systematically removing signals told me which ones were load-bearing versus merely correlated. Those are different questions and only the second one tells you what to keep.
Semantic similarity to a known-good patch (CodeBLEU) outperformed static analysis by a wide margin. Static analysis catches code that is malformed; it has very little to say about code that is well-formed and wrong. Most agent failures are the second kind.
class TrustOrchestrator:
"""Embeddable trust-aware orchestration layer."""
async def evaluate(self, prompt: str) -> HandoverResult:
# Phase 1: Intent analysis (24-feature RF classifier)
intent = await intent_gate.analyze(prompt, self.config, self.ast_index)
if intent.should_defer:
return handover_manager.defer(intent.clarification_questions)
# Phase 2: Context enrichment for borderline prompts
if intent.should_enrich:
enriched = context_enricher.enrich(prompt, intent, self.bm25_index)
# Phase 3: LLM generation with circuit breaker
generation = await ollama_proxy.generate(active_prompt, session)
# Phase 4: Post-generation trust analysis (SAG, FO, CB, DV)
trust = await trust_calibrator.analyze(generation, self.config)
# Phase 5: CTS computation and zone routing
cts = decision_engine.compute_cts(trust, self.config.weights)
# Phase 6: Route to auto_apply | human_review | defer_to_human
return handover_manager.route(generation, cts, trust)