← Aojie (Justin) Yuan

Writing · 中文解读

Plain-language explainers of my research — trustworthy AI agents, LLM reasoning and interpretability, memory, and evaluation.

Research guides · AI agent safety & auditing · Evaluating and verifying agents · LLM reasoning & interpretability · Memory, systems & optimization

Key terms defined in the glossary · RSS

Research guides

• How Do You Make AI Agents Trustworthy? A Research Guide

  A research guide to trustworthy AI agents: pre-execution audit layers, the five dimensions of auditability, multi-user and skill-supply-chain risks, budget governance, and claim verification.

• Why AI Agent Benchmarks Mislead, and How to Evaluate Agents Better

  Five ways AI agent benchmarks mislead — clean-world assumptions, judging by looks, single scores, ignoring time, binary verdicts — and what to measure instead, from five research papers.

• What Do LLMs Know About Their Own Reasoning? What the Internals Show

  What LLM internals reveal about reasoning: hidden error awareness in chain-of-thought, a shared subspace for prose, code and math, and a weights-only test against the logit lens.

AI agent safety & auditing

• Why Every AI Agent Needs an Audit Layer

  An audit layer records and governs what an AI agent does — every tool call, dollar, and decision. Aojie (Justin) Yuan explains why pre-execution auditing beats post-hoc logging, and how to add one with zero code changes.

• What Makes an AI Agent Auditable?

  Accountability for AI agents requires auditability. Aojie (Justin) Yuan explains the five dimensions of agent auditability, why detect, enforce, and recover mechanisms are all needed, and what the Auditability Card is.

• How Do You Secure Multi-User AI Agent Networks?

  When each person has an AI agent acting on their behalf, harm can travel between users through shared tasks. Aojie (Justin) Yuan explains cross-user agent security and how an auditable sandbox benchmark measures it — the idea behind WeClawArena.

• Do AI Agent Skills Stay Safe Across Languages?

  Public AI agent skills differ by language: English skills lean technical, Chinese skills lean toward media, social, and finance apps — and safety signals are uneven. Aojie (Justin) Yuan on cross-lingual security of the agent skills ecosystem.

• What Is Charter-Governed AI Orchestration?

  Charter-governed AI orchestration puts a machine-readable constitution — mission, budget, and rules — in charge of a multi-agent system, with a CFO that enforces spend and an auditor that scores outcomes. Aojie (Justin) Yuan explains the idea behind Sovereign-OS.

• How Do You Audit Agent Handoffs Without Collecting the Raw Messages?

  Auditing agent-to-agent handoffs without a central copy of raw messages: a metadata-only audit path, a separate local release gate, and what the experiments do and do not show.

Evaluating and verifying agents

• Can Computer-Use Agents Do Visual Software Engineering?

  Real debugging means running software, looking at it, and fixing the code. Aojie (Justin) Yuan explains CUA-SWE, a benchmark where agents must combine screenshots, GUI interaction, and code edits — verified by deterministic tests.

• How Do You Benchmark Tool-Use Agents for the Sim-to-Real Gap?

  Tool-use agents are tested on clean inputs but deployed into typos, timeouts, and duplicate tool names. Aojie (Justin) Yuan explains the sim-to-real gap for tool-use agents and how RobustBench-TC and a domain-randomized RL recipe measure and close it.

• What Is a Self-Evolving Verification Agent?

  A self-evolving verification agent checks whether an AI's claims are supported by its sources, emits structured evidence and error diagnoses, and improves itself with a process reward. Aojie (Justin) Yuan explains the idea behind SEVA.

• When Should an AI Agent Keep Working — and When Should It Deliver?

  Agent chronoception explained: why punctuality and timing invariance can't show an AI agent uses time well, and how to test useful persistence.

LLM reasoning & interpretability

• Do LLMs Know When Their Chain-of-Thought Is Wrong?

  Language models internally detect their own reasoning errors while sounding confident. Aojie (Justin) Yuan explains hidden error awareness in chain-of-thought — a 0.95 AUROC probe signal — and why it is diagnostic, not causal.

• Do LLMs Reason the Same Way in English, Code, and Math?

  Aojie (Justin) Yuan explains the Format-Agnostic Reasoning Subspace (FARS): a 10-dimensional subspace where LLMs represent the same reasoning concept across prose, code, and math — and the declarative-procedural asymmetry it reveals.

• Is a Concept Subspace Just the Logit Lens in Disguise?

  A weights-only geometric test checks whether an extracted concept subspace overlaps the model's output readout. Aojie (Justin) Yuan explains the diagnostic, its controls, and what it shows across 26 models.

Memory, systems & optimization

• How Do AI Agents Remember Long Conversations?

  Long-term agent memory needs retrieval that is cheap when the query is easy and thorough when it is hard. Aojie (Justin) Yuan explains workload-adaptive cascade retrieval — the idea behind AgentIR.

• Do LLM Reasoning Models Need All Their Memory in HBM?

  Long chain-of-thought fills GPU HBM with KV cache, most of which barely matters. Aojie (Justin) Yuan explains a semantics-aware memory hierarchy that tiers the KV cache by importance — the idea behind 'Not All Thoughts Need HBM'.

• Why Does Decision-Focused Learning Often Fail to Beat MSE?

  Decision-focused learning trains predictors on downstream decision loss, yet gains over MSE are often small. Aojie (Justin) Yuan explains Jacobian rank collapse and what the experiments show.

• How Do You Retrieve Medical Images by Lesion Region, Not Just the Whole Scan?

  Aojie (Justin) Yuan explains HMAR, a mixture-of-experts medical image retrieval framework that pairs global and lesion-level experts, trains without bounding boxes, and hashes with KAN layers.