444060c3eb
- Introduced `critic`, an independent adversarial reviewer for security and correctness. - Added `fable-orchestrator` to manage task routing and verification. - Implemented `gauntlet-critic` for fresh-context evaluation of gauntlet rounds. - Created `planner` for generating executable implementation plans with dependencies. - Developed `security-auditor` for application security reviews and audits. - Established `system-steward` to improve agent prompts and skills based on verified failures. - Added `dev-loop` skill for autonomous development loops over repositories. - Implemented `gauntlet-loop` skill for iterative quality benchmarking against reference standards. - Updated project settings to utilize the new orchestrator agent. - Created documentation for `GAUNTLET.md`, `PROGRESS.md`, and `REFERENCE_BAR.md` to track project status and quality benchmarks. - Added detailed prompting style guide to enhance understanding of prompt patterns and agentic loops.
219 lines
10 KiB
Markdown
219 lines
10 KiB
Markdown
```markdown
|
||
From a systems and software engineering perspective, prompt patterns and agentic loops are structured control flow mechanisms built on top of autoregressive transformer models.
|
||
|
||
Below is a detailed technical breakdown of these patterns, covering their state transitions, context memory management, prompt schemas, and failure modes.
|
||
|
||
---
|
||
|
||
## 1. Deterministic & Context-Shaping Patterns
|
||
|
||
These patterns operate at the inference step level to constrain token generation probabilities and enforce structural invariants.
|
||
|
||
### Role & System Conditioning (Logit Shaping)
|
||
* **Mechanism:** Injects instructions directly into the system message block, modifying the baseline attention weights across all subsequent user/assistant turns. It acts as an inductive bias, shifting the probability distribution of generated tokens toward domain-specific terminologies and structured logic.
|
||
* **Prompt Schema:**
|
||
```text
|
||
<system_instruction>
|
||
ROLE: Senior Distributed Systems Architect.
|
||
DOMAIN: Real-time event-driven infrastructure, gRPC, distributed consensus (Raft/Paxos).
|
||
INVARIANT: Prioritize zero-data-loss guarantees over minimal latency. Reject eventual consistency unless explicitly requested.
|
||
OUTPUT_FORMAT: Technical specification markdown with formal system invariants.
|
||
</system_instruction>
|
||
```
|
||
* **Failure Modes & Mitigations:** *Context Decay* (the model forgets constraints in long turns). Mitigate by placing critical invariant rules at the very end of the system block or repeating constraints in system system-reinforcement flags.
|
||
|
||
### Few-Shot Delimiter Scaffolding
|
||
* **Mechanism:** Imprints input-output mapping patterns directly into the model’s Key-Value (KV) cache. Utilizing explicit XML or structural delimiters prevents token boundary confusion during multi-turn parsing.
|
||
* **Prompt Schema:**
|
||
```xml
|
||
<system>Extract operational state from syslog streams.</system>
|
||
|
||
<example>
|
||
<input>2026-08-07T08:12:01Z node-04 dockerd[1042]: Error: OOMKilled process 8841</input>
|
||
<output>{"node": "node-04", "event": "OOMKilled", "pid": 8841, "severity": "CRITICAL"}</output>
|
||
</example>
|
||
|
||
<target>
|
||
<input>2026-08-07T08:14:22Z node-01 kernel: [44211.2] Out of memory: Kill process 1204 (postgres)</input>
|
||
<output>
|
||
```
|
||
* **Failure Modes:** Recency/label bias (overweighting the last example's exact values). Keep examples structurally diverse and balanced across edge cases.
|
||
|
||
---
|
||
|
||
## 2. Multi-Step Inference & Search Graph Patterns
|
||
|
||
These frameworks alter the model’s internal computation path by generating intermediate reasoning tokens before emitting the target response.
|
||
|
||
### Chain-of-Thought (CoT) & Plan-and-Solve
|
||
* **Mechanism:** Forces auto-regressive decoding to populate the context buffer with intermediate rationale steps ($z_1, z_2, \dots, z_n$) prior to predicting the target output ($y$). Mathematically:
|
||
$$P(y \mid x) = \sum_z P(y \mid x, z) P(z \mid x)$$
|
||
* **Execution Protocol:**
|
||
```text
|
||
Perform the following analysis in two explicit, separated phases:
|
||
PHASE 1 (REASONING_BUFFER):
|
||
- Identify state invariants and potential race conditions.
|
||
- Draft intermediate computational dependencies.
|
||
- Evaluate step-by-step edge cases.
|
||
|
||
PHASE 2 (EXECUTION_OUTPUT):
|
||
- Provide the final production-ready implementation wrapped in ```json tags.
|
||
```
|
||
* **When to Use:** Algorithmic execution, mathematical logic, complex SQL/query optimization.
|
||
|
||
### Tree-of-Thoughts (ToT) / Graph-of-Thoughts (GoT)
|
||
* **Mechanism:** Combines LLM generation with classical state-space search algorithms (Breadth-First Search, Depth-First Search, or $A^*$). The LLM acts both as a *Thought Generator* ($S_{t+1} \sim G(S_t)$) and a *State Evaluator* ($V(S_t) \in [0, 1]$).
|
||
|
||
```text
|
||
[Root State: Initial Prompt]
|
||
/ \
|
||
[Thought A] [Thought B]
|
||
v = 0.8 v = 0.2 (Pruned)
|
||
/ \
|
||
[Thought A1] [Thought A2]
|
||
v = 0.95 v = 0.4
|
||
```
|
||
|
||
* **Execution Pseudocode:**
|
||
```python
|
||
def tree_of_thoughts_search(root_prompt, beam_width=3, max_depth=4):
|
||
current_states = [root_prompt]
|
||
for depth in range(max_depth):
|
||
candidates = []
|
||
for state in current_states:
|
||
# 1. Expand candidate branches via LLM
|
||
branches = llm_generate_branches(state, num_samples=3)
|
||
# 2. Evaluate state heuristic score V(s) via LLM
|
||
scores = [llm_evaluate_state(branch) for branch in branches]
|
||
candidates.extend(zip(branches, scores))
|
||
|
||
# 3. Prune low-scoring branches (Beam Search)
|
||
candidates.sort(key=lambda x: x[1], reverse=True)
|
||
current_states = [branch for branch, score in candidates[:beam_width]]
|
||
return current_states[0] # Best evaluated path
|
||
```
|
||
* **When to Use:** Strategic planning, complex refactoring across multiple files, architecture synthesis.
|
||
|
||
---
|
||
|
||
## 3. Agentic Loops & State-Machine Architectures
|
||
|
||
Agentic frameworks wrap the LLM inside an external, deterministic control loop (e.g., Python/Go runtime, orchestration engines like OpenClaw, or custom middleware).
|
||
|
||
### ReAct (Reasoning + Action Protocol)
|
||
* **State Machine:**
|
||
$$\text{State}_t \rightarrow \text{Thought}_t \rightarrow \text{Action}_t(\text{Tool Call}) \rightarrow \text{Observation}_t \rightarrow \text{State}_{t+1}$$
|
||
|
||
```text
|
||
+--------------+ +-------------------+ +-----------------+
|
||
| LLM Engine | ----> | Action (Tool Call)| ----> | Execution Runtime|
|
||
+--------------+ +-------------------+ +-----------------+
|
||
^ |
|
||
|-------------- Observation (Payload) <--------------+
|
||
```
|
||
|
||
* **Prompt Engine Specification:**
|
||
```text
|
||
You operate in a strict execution loop. Available Tools: [exec_bash, query_sql, HTTP_GET].
|
||
|
||
Use the following format strictly:
|
||
Thought: <Logical about current reasoning state>
|
||
Action: <Tool_Name>(<JSON_Arguments>)
|
||
Observation: <Result by environment injected>
|
||
|
||
Loop terminates ONLY when you emit:
|
||
Final Answer: <Summary of outcome>
|
||
```
|
||
* **Failure Modes:** Infinite loops caused by unhandled tool errors.
|
||
* **Mitigation:** Enforce hard step budgets (`max_iterations = 10`) and circuit breakers on duplicate tool signatures.
|
||
|
||
### Plan-Execute-Verify (PEV) with Re-Planning
|
||
* **Mechanism:** Decouples task breakdown from task execution. The planner generates a Directed Acyclic Graph (DAG) of sub-tasks. An execution loop steps through nodes sequentially, running validation assertions after each step. If a step fails, control yields back to a Re-Planner node to mutate the remaining DAG.
|
||
|
||
```text
|
||
+--------------+
|
||
| Generate DAG |
|
||
+--------------+
|
||
|
|
||
v
|
||
+-----------------+
|
||
+->| Execute Node N |
|
||
| +-----------------+
|
||
| |
|
||
| v
|
||
| +-----------------+ FAIL +---------------+
|
||
| | Assert / Verify | -------------> | Re-Plan DAG | --+
|
||
| +-----------------+ +---------------+ |
|
||
| | PASS |
|
||
| v |
|
||
| [More Nodes Remaining?] --YES--------------------------+
|
||
| | NO
|
||
| v
|
||
| +-----------------+
|
||
+--| Final Outcome |
|
||
+-----------------+
|
||
```
|
||
|
||
### The Gauntlet Loop (Adversarial Multi-Agent Architecture)
|
||
* **Mechanism:** Implements a strict **Maker-Checker Isolation Model**. The Builder Agent generates code/artifacts. A *blind* Critic Agent—instantiated in a zero-history, isolated context window—evaluates the output against a hard reference standard or test harness.
|
||
|
||
```text
|
||
+------------------+ +--------------------+
|
||
| Builder Agent | --- Generates ---> | Artifact Payload |
|
||
| (Context Window) | +--------------------+
|
||
+------------------+ |
|
||
^ v
|
||
| +--------------------+
|
||
|-- Injects Actionable Feedback| Judge Agent |
|
||
| (No Excuses Allowed) | (Isolated Context) |
|
||
| +--------------------+
|
||
| |
|
||
+<-- [Fails Reference Standard] ---------+
|
||
```
|
||
|
||
* **System Architecture Protocol:**
|
||
```python
|
||
def gauntlet_loop(task_spec, reference_standard, max_gauntlet_runs=5):
|
||
builder_context = init_builder_context(task_spec)
|
||
|
||
for iteration in range(max_gauntlet_runs):
|
||
# Step 1: Builder generates artifact
|
||
artifact = builder_agent.run(builder_context)
|
||
|
||
# Step 2: Instantiate Judge in FRESH context window (Zero memory leak)
|
||
judge_prompt = f"""
|
||
TASK: Compare Artifact against Reference Standard.
|
||
REFERENCE: {reference_standard}
|
||
ARTIFACT TO EVALUATE: {artifact}
|
||
|
||
OUTPUT RULES:
|
||
1. Determine if Artifact >= Reference Standard in quality/correctness.
|
||
2. If FAIL, list the single most critical structural deficiency. Do not offer encouragement.
|
||
FORMAT: STATUS: [PASS|FAIL] | FEEDBACK: <concise directive>
|
||
"""
|
||
|
||
verdict = judge_agent.run_fresh_context(judge_prompt)
|
||
|
||
if verdict.status == "PASS":
|
||
return artifact
|
||
|
||
# Step 3: Append harsh feedback to builder context
|
||
builder_context.append_user_message(f"GAUNTLET REJECTION: {verdict.feedback}")
|
||
|
||
raise MaximumGauntletDepthExceeded("Quality threshold not met within limit.")
|
||
```
|
||
|
||
---
|
||
|
||
## Technical Summary Matrix
|
||
|
||
| Pattern / Loop Style | Latency Cost | Context Consumption | Determinism | Best Architectural Use Case |
|
||
| :--- | :--- | :--- | :--- | :--- |
|
||
| **Few-Shot / Schema** | Low ($O(1)$) | Low | High | API Payload Generation, Format Standardization |
|
||
| **Chain-of-Thought** | Medium ($O(k)$) | Medium | Medium | Intermediate Math, Single-Query Logic Tracing |
|
||
| **Tree-of-Thoughts** | High ($O(b^d)$) | High | High | Complex Codebase Refactoring, Architecture Search |
|
||
| **ReAct Agent** | Dynamic | Medium-High | Medium | Runtime API Orchestration, Infrastructure Ops |
|
||
| **Plan-Execute-Verify** | High | High | High | Multi-Step Migration Pipelines, CI/CD Automation |
|
||
| **Gauntlet Loop** | Very High | Extreme | Maximum | Autonomous End-to-End System/Software Synthesis |
|
||
|
||
``` |