TECHNIQUE
Tool Use & Structured Output
Code generation and execution is deployed as constrained, tool-backed workflow machinery: operators scope the task, let the model generate or drive code/query steps, and validate outputs with tests, sandboxes, rules, traces, or reviewers.
Wrap code generation/execution in explicit workflows or tool-backed loops instead of one-shot prompts.
5 of 7 operators with cited teardown evidence show workflow or tool-loop wrappers around model-driven code execution/generation.Use executable code or generated queries for bounded transformation tasks: Python runtime problem solving, input normalization, repository edits, test repair, or Cypher query generation.
6 of 7 operators with cited teardown evidence show bounded code/query generation or execution tasks.Scope the model’s context before generation or execution so it sees only the relevant files, functions, domains, chunks, or prompt step.
5 of 7 operators with cited teardown evidence show context scoping before model-driven code/query work.Validate generated code or tool outputs with deterministic tests, language services, rule validators, sandboxes, or accuracy/evaluation frameworks.
6 of 7 operators with cited teardown evidence show validation or evaluation around generated code/query/tool outputs.Separate LLM reasoning from deterministic formatting, filtering, or execution where reliability matters.
4 of 7 operators with cited teardown evidence explicitly separate model reasoning from deterministic code, filters, or tooling.Keep debugging and evaluation artifacts for model-driven workflows, including traces, logs, dashboards, transcripts, or production monitoring.
3 of 7 operators with cited teardown evidence show observability or logging for code-generation/execution workflows.Across the cited deployments, code generation/execution is not described as an unconstrained model prompt; it is embedded in retrieval, workflow orchestration, tool servers, agents, or validation pipelines.
Operators differ in where generated or executable code runs.
APPROACH 01
Python runtime or interpreter inside an AI platform or agent system.
APPROACH 02
Sandboxed code execution for normalizing structured client inputs before internal tools consume them.
APPROACH 03
Developer-tool execution loop around repository edits, diagnostics, and tests.
APPROACH 04
Generated graph queries routed to knowledge graph or GraphQL backends.
APPROACH 05
LLM-generated optimization findings are validated and then routed into code optimization or manual review workflows.
Operators use different orchestration forms around the model.
APPROACH 01
Declarative or drag-and-drop deterministic workflows.
APPROACH 02
Supervisor agent coordinating specialized subagents.
APPROACH 03
Dedicated migrator process plus tool server for file selection, prompting, diagnostics, and tests.
APPROACH 04
Retrieval-first multi-step agent orchestration for search and knowledge work.
APPROACH 05
Profiling-first optimization pipeline that filters hot functions before LLM antipattern detection.
Validation differs by deployment target.
APPROACH 01
LLM jury plus domain-specific rule validators to catch false positives and reduce hallucinations.
APPROACH 02
Compiler, language-server, linter, autocorrect, and test feedback loops.
APPROACH 03
Sandboxing, minimal interpreter design, security reviews, and production traces/evals.
APPROACH 04
Accuracy testing and semantic-search refinement for generated retrieval/query behavior.
Unconstrained or naive AI was reported as unreliable for code work: Block said naive AI usage would be too unreliable, and Shopify said letting AI roam across millions of lines of code did not work well because non-determinism hurts reliability.
Operators monitor or mitigate incorrect model outputs, including false positives, hallucinations, drift, and query inaccuracies.
Safe code execution is treated as a deployment concern: Dropbox built a minimal Python interpreter with testing and security reviews, while Rippling uses sandboxed code execution for action agents.
Context size and task complexity require aggressive scoping or decomposition before code generation/execution.
| Name | Kind | When | Maturity |
|---|---|---|---|
| E2B sandboxes | service | managed isolated execution of model-written code | emerging |
| Jupyter kernel execution | pattern | data-analysis loops where state persists across generated cells | established |
| WASM sandboxing | pattern | untrusted snippets must run with tight, portable isolation | emerging |