♠University of California, Los Angeles
♣AWS GenAI
Recent multimodal search agents use reasoning, vision, and live web search across many turns to answer questions beyond parametric knowledge by orchestrating a small library of hand-crafted search and visual tools. However, the library is laborious to create and maintain, and frozen when new query types, evidence formats, and failure modes are encountered in the wild. Human practitioners, by contrast, cope with insufficient tools by fashioning new ones from familiar parts and reusing them across tasks. Inspired by this, we propose ToolSmith, an automatic tool-generation framework that grows and reuses modules on top of a lean base tool library. We evaluate ToolSmith on three in-domain multimodal search benchmarks and ten out-of-domain benchmarks under a VLM-as-a-Judge protocol. ToolSmith consistently improves over the base library across deployed agents of varying scales, with larger backbones reaping greater gains, demonstrating the scalability of the approach. Our code and data will be publicly available.
ToolSmith runs a four-phase pipeline on top of an eight-primitive base library. Phase 1 (Trajectory Mining) clusters the agent's failed trajectories into capability gaps. Phase 2 (Reuse-or-Grow) checks each gap against the current catalog and escalates to generation only when no existing tool fits. Phase 3 (Tool Generation) prompts a generator LLM to author the new tool as Python source under structural and anti-hardcode constraints. Phase 4 (Two-Gate Verification) admits a candidate only after a sandboxed execution check and an LLM static code review. At inference, a lightweight retriever routes the top-K generated tools to each query alongside the always-visible base primitives.
Across 3 in-domain and 10 out-of-domain benchmarks, tools generated by ToolSmith (GPT-5 generator) consistently improve over an already strong base-tool agent:
| Inference model | ID avg (base → +ToolSmith) | OOD avg (base → +ToolSmith) |
|---|---|---|
| Qwen3-VL-8B | 47.94 → 50.14 (+2.20) | 26.57 → 27.90 (+1.33) |
| Qwen3-VL-32B | 55.47 → 57.14 (+1.67) | 31.27 → 34.25 (+2.98) |
| GPT-5 | 51.80 → 52.84 (+1.04) | 39.80 → 42.20 (+2.40) |
Gains appear in 36 of 39 backbone×benchmark cells. Ablations show the tool-generation model should be as strong as possible, while the trajectory-mining model should match the inference model; a cluster-count sweep exposes a quality–coverage trade-off.
The ToolSmith logo takes its colors from UCLA: the blue and gold are UCLA's official colors. Its style draws on the ukiyo‑e woodblock prints of Katsushika Hokusai, which inspired the flowing golden lines that run across the hammer.
@article{sun2026toolsmith,
title = {ToolSmith: An Automatic Tool-Generation Framework for Multimodal Search Agents},
author = {Sun, Rui and Huang, Junhua and Zancato, Luca and Islam, Md Mofijul and Chang, Kai-Wei},
journal = {arXiv preprint},
year = {2026},
note = {arXiv link coming soon}
}