Prompt engineering is often seen as an art form — skilled individuals iterating through many prompt versions to get the outcome they want. In my work, I noticed that this process leaves valuable knowledge either buried in the prompt itself or lost entirely. That's what motivated me to develop meta-prompting for behavior specification, a methodology for systematically capturing and documenting the knowledge gained during iterative refinement. I've written it up as a preprint: https://doi.org/10.5281/zenodo.19801926
The core idea is a tightly coupled loop where each refinement cycle includes a generalization phase. When a specific fix corrects a behavioral failure, I immediately ask the model to extract a general rule from that fix, scoped to the task at hand. Over iterations, this accumulates a set of grounded, transferable design principles — not just a better prompt for one task, but a structured guide for producing good prompts across a class of tasks.
This tackles a real problem: prompt engineers develop genuine expertise, but that expertise is often tacit — hard to articulate, teach, or reuse. By embedding the generalization step within each refinement cycle, the methodology avoids what I call the reconstruction problem, producing principles that are contemporaneous records rather than retrospective summaries.
I examined this through a case study in slot-filling conversational AI design. The resulting guide contains principles spanning structural decisions, localized concerns, and diagnostic practices. For example, one principle I derived separates slot capture from workflow steps to avoid the model re-asking for information already provided; another separates slot confirmation from go-ahead authorization to prevent premature actions.
I assess principle quality through qualitative analysis, examining internal coherence and the traceable connection between each principle and a class of real failure. The guide reads as a progression from architectural decisions to local refinements to diagnostic practice — reflecting how the methodology accumulates knowledge across iterations.
Practically, this transforms an invisible refinement process into a reusable artifact. Theoretically, it raises questions I find genuinely interesting: what do LLMs know about their own behavior, and how can that knowledge be surfaced through structured reflection at the moment of repair?
If you're working on complex LLM behavior specifications, I'm open to consulting engagements in this space. I also build on this work through my platform Silex, which enables the implementation of complex behaviors via email — a concrete example of what's possible being Aleik.
I believe prompt engineering doesn't have to remain a tacit discipline. By adopting this methodology from the start of a refinement process, practitioners can avoid the reconstruction problem and produce richer, more actionable guides. The preprint is linked above — I'd love to hear thoughts from others working in this space.