Refactoring should usually come first when the source code itself is hard to navigate. Prompt tuning can help at the margin, but the article shows that structural quality changes token use and reread behaviour directly. If the repository is tangled, better prompts will still have to compensate for a poor search surface.
Why Refactoring Usually Comes Before Prompt Tuning
Refactoring is the better first move when the repository itself is the bottleneck. If code is hard to trace, the model or developer has to spend more effort recovering context, and prompt tuning only improves the margin around that friction. Structural quality changes how quickly relevant paths, names, and dependencies can be found.
A prompt can improve instruction following, but it cannot fully compensate for a weak codebase search surface. When the underlying structure is noisy, ambiguous, or overly coupled, each interaction costs more tokens and more rereading. That makes the problem partly one of code hygiene, not just prompt design.
The practical decision is simple: if the team is spending time asking the model to re-explain or re-locate the same areas of the code, refactoring is addressing the root cause. If the repository is already coherent and the remaining issue is how tasks are phrased or constrained, prompt tuning can be a useful secondary optimisation.
Where Prompt Tuning Still Helps
Prompt tuning matters most once the codebase is reasonably navigable. It can improve task framing, reduce ambiguity, and make the model more likely to focus on the right files, symbols, or edge cases. In other words, it sharpens execution, but it does not rebuild the search surface the team is working on.
Teams should treat prompt tuning as a leverage multiplier on a decent substrate, not a substitute for one. A better prompt can reduce back-and-forth, but when the source material is tangled, the model still has to infer intent from poor structure. That is why prompt work often looks effective in the short term yet stalls when the repository remains messy.
The strongest use case for prompt tuning is repeatable work with a stable codebase: code review assistance, targeted analysis, test generation, or change planning. In those settings, a more precise prompt can improve consistency and reduce wasted iterations without masking deeper maintainability problems.
How to Decide What to Do First
Use the shape of the work, not the novelty of the tool, to set priority. If the main issue is tangled modules, unclear ownership boundaries, duplicated logic, or poor naming, start with refactoring. If the main issue is inconsistent instructions, weak task decomposition, or overbroad requests, start with prompt tuning.
Decision rule: choose refactoring first when repeated model failures trace back to code structure, not to instruction quality. Choose prompt tuning first only when the code is understandable and the team is trying to improve how the model is directed, constrained, or verified.
What good looks like: the team should see fewer reruns, less rereading, and less context hunting after the first intervention. If prompt tuning is working, it should improve focus without needing constant human correction. If refactoring is working, it should make the repository easier to inspect even before the prompts get better.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP ASVS, OWASP SAMM and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP ASVS | V15 — Secure Coding and Architecture | Refactoring improves code structure and maintainability, which directly affects secure architecture and code quality. |
| Recommendation — Refactor tangled code so task execution and review become easier to reason about. | ||
| OWASP SAMM | Implementation Governance | This question concerns improving development practice quality through better code structure and task execution. |
| Recommendation — Review how development practices reduce rework and improve maintainability. | ||
| NIST CSF 2.0 | ID.IM-01 — Improvements are identified and implemented based on evaluations | Teams are deciding whether to improve the system structure or the instruction process first. |
| Recommendation — Use evaluation feedback to choose the highest-leverage improvement first. | ||
Practitioner Guidance
What to prioritise: fix the structural friction first when the repository is the source of repeated interpretation overhead. A prompt cannot reliably recover signal from a codebase that forces every reader, human or model, to reconstruct the same context repeatedly.
What to measure: watch the amount of rereading, context switching, and repeated clarification needed to complete the same task. If those costs fall after refactoring, the team is addressing the real constraint; if they remain flat, prompt work alone is probably not attacking the right problem.
Common mistake: teams often tune prompts to compensate for poor code organisation because it is faster than changing the code. That can create temporary relief, but it leaves the underlying search and comprehension burden in place, which means future work keeps paying the same tax.
Practitioner takeaway: optimise the substrate before the instruction layer when the substrate is what is making every task expensive. Prompt tuning can refine performance, but refactoring usually determines whether the team is fighting the code or simply guiding it.
Related resources from NHI Mgmt Group
- How should security teams improve prompt quality using iterative evaluation rather than first-draft tuning?
- How should teams think about transformer models when choosing between prompt engineering and fine tuning?
- What do teams get wrong when they treat prompt engineering as a substitute for fine-tuning?
- What do security teams get wrong about prompt engineering for AI agents?