A calling convention is the set of rules a programme uses to pass arguments to a function and receive its return value. It also defines responsibilities such as which registers or stack areas are preserved, making it a core part of binary interoperability between compiled components.
Binary Compatibility and ABI Contracts
A calling convention is more than a syntax rule for parameters, it is part of the application binary interface that makes separately compiled code work together. It tells each side where arguments live, how return values are delivered, and which state must survive the call so that components remain interoperable across compilation units, languages, and toolchains.
That contract is why the same function can be callable from different objects or libraries without source-level coordination. When the convention differs, the code may still compile, but the callee can read the wrong values, return data in the wrong place, or corrupt the execution state that the caller expects to remain intact.
What the Caller and Callee Must Agree On
The convention defines the mechanics of a call boundary: argument ordering, how large or complex return values are handled, whether parameters are passed in registers or on the stack, and which side is responsible for stack cleanup. It also covers preserved registers, which is critical because the caller may rely on those registers after the function returns.
In practice, the calling convention is a low-level interoperability rule that sits underneath source code, but above raw machine instructions. It is one of the reasons binary interfaces are stable only when the whole toolchain, platform, and compiler settings agree on the same call semantics.
Why Mismatches Break Software
When two components disagree on the convention, the failure is often immediate and opaque. A function may appear to run, but it will misinterpret its inputs, return the wrong value, or restore the stack incorrectly. Those faults can produce crashes, silent data corruption, or control-flow instability that is difficult to trace back to the call boundary.
The issue is especially sharp in mixed-language systems, plugin architectures, hand-written assembly, and code that crosses compiler or platform boundaries. Even a small mismatch, such as caller-cleanup versus callee-cleanup, can destabilise the process long before the error becomes visible at the source level.
Why It Matters for Secure Systems
Calling conventions are usually discussed as a compatibility concern, but they also matter to security because binary mismatches can create undefined behaviour, memory corruption, and unpredictable control flow. In a security-sensitive component, those faults can widen the impact of bugs elsewhere, make exploitability harder to reason about, and undermine assumptions about trustworthy execution boundaries.
They also affect defensive tooling and low-level analysis. If instrumentation, hooks, or embedded stubs assume the wrong convention, the observer may distort register state or miss the real data flow, which weakens debugging, crash analysis, and exploit triage.
Risk and Threat Considerations
Calling-convention mismatches are a common source of hard-to-diagnose crashes and memory corruption, especially in mixed-language, plugin, and native extension environments. The risk is not the convention itself, but the fragility of the boundary it defines, because a single disagreement can turn a valid call into undefined behaviour.
Failure mechanism: The caller and callee disagree about argument placement, return-value handling, register preservation, or stack cleanup, so execution state is interpreted incorrectly and the process becomes unstable or corrupt.
Impact: The result can be denial of service, silent data corruption, broken instrumentation, or conditions that increase the likelihood or severity of a memory-safety exploit.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while NIST SP 800-53 Rev 5, CIS Controls v8 and OWASP ASVS set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | SA-11 — Developer Testing and Evaluation | Calling-convention mismatches surface as interface and binary-behaviour defects that testing must catch. |
| SI-2 — Flaw Remediation | ABI defects and boundary bugs require controlled correction when discovered in shipped components. | |
| Recommendation — Test cross-language and cross-compiler call boundaries for ABI mismatches before release. Remediate ABI and calling-convention defects promptly when they can cause instability or corruption. | ||
| CIS Controls v8 | CIS-16 — Application Software Security | Native interfaces and low-level integration require secure handling of implementation defects and unsafe boundaries. |
| Recommendation — Review native integrations and extension points for binary-interface defects that can destabilise execution. | ||
| OWASP ASVS | V15 — Secure Coding and Architecture | Secure design must account for low-level interface assumptions, including native call boundaries. |
| Recommendation — Specify and verify binary-interface assumptions in secure architecture and code review. | ||
| MITRE ATT&CK | T1068 — Exploitation for Privilege Escalation | Boundary corruption from ABI mistakes can contribute to exploit conditions that raise process privilege. |
| Recommendation — Map boundary-corruption findings to escalation risk and validate exploitability in your analysis. | ||
Practitioner Guidance
Why practitioners should care: Treat the calling convention as part of the compatibility contract, not as an implementation detail. Any interface that crosses a language boundary, compiler boundary, or hand-assembled boundary should be reviewed as carefully as a public API.
What to watch for: Unexpected crashes after a seemingly successful call, corrupted stack traces, register-state anomalies, or behaviour that changes when a function is compiled, inlined, or linked differently. Those are classic signs that the boundary contract is wrong or inconsistently applied.
Practitioner takeaway: If a binary interface is unstable, verify the convention first, because many low-level failures look like logic bugs until the call boundary is inspected.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on November 2, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org