How should security teams detect AI-written malware without relying on signatures?

Why This Matters for Security Teams

Signature-based detection is a weak control when malware can be regenerated on demand, slightly rewritten, or wrapped in fresh tooling each time it is deployed. For security teams, the real issue is not whether a sample has been seen before, but whether a process is behaving like malware once it starts executing. That shift aligns with the behaviour-first model described in the NIST Cybersecurity Framework 2.0, where detection should focus on observable events, not just known artifacts.

This matters especially when AI-generated payloads inherit the same tradecraft patterns seen in real-world NHI abuse. NHIMG research on Shai Hulud npm malware campaign shows how fast-moving attacks can target secrets, repositories, and runtime trust boundaries rather than relying on a single static file. In practice, many security teams encounter AI-written malware only after it has already tested privileges, touched sensitive paths, or attempted persistence, rather than through intentional detection of the initial binary.

How It Works in Practice

The practical answer is to detect execution patterns that are hard to fake consistently across environments. Start by instrumenting the endpoint and workload so every suspicious process is evaluated in context: parent process, command line, token scope, file writes, registry or launch-agent changes, outbound destinations, DNS behaviour, and attempts to enumerate credentials or reach cloud metadata services. This is where identity context becomes crucial. A process launched under a build agent, service account, or automation token should not have the same freedom as an interactive admin session.

Current guidance suggests combining telemetry across three layers:

• Process behaviour: privilege checks, injection attempts, script execution, compression, encryption, and disabling of security tools.

• Identity and access context: which NHI, token, workload, or service account executed the action, and whether that access was expected.

• Network and persistence signals: unusual beaconing, short-lived domain lookups, scheduled tasks, startup entries, cron changes, and outbound connections to newly seen infrastructure.

This approach is stronger when paired with workload identity and lifecycle discipline. The NHI Lifecycle Management Guide is useful here because malware frequently abuses the same trust gaps that exist when credentials, tokens, and service identities are over-permissioned or left in place too long. Teams should also treat leaked or abused secrets as a direct detection input, not just a remediation problem, especially given how quickly exposed credentials are abused in the wild, as highlighted in NHIMG’s DeepSeek breach research and the broader secrets hygiene issues documented in The State of Secrets in AppSec.

These controls tend to break down in highly ephemeral container clusters with sparse endpoint telemetry because process lineage, file-system change, and network context are often incomplete.

Common Variations and Edge Cases

Tighter behaviour-based detection often increases telemetry volume and tuning overhead, requiring organisations to balance visibility against alert fatigue and performance cost. That tradeoff becomes sharper in developer laptops, CI/CD runners, and serverless workloads, where legitimate automation can resemble malware if policy is too blunt. Best practice is evolving, and there is no universal standard for this yet, but runtime allowlisting by identity and task context is more durable than static file reputation alone.

Some edge cases deserve special handling. Packed or memory-only malware may never leave a useful file hash, so detections need to follow in-memory actions, not just disk artifacts. Living-off-the-land techniques may look benign at the process name level, which is why command-line arguments, child-process chains, and unusual privilege escalations matter more than the executable label. AI-written malware may also mutate just enough to bypass simple heuristic rules while preserving the same objective, so detections should weight intent signals such as credential harvesting, lateral movement, and persistence attempts.

For teams governing autonomous or semi-autonomous systems, the broader lesson is that runtime authorisation and identity correlation matter more than static trust assumptions. The Top 10 NHI Issues is a useful reference point for understanding how over-privileged non-human access creates the same detection blind spots malware exploits. In practice, detections work best when they ask not only “what is this file?” but also “what is this process trying to do, and under whose authority?”

Related resources from NHI Mgmt Group

• What steps should security teams take to prevent Shadow AI risks?
• How should security teams handle AI-driven identity fraud in remote onboarding?
• How should security teams verify identity when AI-generated profiles look authentic?
• How should security teams automate ITDR without causing unnecessary outages?