Character Bigram

What Character Bigrams Capture in Text Analysis

Character bigrams model local character-to-character structure rather than whole words, so they can surface patterns that survive spelling variation, casing changes, inserted separators and many forms of obfuscation. That makes them especially useful when the goal is to detect repeated low-level text signals instead of relying on dictionary meaning.

Because they operate at the character level, bigrams are less sensitive to token boundaries than word features. In security analysis, that often makes them a stronger signal when scripts, commands or payloads have been manipulated to evade simpler text matching.

Why Character Bigrams Work Well for Obfuscated PowerShell

PowerShell analysis is a good fit for character bigrams because malicious or suspicious scripts are often altered just enough to defeat exact word matching while leaving recurring character patterns intact. Bigrams can still represent fragments of cmdlets, operators, quoting styles, parameter patterns and concatenation habits even when the script has been reformatted.

This does not make bigrams a full understanding mechanism. They are a statistical feature, not a semantic one, so they work best as part of a broader detection pipeline that can compare many examples and learn which local sequences tend to appear in benign or malicious code.

In practice, their value is that they preserve signal when an analyst needs resilience against surface-level evasion. The same idea also applies beyond PowerShell, wherever an attacker may try to disguise intent by changing spacing, punctuation or character order without fully changing the underlying structure.

How Character Bigrams Are Used in Machine Learning Pipelines

To use character bigrams, text is typically converted into overlapping two-character pairs, then counted or weighted so a model can learn which pairs occur frequently in each class. That representation turns raw text into a numeric feature space that supports classification, clustering or anomaly detection.

The approach is often paired with other features, because bigrams capture only local context. They are strong at recognising patterns of form, but weaker at representing long-range meaning, command intent or execution logic on their own.

For security work, that trade-off is useful. A model can pick up character-level signatures of obfuscation, scripting habits and unusual formatting even when higher-level parsing is incomplete or unreliable.

Strengths and Limits of Character Bigram Features

Character bigrams are robust against many small text changes, which is why they are popular in malware classification, language identification and tamper-aware text analysis. They are also simple to compute and easy to feed into classical machine learning models.

Their main limitation is precision. Because they ignore broader syntax, they can overgeneralise and may miss deeper differences between benign and malicious content when those differences are not visible in short character windows. They also depend heavily on good feature engineering and representative training data.

For that reason, character bigrams are usually most effective when treated as one layer in a layered detection design, not as a standalone verdict. They help expose patterns, but they do not replace structural parsing, behavioural analysis or analyst review.

Risk and Threat Considerations

Character bigrams themselves are not a threat, but the security value of the technique comes from detecting text that has been deliberately altered to avoid easier inspection. If defenders rely only on whole-word matching, small edits can hide suspicious PowerShell activity, reduce detection quality and create blind spots in triage.

Failure mechanism: Obfuscation changes the visible form of a script while preserving enough low-level character structure for a bigram model to notice. When tuned well, that model can flag manipulated commands that would otherwise slip past simple signatures or naïve keyword filters.

Impact: Better bigram-based detection can improve identification of suspicious scripts, reduce false negatives in malware and abuse screening, and give analysts earlier visibility into encoded, fragmented or intentionally noisy content.

Practitioner Guidance

Why practitioners should care: Character bigrams are most useful when you expect adversaries to change surface text faster than they can change underlying behaviour. They are a practical feature choice for detection problems where local form matters more than human-readable meaning.

What to watch for: Bigram performance depends on preprocessing decisions such as casing, normalisation and token handling. Small changes there can materially alter model behaviour, so the feature pipeline should be treated as part of the control, not just a technical detail.

Practitioner takeaway: Use character bigrams to complement, not replace, higher-level script analysis, especially when the detection problem is shaped by evasion and text manipulation.

Related resources from NHI Mgmt Group

• Why do AI coding agents make hidden-character attacks more dangerous?
• Why do hidden-character attacks complicate extension ecosystem security and incident detection?
• How should security teams implement consistent character encoding controls in CI/CD pipelines?
• Why do inconsistent character encodings create security risk in application pipelines?