A character bigram is a pair of adjacent characters used as a text feature in machine learning. In PowerShell analysis, bigrams help capture low level structure that survives obfuscation better than whole words do. This makes them useful for distinguishing normal scripts from manipulated ones.
How character bigrams work
Character bigrams are pairs of adjacent characters extracted from text, such as th, he, or er. They turn raw strings into compact features that preserve local shape, punctuation patterns, and partial word structure without needing full tokenization.
That makes bigrams especially useful when text is noisy, short, or intentionally altered. In security analysis, they often expose patterns that still survive simple obfuscation, while remaining simple enough to scale across large script corpora.
Why they help in text classification
Bigrams sit in a useful middle ground between single characters and whole words. Single characters are too sparse to capture meaningful context, while whole words can be brittle when attackers change casing, insert separators, or substitute lookalike characters.
Because adjacent-character patterns are easier to retain than exact words, bigrams can improve discrimination in models that classify scripts, malware-adjacent text, spam, or other manipulated content. Their value comes from capturing recurring local sequences rather than semantic meaning.
Where they are most useful in PowerShell analysis
In PowerShell analysis, character bigrams are a practical feature for spotting low-level syntax and lexical structure. They can reflect command composition, operator spacing, quoting habits, variable usage, and other local signatures that remain even when an author tries to disguise intent.
This is one reason they are common in detection pipelines that compare normal automation against manipulated or suspicious scripts. The feature does not understand the script, but it can still distinguish stable structure from heavily edited text when the surrounding vocabulary changes.
Limitations and interpretation
Bigrams are descriptive features, not a verdict. They work best as part of a wider model that also considers tokens, syntax, script metadata, and behavioral signals, because character patterns alone can overfit to formatting quirks or benign house style.
They also tend to be sensitive to language, encoding, and preprocessing choices. A model built on bigrams should be evaluated against realistic text variation so that it does not mistake ordinary formatting differences for malicious manipulation.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while NIST SP 800-53 Rev 5 sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | SI-4 — System Monitoring | Character bigrams support detection of manipulated script patterns in monitoring workflows |
| SI-7 — Software, Firmware, and Information Integrity | Feature-based analysis helps identify altered or suspicious script content before execution | |
| Recommendation — Use SI-4 to monitor script features that indicate obfuscation or manipulation. Use SI-7 to validate scripts and flag content that shows tampering patterns. | ||
| MITRE ATT&CK | T1027 — Obfuscated Files or Information | Bigram features are useful for spotting text-level obfuscation in scripts and commands |
| Recommendation — Map suspicious text patterns to T1027 and hunt for obfuscation in script telemetry. | ||
Related resources from NHI Mgmt Group
- Why do AI coding agents make hidden-character attacks more dangerous?
- Why do hidden-character attacks complicate extension ecosystem security and incident detection?
- How should security teams implement consistent character encoding controls in CI/CD pipelines?
- Why do inconsistent character encodings create security risk in application pipelines?
