Jupyter Notebooks for Threat Hunting with MSTICPy

Most threat hunts live in a hunter's head and a browser history, so they cannot be repeated, reviewed or handed over. Jupyter notebooks fix that by putting the hypothesis, the queries, the enrichment and the conclusions in one executable document. This guide covers how security teams use notebooks for hunting, with a focus on MSTICPy, the open source Python library maintained by the Microsoft Threat Intelligence Center (MSTIC), and on papermill for running the same hunt on a schedule with different parameters. It is written for threat hunters, detection engineers and SOC leads who already know the hunting process and want to make it repeatable. Securing the Jupyter server itself is a separate topic: see the Jupyter Server Security Guide. For hunting method and hypothesis design, see the Threat Hunting Guide.

Key takeaways

A notebook is a hunt you can rerun. Text, queries, code and expected output sit together, so a peer can review the logic and a junior analyst can execute it against new data.
MSTICPy supplies the plumbing. It queries Microsoft Sentinel, Defender, Splunk, Kusto and other sources into pandas DataFrames, then adds threat intelligence lookups, GeoIP, WHOIS, IoC extraction, anomaly analysis and timelines.
Parameterise, then schedule. Tag one cell parameters and papermill can execute the notebook with a new time window or host list, producing a dated output notebook as evidence.
The notebook holds credentials and sensitive output. Keep API keys in Key Vault or environment variables through msticpyconfig.yaml, and strip outputs before anything reaches Git.
Hunts should graduate. A hunt that finds something repeatable belongs in your detection pipeline as a rule, with the notebook kept as the investigation companion.

What the tool does

Jupyter runs code cells against a kernel (usually Python) and stores code, markdown and results in a single .ipynb JSON file. On its own it is a general data science environment. For hunting, three components turn it into a repeatable workflow:

Component	What it adds to a hunt	Licence and maintainer
MSTICPy	Data providers, a built-in query library, threat intelligence and enrichment, pivot functions, analysis and visualisation (timelines, process trees)	MIT, Microsoft (MSTIC)
papermill	Parameterises and executes notebooks programmatically, writing an output notebook per run	BSD 3-Clause, nteract
Kqlmagic	Runs KQL queries from a notebook cell against Sentinel data, as described in Microsoft's Sentinel notebooks documentation	Open source, Microsoft
Threat Hunter Playbook and Security Datasets	Community detection notebooks organised by MITRE ATT&CK, with pre-recorded malicious and benign datasets to run them against, locally or through BinderHub	MIT, Open Threat Research (OTRF)

MSTICPy building blocks

QueryProvider: created by naming a data environment, for example QueryProvider("MSSentinel") or QueryProvider("Splunk"), then connect(). The documented environments include MSSentinel (with LogAnalytics and AzureSentinel as aliases), Kusto, MDE, M365D, Splunk, Sumologic, ResourceGraph, Cybereason, LocalData and Mordor. The list changes between releases, so check list_data_environments() on your installed version.
Query library: list_queries() returns named queries in the form Family.query_name (for example SecurityAlert.list_alerts), search() finds them by table or parameter, and every query returns a DataFrame. Ad hoc queries run through exec_query(); long time ranges can be chunked with split_query_by on predefined queries.
Custom queries: add_custom_query() registers your own parameterised query so it is called like the built-ins, which is how a team turns a hunt query into a reusable function.
Multiple instances: add_connection() attaches several workspaces or tenants of the same environment, and queries then run across all of them and return one DataFrame, with up to four threads by default (max_threads).
TILookup: queries configured threat intelligence providers. The documentation lists VirusTotal, AlienVault OTX, RiskIQ, IBM X-Force, Microsoft Sentinel TI, GreyNoise, CrowdSec, AbuseIPDB and ScanMalware, plus pseudo-TI sources such as Tor exit nodes. lookup_iocs() takes a list or DataFrame and returns results as a DataFrame.
Pivot functions: wrappers attached to entities (IpAddress, Host, Account, Url, Dns) so you can call, for example, IpAddress.util.whois() or a provider query on a Host. The mp_pivot pandas accessor chains steps into a pipeline, and pipelines can be saved as YAML and rerun.

Defender use cases

Notebooks earn their place where the portal query language runs out: joining sources, enrichment at scale, statistics and repeatable investigation narratives. Typical hunts that teams build as notebooks:

Hunt	Data and notebook steps	ATT&CK technique
Password spraying against cloud sign-ins	Pull failed sign-ins per source IP across all tenants, count distinct accounts per IP per hour, enrich top IPs with TILookup and WHOIS, plot on a timeline	T1110.003 Password Spraying
Abnormal use of cloud or service accounts	Baseline sign-in locations and applications per account, flag new ASNs or apps, pivot from account to audit log changes	T1078.004 Cloud Accounts
Stolen application tokens	Compare token use by client app and IP against the issuing session, flag tokens replayed from new infrastructure	T1550.001 Application Access Token
Persistence through new credentials or roles	List credential additions and role grants on applications and service principals, join to who made the change and from where	T1098 Account Manipulation
LSASS access on endpoints	Query endpoint process access events targeting lsass.exe, exclude known security tooling by signer, build process trees for the remainder	T1003.001 LSASS Memory
Unusual RDP lateral movement	Build a host-to-host logon graph from Windows logon events (logon type 10), flag first-seen pairs and off-hours sessions	T1021.001 Remote Desktop Protocol

Guided investigation: Microsoft ships ready-to-use Sentinel notebooks such as Guided Investigation - Process Alerts and Credential Scan on Azure Log Analytics, and the Azure-Sentinel-Notebooks repository holds scenario notebooks and MSTICPy feature tutorials to adapt.
Training and validation: Threat Hunter Playbook notebooks run against Security Datasets recordings, so hunters can practise and check analytic logic without production data.
Identity hunts: many of the highest value notebook hunts are about credentials, tokens and service principals. The Identity Threat Hunting Guide lists hypotheses worth building as notebooks, and the KQL for Threat Hunting Guide and Splunk SPL for Threat Hunting Guide cover the query side.

Deployment and safe use

Choosing where notebooks run

Option	Fits	Trade-offs
Local JupyterLab on an analyst workstation	Small teams, ad hoc hunts, offline datasets	Data lands on laptops; hard to standardise packages; no central audit
JupyterHub on internal infrastructure	Teams that need shared environments, central authentication and fixed images	You own patching, authentication and storage; see the Jupyter Server Security Guide
Managed cloud notebooks (for example Sentinel notebooks on Azure Machine Learning)	Teams already on that SIEM who want access tied to cloud roles	Needs both a Sentinel role (Reader, Responder or Contributor) and Azure Machine Learning workspace access; one compute instance can run many notebooks, so heavy jobs affect others
Isolated or air-gapped analysis host	Regulated environments and malware-adjacent work	Packages must be moved as wheels; MSTICPy documents a `pip download` and wheel install method for this

Microsoft has announced that after 31 March 2027 Microsoft Sentinel will be available only in the Microsoft Defender portal, so teams building Sentinel notebook workflows should check how access paths change during that transition.

Rolling it out

Build a pinned environment. Install msticpy with only the extras you need (documented extras include azure, kql, sentinel, splunk, keyvault, ml), pin versions in a lock file and bake them into an image. Treat it like any Python supply chain; see the Python Packaging and pip Security Guide.
Centralise configuration. Point every analyst at one reviewed msticpyconfig.yaml. MSTICPy reads the path in the MSTICPYCONFIG environment variable first, then the current directory, then .msticpy under the user's home directory.
Remove secrets from the file. Replace plaintext AuthKey and ApiID values with EnvironmentVar references or KeyVault references; the config2kv.py tool migrates existing values. Keyring caching is on by default and cached values are not synced with Key Vault, so use refresh_keyring after a rotation.
Give the notebook its own identity. Scheduled runs should authenticate as a dedicated service principal or managed identity with read-only access to the log workspaces, not as a hunter's personal account.
Template the hunt. Use a standard layout: hypothesis and ATT&CK mapping in markdown, a parameters cell (time window, tenant, entity list), data retrieval, enrichment, analysis, findings, and next action.
Pilot on recorded data. Run the template against Security Datasets or a lab workspace before production, and compare results with a known expected output.
Schedule with papermill. papermill inserts an injected-parameters cell after the cell tagged parameters and executes the notebook, for example papermill hunt.ipynb out/2026-11-08.ipynb -p days 7. Keep the output notebooks as run evidence with restricted access.
Roll back safely. Keep notebooks and environment images in version control so a broken library upgrade can be reverted to the last known good image and notebook commit.

Handling data and credentials

Outputs are evidence and also leaks: a saved notebook contains query results, usernames, IPs and sometimes tokens. Use nbstripout as a Git filter or pre-commit hook so outputs never reach the repository, and run a secrets scanner such as detect-secrets on commits.
Third-party lookups disclose indicators: every TI query sends the indicator to that provider. Decide which providers may receive internal hostnames, user principal names or file hashes from sensitive cases, and disable the rest with disable_provider().
Respect provider quotas: MSTICPy notes providers enforce per-minute or per-hour limits and a 403 response usually means the quota is exceeded; bulk enrichment of thousands of IPs needs filtering first.
Code execution risk: opening and running a notebook runs its code with the analyst's credentials. Only run shared notebooks from reviewed repositories.

Detection and hardening value

A notebook programme should make detection better, not just produce interesting charts. Feed every hunt into one of three outcomes: a new or tuned detection rule, a logging or visibility gap ticket, or a documented negative result with the data range covered. The Detection as Code Guide covers how to move a confirmed hunt query into a tested rule, and the Sigma Rules Guide covers a portable rule format.

What to log about the hunting platform

Authentication to JupyterHub or the managed workspace, and kernel starts per user.
Queries issued by notebook service principals against log workspaces (volume and tables), so a stolen hunting identity stands out.
Changes to msticpyconfig.yaml, the environment image and the shared notebook repository.
Key Vault secret reads by hunting identities.

Metrics

Hunts converted to detections: share of completed hunts that produced a new or tuned rule per quarter.
Repeat rate: number of hunts run more than once through papermill, which shows the library is reused.
Coverage: ATT&CK techniques with at least one maintained hunt notebook.
Time to answer: median time from hypothesis to documented conclusion.
Hygiene: secrets found in notebook repositories (target zero) and notebooks with outputs committed.

Framework mapping

Framework	Reference	How notebook hunting supports it
NIST SP 800-53 Rev 5	RA-10 Threat Hunting	Documented, repeatable hunting capability with records of each hunt
NIST SP 800-53 Rev 5	AU-6 Audit Record Review, Analysis, and Reporting; SI-4 System Monitoring	Analysis of log data beyond rule-based alerting
NIST SP 800-53 Rev 5	IA-5 Authenticator Management; AU-9 Protection of Audit Information	Protecting hunting credentials and the log data notebooks retrieve
NIST CSF 2.0	DE.AE-02, DE.AE-03	Analysing potentially adverse events and correlating information from multiple sources
NIST CSF 2.0	ID.IM-01	Hunt findings drive identified improvements
CIS Controls v8.1	Safeguard 8.11 Conduct Audit Log Reviews	Regular review of logs for anomalies
ISO/IEC 27001:2022	A.8.16 Monitoring activities; A.5.7 Threat intelligence	Monitoring for anomalous behaviour and use of threat intelligence

Alternatives

Approach	Choose it when	Limits compared with notebooks
Native SIEM hunting queries (KQL, SPL) in the portal	The hunt needs one data source and no external enrichment	Less suited to multi-source joins, statistics and narrative documentation
Detection rules in a detection-as-code pipeline	The behaviour is well understood and should alert continuously	Not exploratory; built after the hunt proves the logic
Python scripts without notebooks	The analysis is stable and runs headless	Loses the inline explanation and visual output that makes review easy
Collaborative timeline and forensic tools	The work is a single incident timeline with many investigators	Less suited to broad statistical hunts across months of data

Small teams get most value from local JupyterLab with MSTICPy, a shared Git repository and nbstripout. Larger SOCs should centralise on JupyterHub or managed notebooks with single sign-on, fixed images, service principal identities for scheduled runs, and a review step before a notebook enters the shared library. For tool options beyond notebooks, see the Open Source SIEM and Detection Tools Landscape Guide.

Common failure patterns

API keys pasted into cells: the key ends up in the saved notebook, then in Git history. Use EnvironmentVar or KeyVault references from the start.
Personal accounts on schedules: a papermill job running as a hunter breaks when they leave and gives the job their full rights.
Hunts that never graduate: the same notebook is rerun manually for months instead of becoming a rule.
Unpinned libraries: a minor MSTICPy or pandas upgrade changes query names or output columns and silently breaks scheduled hunts.
Uncontrolled enrichment: internal identifiers are sent to every configured TI provider during a sensitive investigation.
Outputs shared as files: notebooks with customer data are emailed or committed because nobody stripped outputs.

Practitioner checklist

Pin MSTICPy, papermill and dependencies in a reviewed image with only the extras you use.
Use one central msticpyconfig.yaml with all secrets as Key Vault or environment variable references.
Run scheduled hunts as a dedicated read-only service principal or managed identity.
Adopt a hunt template with hypothesis, ATT&CK ID, parameters cell, findings and next action.
Install nbstripout and a secrets scanner on every notebook repository.
Decide which TI providers may receive which indicator types and disable the rest.
Test each hunt against Security Datasets or a lab workspace before production use.
Log platform authentication, hunting identity queries and configuration changes.
Review the hunt library quarterly and convert repeatable hunts into detection rules.

Jupyter Notebooks for Threat Hunting: Repeatable Hunts With Python and MSTICPy