Elastic Security Guide: SIEM, Detection Rules and Endpoint Protection

Elastic Security combines a SIEM, an open detection rule library and an endpoint agent on one Elasticsearch data platform, so a single misconfigured role or API key can expose both the security data and the controls that act on it. Teams adopt it for its search speed, its public rule repository and its pricing flexibility, then discover that detection rules run with the privileges of whoever last edited them, that API keys and service tokens never expire by default, and that AI features send alert data to an external model. This guide explains how Elastic Security is organised in the 9.x release line, how its rule types, ES|QL and Elastic Defend work, and how to secure the roles, API keys and service accounts underneath. It is written for detection engineers, SOC leads and Elastic administrators. General SIEM design and selection are covered in the SIEM Guide and the SIEM Buyer's Guide.

Key takeaways

• Rules inherit their editor's privileges. Each Kibana rule stores an Elasticsearch API key holding a snapshot of the privileges of the user who created or last edited it, so a more privileged editor silently escalates the rule.
• Credentials do not expire unless you make them. API keys created through the API never expire by default and service account tokens never expire at all, so ownership and rotation have to be designed in.
• Seven rule types cover different jobs. Custom query, EQL, ES|QL, threshold, indicator match, new terms and machine learning rules each suit a different detection problem; picking the wrong one causes noise or blind spots.
• The rule library is public and testable. The elastic/detection-rules repository holds the prebuilt rules in TOML with tooling for detections-as-code, so content can be reviewed and promoted like software.
• AI features need data governance. AI Assistant and Attack Discovery send alert and event data to the LLM connector you configure; anonymisation settings decide which fields leave the cluster in clear text.

How Elastic Security is organised

Elastic Security runs in Kibana on top of Elasticsearch, on self-managed clusters, Elastic Cloud Hosted, Elastic Cloud on Kubernetes or as a Serverless security project. As of 24 October 2026 the Elastic Security release notes list 9.5.5 as the latest version, and the Elastic Common Schema (ECS) reference is at 9.5.0. The main components are:

Recent changes that matter: in 9.4 the detection engine began adding METADATA _id to ES|QL rule queries automatically; in 9.5 Attack Discovery moved to workflow-based generation scheduled through the alerting framework, the default chat experience in Security became the AI Agent (the classic AI Assistant can be restored from GenAI settings), and Elastic Defend added DLL masquerading detection on Windows and attribution of processes to AI agents.

Detection rules, ES|QL and content

Rule types

Building block rules generate low-severity alerts that are hidden from the main alert view by default and serve as inputs to other rules, which is Elastic's way of layering weak signals. Alert suppression ("Suppress alerts by") groups duplicates on chosen fields.

ES|QL rules

ES|QL rules set their data sources in the FROM clause, and the user who creates or last edits the rule must be able to read those indices. Aggregating queries use STATS ... BY and produce one alert per result row containing only the grouped fields and computed values. Non-aggregating queries produce one alert per matching event and rely on the document _id to deduplicate; on 9.0 to 9.3 you must add METADATA _id yourself, while 9.4 and later add it at execution time. LIMIT interacts with the rule's maximum alerts per run, and the lower value wins. An illustrative aggregating rule for encoded PowerShell (ATT&CK T1059.001 with T1027 Obfuscated Files or Information):

FROM logs-endpoint.events.process-*
| WHERE event.type == "start" AND process.name == "powershell.exe"
    AND process.command_line LIKE "*-enc*"
| STATS executions = COUNT(*), first_seen = MIN(@timestamp) BY host.name, user.name
| WHERE executions >= 3

Use a custom query rule for simple field matches, EQL for ordered sequences and machine learning for anomaly detection; ES|QL earns its place when you need to compute something first. For query technique, see the Threat Hunting Guide.

Prebuilt rules and detections-as-code

• Repository: elastic/detection-rules holds rules in rules/, building blocks in rules_building_block/ and a hunting query package in hunting/, under the Elastic License v2.
• Tooling: the detection_rules Python module provides create-rule, validate-rule, validate-all, test, and import-rules-to-repo / export-rules-from-repo for round-tripping between Kibana and Git.
• Customising prebuilt rules: any tier can duplicate rules, add exceptions and configure actions. Editing a prebuilt rule in place or reverting it to Elastic's version requires an Enterprise subscription on Elastic Stack 9.1 or later, or Security Analytics Complete on Serverless, and update conflict resolution is Enterprise-only.
• Coverage: prebuilt rules are mapped to ATT&CK; use that mapping to measure coverage against your threat model, not rule count. See the Detection Engineering Guide and the MITRE ATT&CK Guide for Defenders.

Elastic Defend and endpoint protection

Elastic Defend is an integration added to Elastic Agent through Fleet. It cannot run with an agent in standalone mode, and Fleet Server must be running before agents are added. Its preventions cover malware, ransomware, memory threats, malicious behaviour and credential theft.

Choose Data Collection only for a short baselining phase; leaving production endpoints on it gives telemetry without protection. Cloud workload presets suit Linux servers where prevention could disrupt services, but they need compensating detection rules. Turn on the agent self-protection and uninstall prevention features so an attacker with local admin rights cannot simply remove the sensor (ATT&CK T1685 Disable or Modify Tools). See the Endpoint Security Guide and the EDR Evasion and Defence Guide.

How identity and access work in Elastic Security

Elasticsearch authenticates users through realms (native, LDAP, Active Directory, SAML, OpenID Connect, PKI and others) and authorises them through roles. A role combines cluster privileges, index privileges and Kibana feature privileges, which are granted per Kibana space. Users with several roles receive the union of all their privileges, so combinations need review.

• Index privileges are the data boundary: read on logs-* or .alerts-security* decides what a user, and every rule that user edits, can see.
• Kibana feature privileges control the workflow: Security features such as rules, alerts, cases, timelines and AI Assistant can be set to None, Read or All, with sub-feature privileges for finer control.
• Spaces separate teams: use them to keep, for example, a managed service provider's analysts away from internal investigation cases.
• Rule authorisation follows the editor: when a rule is created or edited, Kibana creates an API key capturing that user's privileges. If a user lacking the required privileges edits a rule it stops working; if a more privileged user edits it, it runs with more privilege. Imported rules get a new key when enabled, and keys can be regenerated with Update API key.

Admin roles and privileged access

• Build three custom roles at minimum: analyst (alerts and cases All, rules Read), detection engineer (rules All, alerts Read) and platform administrator. This gives separation of duties and stops analysts disabling the rules they are measured on.
• Use a dedicated rule-owner identity: because rules inherit their editor's privileges, have production rules deployed through a pipeline under a service identity with exactly the index read rights the rules need, then regenerate keys after role changes.
• Put human administrators behind SSO and MFA: map SAML or OIDC groups to roles, and keep the built-in elastic superuser credential vaulted for emergencies. See the Privileged Access Management Guide.

Integrations, API tokens and service accounts

Elastic deployments run on non-human credentials: agents, Fleet Server, Kibana, Logstash, SOAR tools, ticketing systems and LLM connectors. See the Service Account Security Guide.

The default "never expires" behaviour is the most common NHI gap in Elastic estates: keys created for a proof of concept survive for years with the privileges of whoever created them. Inventory keys regularly through Stack Management, where manage_api_key holders can see all users' keys, and invalidate any without an owner. A stolen key or token can be replayed from anywhere (ATT&CK T1528 Steal Application Access Token, T1552.001 Credentials In Files). See the API Key Management Guide and the Secrets Management Guide.

AI connectors: AI Assistant and Attack Discovery send alert or event data, rule configurations, queries and prompts to the connected LLM provider. On the Anonymization tab of Security AI settings, only fields marked Allowed are sent, and allowed fields marked Anonymized are replaced with placeholders. Changing these settings requires the Elastic AI Assistant: All privilege with sub-feature customisation, so restrict it. See the AI SOC Guide.

Logging and detection

• Enable audit logging on both tiers: set xpack.security.audit.enabled: true in elasticsearch.yml and, separately, in kibana.yml. Elasticsearch writes <clustername>_audit.json on each node; from 9.5 the setting is dynamic. Availability depends on subscription level, so confirm it before relying on it.
• Ship audit logs out of reach: send them to a separate monitoring deployment that cluster administrators cannot modify, so an attacker with superuser cannot erase their trail.
• Detect credential misuse: alert on API keys or service tokens used from new source addresses, new API key creation with broad role descriptors, and authentication failures for service accounts (ATT&CK T1078 Valid Accounts).
• Detect control tampering: alert on rules disabled or deleted, exceptions added to high-severity rules, Elastic Defend policies switched to a non-prevention preset and agents going silent.
• Use entity analytics for NHIs: include service accounts in user risk scoring so unusual hosts or times for a service identity raise its score.

Metrics that show it is working

Hardening baseline

1. Inventory first: list deployments, Kibana spaces, roles, API keys, service tokens, connectors and rules, with an owner for each.
2. Lock down administration: SSO with MFA for people, superuser for break-glass only, custom least-privilege roles for analysts and engineers.
3. Fix credential defaults: set expiry and role descriptors on all new API keys, replace keys without them, and delete service tokens that no longer map to a running service.
4. Turn on audit logging on Elasticsearch and Kibana and ship it to a separate deployment.
5. Pilot rules safely: enable new or changed rules in a test space or with actions off, preview them against historical data, then enable notifications. Roll back by disabling the rule or restoring its previous TOML from Git.
6. Roll out Elastic Defend in rings: Data Collection on a pilot group for baselining, then Essential or Complete EDR with prevention, adding trusted applications and exceptions as false positives appear. Roll back a ring by assigning its previous policy.
7. Gate AI features: choose the LLM connector deliberately, review anonymisation fields before enabling Attack Discovery schedules, and document the data flow for privacy review.

Small teams on Elastic Cloud or Serverless should enable prebuilt rules for their data sources and avoid editing them in place, so updates apply cleanly. Large SOCs should run detections-as-code from elastic/detection-rules, deploy through a pipeline identity and use spaces to separate tenants.

Common misconfigurations

• Rules edited by a superuser: the rule now runs with full cluster privileges, and nobody notices until the editor's account is removed or downgraded and the rule silently fails.
• Never-expiring API keys from pilots: keys created for integration tests keep their creator's broad privileges for years. The LastPass breach shows how long-lived secrets in a DevOps engineer's reach can unlock an entire environment.
• ES|QL rules without _id on older versions: non-aggregating rules on 9.0 to 9.3 produce duplicate alerts, or a query that drops _id is saved with errors.
• Endpoints left on Data Collection: the console shows healthy agents, but nothing is prevented.
• AI anonymisation left at defaults: user names, host names and command lines go to a third-party model in clear text without a privacy decision.
• Audit logs on the same cluster: an attacker with administrative rights can remove the evidence of what they did.

Framework mapping

Practitioner checklist

• Record the Elastic Stack version and check ES|QL rule behaviour against it (9.4 and later add _id automatically).
• Deploy production rules under a dedicated identity and regenerate rule API keys after role changes.
• Replace editor and superuser use with custom analyst, engineer and admin roles.
• Set expiry and role descriptors on every API key, and delete keys without an owner.
• Keep one named service token per service instance and delete tokens on decommission.
• Enable Elasticsearch and Kibana audit logging and ship it to a separate deployment.
• Move endpoints off Data Collection to a prevention preset, with self-protection on.
• Manage custom rules as code with elastic/detection-rules tooling and review in Git.
• Review AI anonymisation fields and the LLM connector before enabling Attack Discovery schedules.

Standards and references

• Elastic: About detection rules
• Elastic: ES|QL rules
• Elastic: Alerting setup and rule authorisation
• Elastic: Create an API key API
• Elastic: Service accounts
• Elastic: Install Elastic Defend
• Elastic: detection-rules repository
• NIST: SP 800-53 Rev. 5

Related NHI Mgmt Group resources: SIEM Guide · Detection Engineering Guide · Splunk Enterprise Security Guide · Microsoft Sentinel Guide · Security Data Lake Guide