Linux Incident Response Triage: Processes, Network, Users, Logs, and Persistence

1. Introduction

When a Linux host is suspected of compromise, the first minutes matter. Responders need a repeatable triage process that captures volatile evidence without unnecessarily destroying context. The goal is not to run every forensic tool immediately, but to answer a few urgent questions: what is running, who is logged in, what network connections exist, what persistence is present, and what changed. This article focuses on defensive administration, auditing, and authorized lab work. The objective is to understand the control, identify misconfiguration before it becomes an incident, and build a repeatable hardening process that can be applied across real Linux environments.

2. How It Works

Linux incident triage combines volatile runtime data with filesystem and log evidence. Memory-resident information such as processes, sockets, logged-in users, namespaces, and open files may disappear after reboot, so responders usually capture it before making disruptive changes. The exact procedure depends on legal, business, and containment requirements. Linux security is layered, so this control should be considered together with identity, filesystem permissions, service configuration, kernel policy, network segmentation, and centralized monitoring. A single command rarely tells the whole story; the effective security posture comes from how these layers interact.

3. The Security Problem

Common mistakes include rebooting too early, killing suspicious processes before collecting context, relying on potentially compromised local tools without validation, and failing to record timestamps. Another failure is collecting huge volumes of data without a clear question, which slows down containment and analysis. These weaknesses are often created gradually by troubleshooting, legacy migrations, vendor installation scripts, or temporary exceptions that later become permanent. A good security review therefore looks not only for obviously dangerous settings, but also for configuration that no longer has a clear owner or business justification.

4. How Attackers Abuse It

Attackers may use temporary directories, scheduled tasks, modified systemd units, SSH keys, shell profiles, hidden processes, or deleted-but-open files for persistence and evasion. A disciplined triage checklist helps defenders identify these patterns quickly while preserving evidence for deeper analysis. In an authorized assessment, the goal is to demonstrate exposure safely and report the exact condition that creates the risk. In production incident response, avoid making destructive changes before collecting enough context to understand whether the issue is a simple misconfiguration, active abuse, or a persistence mechanism.

5. How to Detect / Enumerate & Audit It

Command

date -Is; uptime; who; w

What it does: Capture time, uptime, and interactive users.

What to look for: Unexpected sessions and evidence of recent reboot.

Defensive use: Establish a timeline anchor.

Command

ps auxwwf

What it does: Capture a detailed process tree.

What to look for: Suspicious parents, root shells, binaries from writable paths.

Defensive use: Identify active execution.

Command

ss -plant; ss -planu

What it does: List active TCP/UDP sockets with processes.

What to look for: Unexpected outbound connections and listeners.

Defensive use: Map runtime network activity.

Command

lsof +L1 2>/dev/null | head -n 100

What it does: Find deleted files still held open.

What to look for: Deleted executables or logs that remain in use.

Defensive use: Detect common evasion artifacts.

A useful operational habit is to save the output of these checks in a baseline for each server role. A database server, bastion host, developer workstation, and container node will not look identical, so comparison should be against a known-good peer or documented configuration rather than against a generic assumption.

6. How to Audit the Environment

• Capture volatile data before reboot if policy and safety allow.
• Record exact timestamps and timezone.
• Collect process trees, sockets, logged-in users, mounts, scheduled tasks, recent package changes, and authentication logs.
• Check SSH authorized_keys, cron, systemd units, shell profiles, and startup scripts for persistence.
• Hash and preserve suspicious files before modifying them.
• Document every exception with an owner, reason, and review date.
• Re-check the control after upgrades because packages and deployment automation can silently change defaults.

7. How to Fix / Harden It

• Contain according to incident severity: network isolation may be preferable to shutdown when evidence is valuable.
• Reset credentials only after understanding whether the attacker still has persistence.
• Rebuild from a trusted image when system integrity cannot be established.
• Patch the original entry point before returning to service.
• Document every responder action for the timeline.
• Apply changes through configuration management when possible so the desired state is reproducible.
• Test hardening first on a staging host or a disposable snapshot and verify that critical services still function.

8. Detection & Monitoring

• After containment, increase logging around the affected account, service, and peer systems.
• Hunt for the same indicators across the environment.
• Track recurrence of the same network destinations, file hashes, account activity, or persistence mechanisms.
• Keep time synchronization reliable so host events can be correlated with firewall, identity, EDR, and SIEM data.
• Send high-value security logs off-host because a fully compromised root account can often alter local evidence.

9. MITRE ATT&CK Mapping

MITRE ATT&CK is most useful here as a common language for detection engineering and incident reporting. The mapping does not mean every misconfiguration is an active attack; it shows which attacker behaviors may become easier when the control is weak.

10. Practical Lab

In an isolated Linux lab or disposable VM:

1. Create a benign lab scenario with a background process, listener, test cron job, and added authorized key.
2. Run the triage commands without stopping the artifacts.
3. Build a short timeline from logs and file timestamps.
4. Remove the artifacts only after documenting them.
5. Write a one-page incident note with findings and containment actions.
6. Capture screenshots or command output before and after the change so you can prove what the control actually changed.
7. Revert the VM or remove the test configuration when finished.

11. Key Takeaways

• Linux Incident Response Triage should be treated as part of a layered security model, not as an isolated setting.
• Inventory the current state first, because unknown configuration is difficult to secure.
• Prefer least privilege, explicit ownership, and documented exceptions.
• Monitor changes to the control itself; attackers often try to weaken defenses after gaining access.
• Test fixes in a lab and verify the effective runtime state, not only the configuration file.

Continue learning: Linux Security

This guide is part of our Linux Security series. Related guides in this series cover permissions, sudo, SUID/SGID, Linux capabilities, SSH, PAM, systemd, logging, firewalls, DNS, process security, filesystems, encryption, auditd, kernel hardening, containers, mandatory access control, package security, incident response, and backup protection.