Advanced Linux Forensics: Deleted Files, /proc, File Descriptors, and Memory Artifacts

1. Introduction

Advanced Linux Forensics: Deleted Files, /proc, File Descriptors, and Memory Artifacts is an advanced Linux security topic because volatile Linux evidence often survives through open file descriptors and /proc even after deletion. At beginner level, Linux security is often described through users, groups, permissions, sudo, and exposed services. At advanced level, the real security boundary is usually created by several kernel and userspace mechanisms working together. A system may look hardened from one angle while still exposing a dangerous path through another.

For defenders, the goal is not simply to know one command. The important skill is understanding what the feature protects, which assumptions it makes, what happens when those assumptions are broken, and how to verify the effective runtime state. This article focuses on defensive administration, auditing, detection, hardening, and isolated laboratory validation.

2. How It Works

In this topic, volatile Linux evidence often survives through open file descriptors and /proc even after deletion. Linux implements the behavior through a combination of kernel interfaces, process state, configuration files, service-manager settings, filesystem metadata, and runtime policy. That means two machines with apparently similar configuration files may behave differently because of kernel version, distribution defaults, container runtime settings, systemd drop-ins, capabilities, mandatory access control, or inherited process state.

A mature security review therefore checks both configuration and execution. Configuration tells us what administrators intended. Runtime state tells us what the kernel is actually enforcing. Logs and telemetry tell us what processes attempted to do. Package metadata and file integrity tell us whether trusted components have changed.

This distinction is particularly important in modern Linux systems because containers, systemd, namespaces, cgroups, capabilities, seccomp, SELinux or AppArmor, and eBPF may all overlap. Strong defense comes from reducing unnecessary privileges at every layer.

3. The Security Problem

The primary security problem is excessive trust. Administrators often enable broad functionality because it makes deployment easier, leave old exceptions in place after troubleshooting, or copy a configuration from another workload without understanding its consequences. Over time, the host accumulates privileges and access paths that are no longer required.

Another problem is hidden interaction between features. A service may run as a non-root user but still possess a dangerous capability. A container may use namespaces but mount sensitive host directories. A firewall may look restrictive while policy routing sends traffic through another interface. A binary may have safe Unix permissions but an ACL grants additional access.

Attackers search for these inconsistencies because they provide paths around the control administrators think is protecting the host.

4. How Attackers Abuse It

After gaining a low-privilege foothold, an attacker normally enumerates the environment before attempting escalation or persistence. For this topic, the attacker is interested in whether volatile Linux evidence often survives through open file descriptors and /proc even after deletion. They look for weak defaults, unnecessary privileges, writable configuration, unexpected inheritance, and differences between intended and actual state.

The attacker does not need every control to be weak. One broken link in the chain may be enough. For example, strong file permissions are irrelevant if a privileged process loads code from a writable location. Network isolation is weakened if a workload shares the host network namespace. A hardened service can still expose secrets through an unrestricted core dump.

In an authorized assessment, these conditions should be demonstrated carefully in a lab or explicitly scoped environment. The goal is to prove impact without damaging production systems.

5. How to Detect / Enumerate & Audit It

Command

lsof +L1

What it does: Provides a primary view of the feature or its current runtime state.

What to look for: Unexpected privilege, unusual configuration, objects that do not match the documented server role, or settings that are broader than necessary.

Defensive use: Establishes a baseline and identifies systems that require deeper inspection.

Command

find /proc/[0-9]*/fd -lname '*deleted*' -ls 2>/dev/null | head

What it does: Provides a second view that helps validate the first result from another Linux interface.

What to look for: Differences between configuration and runtime state, unfamiliar objects, and evidence of local customization.

Defensive use: Cross-checks the environment instead of relying on a single command.

Command

readlink /proc/<PID>/exe

What it does: Examines another security-relevant part of the same subsystem.

What to look for: Privilege that is not required by the workload, unexpected changes, and objects created outside the normal deployment process.

Defensive use: Helps defenders understand the complete security boundary.

Command

cat /proc/<PID>/maps | head

What it does: Provides a deeper inspection or controlled troubleshooting view.

What to look for: Runtime behavior that contradicts the documented design.

Defensive use: Useful during security reviews, troubleshooting, incident response, and isolated labs.

The strongest audits compare multiple sources. Do not automatically trust a single tool when investigating a potentially compromised host. Compare /proc, systemd, kernel interfaces, configuration files, network telemetry, package state, and centralized logs wherever possible.

6. How to Audit the Environment

• Build an inventory of where this feature is used and which services depend on it.
• Define the secure expected state for each server role rather than using one universal configuration.
• Compare production systems with a known-good baseline.
• Identify exceptions and require a documented owner and business justification.
• Review custom scripts, locally built software, container definitions, and systemd drop-ins.
• Check whether user-controlled data can influence privileged behavior.
• Inspect both runtime state and persistent configuration.
• Review the feature again after kernel, distribution, runtime, or application upgrades.
• Include the control in configuration-management and compliance checks.
• Record findings in a format that can be compared over time.

7. How to Fix / Harden It

The hardening strategy should follow least privilege. Remove functionality that the workload does not need rather than trying to monitor every possible misuse of unnecessary functionality. When the feature supports allowlists, bounding sets, namespaces, resource limits, filtering, or explicit trust relationships, make them as narrow as the application allows.

Apply high-impact changes first in a disposable VM or staging environment. Advanced Linux controls can break applications in non-obvious ways because software sometimes depends on undocumented kernel behavior.

Use version-controlled automation for production changes. A manual fix on one server is not a sustainable security control. The desired state should be reproducible, reviewable, and automatically checked for drift.

Finally, verify the effective state after deployment and after reboot. A correct file on disk does not prove that the running service or kernel actually loaded it.

8. Detection & Monitoring

• Alert on changes to the configuration files or runtime objects that define this boundary.
• Monitor newly created privileged services, modules, namespaces, routes, capabilities, or security exceptions where relevant.
• Send important telemetry off-host so local root compromise cannot easily erase all evidence.
• Correlate Linux host events with firewall, identity, VPN, container, cloud, and SIEM data.
• Watch for rare administrative actions outside maintenance windows.
• Establish normal behavior for sensitive servers so unusual changes are easier to identify.
• Investigate unexpected execution from temporary directories, user-controlled paths, or unapproved binaries.
• Monitor attempts to disable or weaken security controls.

9. MITRE ATT&CK Mapping

MITRE ATT&CK is useful here as a common language for detection engineering and reporting. A weak configuration does not automatically mean an attack occurred. The mapping identifies attacker behaviors that become more relevant when the feature is exposed or misconfigured.

10. Practical Lab

In an isolated Linux VM or disposable container lab:

1. Record the default state using the four commands above.
2. Identify which process, configuration file, kernel interface, or service manager controls the feature.
3. Create one harmless configuration change that makes the control more permissive.
4. Observe how the runtime state changes.
5. Restore the secure configuration.
6. Verify the difference after a service restart or reboot.
7. Capture logs or screenshots showing both states.
8. Write a short note explaining the security boundary, the weakness, and the defensive fix.

The most important part of the lab is not merely making the command work. Explain why the behavior changed and which Linux component enforced it.

11. Key Takeaways

• Advanced Linux security depends on interactions between kernel features, processes, services, filesystems, and networks.
• Always compare intended configuration with effective runtime state.
• Reduce privileges and reachable functionality instead of relying only on detection.
• Treat unusual changes to security boundaries as high-value events.
• Use multiple telemetry sources when investigating compromise.
• Maintain role-based baselines so drift becomes visible.
• Test advanced hardening in a lab before production rollout.
• Automate secure configuration wherever possible.

Continue learning: Advanced Linux Security

This guide is part of the Advanced Linux Security series on The Cyber Track. The complete series covers namespaces, cgroups, eBPF, seccomp, advanced capabilities, kernel modules, persistence, dynamic linking, environment abuse, ACLs and xattrs, virtual networking, advanced nftables, routing, memory protections, core dumps, Linux forensics, rootkits, boot security, storage security, and enterprise hardening.