Linux Log Analysis: auth.log, journalctl and grep

Linux log analysis is mostly about knowing which file answers which question, and having a handful of commands you trust. You do not need a SIEM to catch an SSH brute force or to see what an intruder did after getting in. A terminal, grep, awk and journalctl are enough.

This is Part 5 of 30 in the Blue Team Learning Path, and it finishes the log-reading block that began with Windows in Parts 3 and 4. We follow one illustrative SSH compromise from the first failed password to the traces the attacker leaves afterward. Run the commands only on machines you own or are authorized to examine.

Key takeaways

• Know the log locations for your distribution: auth.log and syslog on Debian and Ubuntu, secure and messages on Red Hat-family systems.
• journalctl reads the systemd journal and filters by unit, time and priority.
• grep, awk, sort and uniq turn raw lines into counts, which is how you spot patterns.
• Logs can be rotated, lost or edited. Check rotated files and consider remote logging.

Where Linux Logs Live

Logs are text files under /var/log, plus a binary journal managed by systemd. The exact files depend on the distribution:

Source	What it holds	How to read it
/var/log/auth.log (Debian, Ubuntu)	SSH logins, sudo, user and group changes	grep, less
/var/log/secure (RHEL, Fedora, Rocky)	Same role as auth.log	grep, less
/var/log/syslog or /var/log/messages	General system messages, cron activity	grep, less
systemd journal	Everything logged by services and the kernel	journalctl
/var/log/wtmp	Successful logins and reboots, in binary form	last
/var/log/btmp	Failed logins, in binary form	lastb (needs root)
/var/log/audit/audit.log	Detailed events, if auditd is installed	ausearch, grep

For deeper background on how rsyslog and journald feed these files, read Linux logging with journald and rsyslog. If you want auditd-level detail such as file and command auditing, see auditd rules for security monitoring.

journalctl Examples for Investigations

journalctl is the most flexible way to slice logs by time and service. The service name for SSH is ssh on Debian and Ubuntu and sshd on Red Hat-family systems. The journalctl manual lists every option.

# SSH service messages from the last hour
journalctl -u ssh --since "1 hour ago"
# A specific window
journalctl --since "2026-10-27 22:00:00" --until "2026-10-27 23:00:00"
# Errors and worse since the current boot
journalctl -p err -b
# Kernel messages only
journalctl -k
# One entry in full, to see every field
journalctl -u ssh -n 1 -o json-pretty

Reading the whole journal usually needs root or membership in the adm or systemd-journal group. Also note that on some systems the journal is kept only in memory and disappears at reboot, a point we return to in the pitfalls.

Investigating an SSH Brute Force with grep and awk

Here is the illustrative scenario. A web server called web-01 is reported to have a burst of failed SSH logins. The addresses below come from reserved documentation ranges, so none of them are real hosts. [SCREENSHOT: the same commands run against your own lab VM after you generate failed SSH logins against it]

Step 1: count failed attempts per source address. The address is always the fourth field from the end of a Failed password line, whether or not the username exists:

grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head

Step 2: see which usernames were targeted. Attackers often try root, admin and service names:

grep "Failed password" /var/log/auth.log | sed -E 's/.*for (invalid user )?([^ ]+) from.*/\2/' | sort | uniq -c | sort -rn | head

Step 3: did the busiest address ever succeed?

grep "Accepted" /var/log/auth.log | grep 203.0.113.50

Both password and publickey logins are logged as Accepted lines, so read the method in each one. If the busy address has a success, you have moved from noise to a possible compromise.

Step 4: include rotated logs. Older entries are in auth.log.1 and compressed files, and zgrep reads both plain and gzip files:

zgrep "Failed password" /var/log/auth.log*

Step 5: cross-check with the login records. The binary files give a second opinion that is harder to confuse with text tricks:

last -i -n 20
sudo lastb -n 20

This pattern matches password guessing, T1110.001, followed by use of a legitimate remote service, SSH (T1021.004). For the OpenSSH side, the sshd manual explains the logging options.

What Did the Attacker Do After Logging In?

If a login succeeded, switch from “who got in” to “what changed”. Plain log files give you several leads:

# Accounts or groups created
grep -E "useradd|new user|new group" /var/log/auth.log
# Commands run with sudo
grep "sudo:" /var/log/auth.log
# Cron activity (Debian and Ubuntu)
grep CRON /var/log/syslog
# SSH keys added for a user (a file check, not a log)
ls -l --time-style=full-iso /home/*/.ssh/authorized_keys

The last command is a good reminder that logs are not everything. Dropping a key into authorized_keys leaves no authentication log line by itself. If you need that kind of evidence, you need file auditing, which is what auditd provides. When the evidence points to a real incident, continue with Linux incident response triage.

Turning the Commands into Detections

Every command above hints at a detection you could automate later in a SIEM:

• More than a set number of failed logins from one address in a few minutes.
• A successful login from an address that earlier produced many failures.
• A successful login by an account that normally never logs in remotely.
• A new user, new sudoers entry or new authorized key shortly after a remote login.

Parts 13 to 17 show how to express these as searches and rules. Part 6 introduces the MITRE ATT&CK framework for defenders, which gives each of these detections a name and a place on the matrix.

Building a Quick Timeline

When a login looks suspicious, build a timeline around the source address before you decide anything. Pull every line that mentions it, including rotated files, and read them in order:

zgrep -h "203.0.113.50" /var/log/auth.log* | less

If you prefer the journal, narrow it to the window and ask for ISO timestamps, which are easier to sort and compare across machines:

journalctl -u ssh --since "2026-10-27 22:00:00" --until "2026-10-27 23:30:00" -o short-iso | grep 203.0.113.50

Write down three moments: when the failures started, when the first success happened, and the first action taken afterwards. Those three timestamps are the spine of your incident ticket, and they tell you how long the intruder had access before anyone noticed.

Keeping a Copy of Your Logs Off the Host

Everything above fails if the attacker deletes the files first. The simplest defence is to forward logs to another machine as they are written. With rsyslog, one line in a file such as /etc/rsyslog.d/90-forward.conf sends all messages to a collector over TCP (the double @ means TCP, a single @ means UDP). The address below is a documentation example:

*.* @@192.0.2.10:514

Check the configuration for errors, restart the service and send a test message:

sudo rsyslogd -N1
sudo systemctl restart rsyslog
logger "test message from web-01"

Your collector must be configured to listen, and you should test the whole path before relying on it. In the later SIEM Parts you will replace this simple forwarding with agents, but the principle does not change: the evidence must live somewhere the intruder cannot reach.

Common Pitfalls

• Reading the wrong file. auth.log does not exist on Red Hat-family systems. Use /var/log/secure there, or journalctl.
• Forgetting rotated logs. The event you need may sit in auth.log.1 or a compressed archive. Use zgrep over the wildcard.
• Volatile journals. If /var/log/journal does not exist, the journal may be kept in memory only and lost on reboot. Check before an incident, not during.
• Trusting logs on a compromised host. A privileged attacker can edit or delete log files. Forward logs to a separate system so a copy survives.
• Counting addresses behind a proxy. If connections arrive through a NAT or bastion, one address may represent many users.
• Time confusion. Timestamp formats and time zones differ between systems. Confirm each host’s clock and zone before building a timeline.

Practice Plan

Start two small VMs you own, install and enable SSH on one, and generate a few failed logins from the other. Then run the five steps above and write a short ticket the way Part 1 showed in what is a SOC. Once the commands feel natural, repeat the exercise from the Windows side with the previous Part, the Get-WinEvent guide.

Frequently Asked Questions