Bringing Sexy Back: Linux Live Response

Something I don't do often is perform live response to Linux systems. So I've decided to blog about all of the built up knowledge (or lack of) in my head over the years about this topic and update it when I find other techniques or artifacts may be useful.

Some random "Did you knows" about the EXT4 file system:

• There are no file creation dates until the Fourth Extended File System (EXT4). So if you're examining an EXT3 system, you won't have file creation times :'( 
• EnCase 6 will not read EXT4, best bet is to use Linux to examine Linux. FTK will read EXT4, however, I'm told it may sometimes "fudge" date/times, which is not cool.

Below is my Linux LR Checklist, which I intend to update as time goes on and file systems evolve:


==Currently Logged On Users==
Commands such as "w", "who" and "users" can be used to determine who is actively logged on the system.
Examine the "/var/log/utmp" file for active logins that contain the date/time, username and origin of the login.


==File System==
Go for the jugular and grab a fls listing of the system, starting at the "/" directory. Then run the output against mactime to make things nice and pretty to read date/timestamps.

fls -r -m "/" /dev/sda1 > fls.bodyfile
mactime -b fls.bodyfile -d -y > Victim.timeline

You can view your .timeline file in your examination Linux machine or if it is small enough, MS Excel.


==Network Activity===
Running the "netstat -anp" command will display the status of active network connections, the IP address of the remote systems and the process PID that is using the connection.

The "lsof" command allows you to see a list of all OPEN files and which users are associated with the open files on the system. It also outputs a list of files/processes associated with network addresses. (lsof -i)


==Running Processes==
Use "ps -aux" to output a list of running processes, associated user, start time, and command.

USER       PID %CPU %MEM    VSZ     RSS    TTY      STAT START   TIME COMMAND
root         1           0.0        0.1        24604  2464     ?             Ss      Mar13   0:02   /sbin/init

==Files to Acquire==

• Bash history of all users, including root: sometimes the attackers get lazy, and forget to wipe their bash history (/home/<username>/.bash_history). This is wealth of good information, that is like Shellbags, AppCompat and Terminal history on a Windows system. You can see what commands the attacker gave the system, where they moved throughout the system, any attempted lateral movement and if they created or deleted files (to name a few).

• Any "nohup.out" files – continues script without user being logged on 

• Logwatch logs - /var/spool/mail (will show file creations, changes, SSH logons)

• Auditd Logs - is a subsystem of the Linux kernel. It is like the AppCompat cache in Windows but on heavy steroids. /var/log/audit (will show program executions, date/time, user)

• Viminfo – can check for evidence of attacker modifying files with vim.  Quick war story: I found in .viminfo the attacker was accessing GCC to compile custom malcode on victim system.

• Service start-up scripts in:

/etc/inittab, /etc/init.d, /etc/rc.d (traditional)

/etc/init.conf, /etc/init  (Upstart)

• Scheduled tasks ("cronjobs")

/etc/cron*

/var/spool/cron/*

• Web Logs

• Poor man's forensics is to acquire all of the logs in "/var/log" and keyword search all of the keywords or dates you have so far in your investigation. Yes it is cheesy, but hey if the end result is you found evil or more evil, that's what matters.

==Basic System Profiling==

Linux distro name/version number:

/etc/*-release

Installation date:

Look at dates on /etc/

ssh/ssh_host_*_key files

Computer name:

/etc/hostname (also log entries under /var/log)

IP address(es):

/etc/hosts (static assignments)

/var/lib/dhclient, /var/log/* (DHCP)

in redhat and the likes it would be /etc/sysconfig/network-scripts/eth0* or some such

and that would be how you can tell if the box is configured with multiple interfaces, you'd have scripts for eth0-up eth1-up and so on

and in /etc/hosts you typiclly won't see anything if DNS is enabled, which is almost 100% nowadays

Redhat:/etc/sysconfig/networkscripts/

Solaris: /etc/inet/hosts

• /etc/localtime stores default time zone data

• Binary file format:

– Use "zdump" on Linux

– Look for matching file under /

usr/share/zoneinfo