The Sleuth Kit Informer Issue 15
http://www.sleuthkit.org/informer
http://sleuthkit.sourceforge.net/informer
Brian Carrier
carrier at sleuthkit dot org
Issue #15
July 15, 2004
Contents
- Introduction
- What's New?
- Call For Papers
- Partition Recovery With TestDisk (Christophe Grenier)
- File Name Searching In Autopsy
Introduction
This issue of The Sleuth Kit Informer includes an article from Christophe Grenier on using the TestDisk program. TestDisk is an open source tool that can be used recover partitions and file systems when the partition table is missing or corrupt. This issue also has a short article on the new file name searching feature of Autopsy 2.01.
What's New?
On June 2, 2004, new releases of The Sleuth Kit and Autopsy were released. TSK 1.70 included a bug fix for setting the allocation status of some FAT file names and many design changes and output for the FAT support. More output for 'istat' and 'fsstat' now exist so that you can extract more data from the file system. Automated file recovery was also added, which was discussed in the previous issue of The Informer.
http://www.sleuthkit.org/sleuthkit/
Autopsy 2.01 fixed a bug that would return the wrong data unit when an unallocated space image (a dls image) was being examined. This bug was introduced in the last version because the syntax for the 'dcat' tool changed and Autopsy was not properly updated. Autopsy also added support for the new file recovery features.
http://www.sleuthkit.org/autopsy/
Call For Papers
The Sleuth Kit Informer is looking for articles on open source tools and techniques for digital investigations (computer / digital forensics) and incident response. Articles that discuss The Sleuth Kit and Autopsy are appreciated, but not required. Example topics include (but are not limited to):
- Tutorials on open source tools
- User experiences and reviews of open source tools
- New investigation techniques using open source tools
- Open source tool testing results
http://www.sleuthkit.org/informer/cfp.html
Partition Recovery With TestDisk
Christophe Grenier
grenier at cgsecurity dot org
Overview
TestDisk is a tool that can help recover lost partitions and make non-booting disks bootable again when the partition table has been damaged by faulty software, viruses, or human error. It can locate partitions and rebuild the DOS partition table accordingly and it can repair FAT and NTFS boot sectors.
TestDisk is open source software and licensed under the GNU Public License. It runs under DOS (either real or in a Win9x DOS-box), Windows (NT 4/2000/XP/2003), Linux or FreeBSD. The tool has the same functionality on all platforms, but the Windows version is really slow, so I recommend using the DOS or Linux version. Regardless of the platform that TestDisk is run from, it will try to recover DOS partitions that contain the following types of file systems:
- BeFS ( BeOS )
- DOS/Windows FAT12, FAT16 and FAT32
- Ext2 and Ext3 ( Linux)
- Linux Swap (versions 1 and 2)
- Netware NSS
- NTFS ( Windows NT/2K/XP/... )
- ReiserFS ( versions 1 and 2 )
- JFS
- FreeBSD/OpenBSD/NetBSD disklabels
Running TestDisk
TestDisk can be run on either a hard disk or a disk image. When run on an actual disk, TestDisk queries the BIOS (DOS/Win9x) or the OS (Linux, FreeBSD, Windows) to locate the disk and determine its characteristics (LBA size and CHS geometry). The system must be able to recognize the hard disk and its correct size. The TestDisk documentation contains the limitations that some OSes have with recognizing the hard disks.
If the hard disk is damaged or you have already acquired the disk, you can use TestDisk to analyze a disk copy. To acquire a disk, you can use dd (See Sleuth Kit Informer #11). TestDisk can write to the disk image, so a copy should be made before it is used for forensics. Although, there are no silent writes to the disk and you will have to confirm each write operation.
TestDisk is command-line based and uses an ncurses (text) interface. It is run with the disk image name as an argument:
# ./testdisk /debug /log disk_image_hdcThe Geometry menu can be used to specify the disk geometry, such as the number of cylinders, heads, and sectors. These values change the way that TestDisk looks for partitions and calculates their sizes, etc. It does not affect the hard drive or the image itself, unless you actually write data about lost partitions to the drive.
In a normal system, the BIOS often uses a geometry (Cylinder/Heads/Sector) that is different than the physical geometry when it accesses data. The i386 partition handling programs very often make partitions that end on cylinder boundaries. A method used by the BIOS is to read the partition table and to guess the number of heads.
The original geometry is important because when the partition table is corrupt or missing then the physical disk geometry may be used to reconstruct the table. It is more difficult for partition recovery utilities to find lost partitions on the hard disk without the correct geometry. This problem affects all platforms.
When working on a disk image, TestDisk tries to auto-detect the geometry by using the ending sector and ending head value from the entries in the Master Boot Record (MBR). If it doesn't work, you can try 255, 16, 32, 64, 128 and 240 heads until the correct geometry is found. The TestDisk logfile may contain some hints about the relevant geometry.
After the geometry is determined, it is time to analyze the disk, which occurs in two phases. First, TestDisk checks the partition table and does fast file systems checks for each partition. During the second phase, TestDisk searches for lost partitions.
Partition Table Sanity Checks
TestDisk's analysis does a quick check of the partition structures (MBR table and extended partitions). The MBR is limited to 4 entries and one of the entries can be an extended partition allowing several logical partitions. Each logical partition is contained by an extended partition/container. The partition table in the MBR and each extended partition must end with the two bytes of 0x55 and 0xAA. A partition entry contains the following fields:
- beginning of partition in CHS
- end of partition in CHS
- filesystem type
- logical start
- size in sectors
- boot flag
Only one primary partition must have the flag bootable set. CHS information storage is limited to a maximum of 1024 cylinders (0-1023), it's the famous 8Gb limitation (1024*255*63=16450560 sectors). Modern OS and BIOS uses LBA mode to access the data, but FAT12/16/32 boot sectors still refer to CHS geometry. TestDisk checks that each value is in a valid range. For example, it checks that CHS sector 0 is not used (the first sector is sector 1) and the CHS sector number is less than the number of sectors per head. TestDisk also checks if the logical address values match the CHS address values, if the partition ends after the end of the disk, and if any partitions are overlapping each other.
Filesystem Sanity Checks
Following the partition table check, TestDisk runs some basic checks on the boot sector/superblock of each filesystem. As EXT2, EXT3, REISERFS, and JFS share the same partition type, number 0x83, TestDisk has to do additional checks for some filesystems.
Examples of sanity checks include checking for magic values or signatures. For example, FAT and NTFS have 0xAA55 at 0x1FE of the boot sector. Another check is of coherent values, such as free_blocks_count being lower than blocks_count in EXT2FS. This phase is very quick as checks are minimal.
Partition Recovery
In the second step, TestDisk searches for lost partitions. It does not use any data from the previous analysis step. It scans all relevant drive cylinders for the existence of a file system and partition. A primary partition may start at the beginning of a cylinder (head=0, sector=1), while a logical partition starts a little further along (head=1, sector=1).
For each possible partition starting location, TestDisk can search for the presence of a file system header (FAT or NTFS boot sector, EXT2/EXT3 superblock, BSD disklabel ...). The file system header will be used to identify the type and starting location of the partition. Thus, the size of a partition is determined directly from its structure on the disk. Each partition that TestDisk discovers is added to a list of found partitions.
Consider how a FAT32 partition is located. TestDisk searches for a 0xAA55 value and the signature FAT32, it also runs the corresponding general FAT filesystem checks:
- jump signature must be 0xeb 0x90 or 0xe0
- sector size is 512
- cluster size must be 1, 2, 4, 8, 16, 32, 64 or 128
- there must be 2 FAT copies
- the media must be 0xF8 (no other value is seen, it's an obsolete feature)
- If you follow MS guideline, the signature "FAT32" is meaningless but your filesystem should have it.
- Following the number of cluster, TestDisk can determine the kind of FAT (number of cluster is more or equal to 65525 for a FAT32).
Some specific checks for FAT32 are done:
- the root cluster number must be between 2 and the maximum cluster number,
- some obsolete values (number of directly entries, 16-bit partition size) must be set to 0,
- FAT32 version (unused) must be 0.0
To detect a NTFS partition, TestDisk searches for a 0x0xAA55 endmark and the signature NTFS, it also checks that some FAT specific values are set to 0: number of reserved sector, number of fat, directories entries number, 16-bit size of filesystem, 32-bit size of filesystem, number of sectors per cluster.
For FAT and NTFS filesystem, the size of the partition will be determined by using the data in the boot sector.
A list of possible partitions will be given and it is up to you to look it over and select the one(s) which should be recovered. In some cases, especially after initiating a detailed search for lost partitions, TestDisk may show partitions from previous installations. To help determine the contents of a partition, TestDisk can list the files in a FAT partition and also EXT2/EXT3, RFS and NTFS partitions (if TestDisk was compiled with the corresponding libraries). It is useful to determine which partitions to recover when they do not have a meaningful label.
If some of the partitions were not found, then you can start a deeper analysis where TestDisk will check each cylinder and search for FAT32 backup boot sectors, NTFS backup boot sectors, and EXT2/EXT3 backup superblocks. You can also manually add a partition to be recovered. Nothing will be written to the disk or image file unless you choose the 'Write' option and confirm it.
FAT and NTFS Boot Sector Recovery
If FAT32 or NTFS boot sector has been corrupted, TestDisk will let you use the backup boot sector. If no backup boot sector is available then TestDisk can rebuild it by searching for the missing values. To rebuild a FAT boot sector, it will search for the locations of the two copies of the FAT and the root directory information. In some case, it will be able to work even if some structures are missing or corrupted. To rebuild an NTFS boot sector, it will search the MFT and its backup. These functionalities are available in the Advanced Menu.
Other Recovery Functions
To deal with a very badly corrupted FAT32 file system (i.e. a file system that most filesystem checkers will refuse to work on), two additional (but very dangerous) functions were added to TestDisk 5.3. The first function allows TestDisk to remove invalid values in the FAT. For example, it will replace the FAT entry if it is for a cluster that is bigger than the total number of clusters. With the second function, TestDisk can rebuild the root directory. In both cases, it is recommended that you work on a hard disk image. Even if it is corrupted, Linux has been able to mount this file system and recover some files.
TestDisk Limitations
TestDisk can not undelete a file or repair filesystem because it is not its job. TestDisk has been designed for partition recovery but it can not recover a partition that contained a filesystem that it doesn't know about. The user has to manually add an unrecognized partition. TestDisk can write to an image, but the user is prompted before doing so.
Conclusions
In forensics, TestDisk can be useful if the hacker has destroyed the partition table before the system was acquired. After using TestDisk, it is possible to use Autopsy or similar utilities to work at the filesystem level. Since TestDisk 5.4 (not yet public), TestDisk can backup the partition structure and reload it. It has been used to restore an Oracle raw partition.
How To Help The Project
C programmers, people doing data recovery, packagers... are welcome. The TODO list is non empty =;-) If interested, you can contact me at grenier at cgsecurity dot org
References
- TestDisk http://www.cgsecurity.org
- Sleuth Kit Informer Issue #11 - 'dd' Acquisitions http://www.sleuthkit.org/informer/sleuthkit-informer-11.html
File Name Searching In Autopsy
Brian Carrier
The 2.01 version of Autopsy added a new method to search for files. In the 'File Analysis' mode, there is a new text box on the left-hand side with the header "Search For File Name". This feature allows you to use Perl regular expressions to search all file names for a given pattern and the list of hits will be shown in the upper right, where directory contents are typically listed.
For example, if I entered the string "badfile" then any file that had the string "badfile" in it would be found. Examples include "badfile-1.dat" and "abadfileb.txt". The search is done by running 'fls' with the '-r' option and running each file through the Perl regular expression. The comparison is done case insensitive.
You can limit the hits by using some of the regular expression features. If you want the file name to start with a string then prepend the "^" symbol to it. For example, if the file name should start with "pre", then you would enter "^pre" into the search box. Similarly, adding "$" to the end of the string forces the values before it to be the last values in the file name. For example, if you are searching for a specific extension and the file name should end with "jpg", then you would enter "jpg$". To use this feature, "$" must be the last value in the string, otherwise it is assumed to be a normal dollar sign. If you want to search for an actual dollar sign at the end of a string, precede it with "\" ("abc\$").
You can combine both values if you know the full name of the file you are searching for and start the string with "^" and end it with "$". For example, to search for the file name "myfile", then you would use "^myfile$".
One confusing part about using Perl regular expressions is that the '.' symbol has special meanings. It will match any value, so if you search for "myfile.jpg" then the search may find "myfile.jpg" and "myfileQjpg". To restrict it to only a ".", escape it with "\". So, the proper search would be "myfile\.jpg".
This feature allows you to more quickly find files that have a specific name or extension. Once the file has been found, its contents can be examined as normal. Note that if the name of a directory is found by the search, then it will be listed only once in the output and all of its subdirectories and files will not be listed.
Autopsy 2.01 http://www.sleuthkit.org/autopsy/
Copyright © 2004 by Brian Carrier. All Rights Reserved
