Home > Uncategorized > Sleuthkit’s ‘hfind’ and the NSRL hash data sets.

Sleuthkit’s ‘hfind’ and the NSRL hash data sets.

Sleuthkit provides the hfind or hash find tool to index and query the NSRL hash database of known good and known bad files and their corresponding hashes. The hash sets are currently available as four ISO files which can be downloaded from here

I downloaded the four iso files from nsrl but rather than burn to CD/DVD I simply mounted them on my Linux workstation and manually copied out the relevant rds*.zip files.

The four downloaded files are:

RDS_228_A.iso
RDS_228_B.iso
RDS_228_C.iso
RDS_228_D.iso

At the time of downloading they are each approx 350MB.

On my Linux workstation I mount the files individually while copying out the zip files.

# mount -t iso9660 -o loop /media/NSRL/RDS_228_A.iso /media/iso
# mount -t iso9660 -o loop /media/NSRL/RDS_228_B.iso /media/iso
# mount -t iso9660 -o loop /media/NSRL/RDS_228_C.iso /media/iso
# mount -t iso9660 -o loop /media/NSRL/RDS_228_D.iso /media/iso

The following four files are copied out of the ISOs. Once unzipped each will contain a file called NSRLFile.txt.

rds_228_a.zip
rds_228_b.zip
rds_228_c.zip
rds_228_d.zip

We then rename each to allow us to concat them together.

root@balldav-laptop:/media/NSRL# ls -l
total 7133016
-rwxrwxrwx 1 balldav balldav 1820912930 2010-03-19 23:17 NSRLFile_a.txt
-rwxrwxrwx 1 balldav balldav 1827880365 2010-03-19 23:19 NSRLFile_b.txt
-rwxrwxrwx 1 balldav balldav 1818817061 2010-03-19 23:20 NSRLFile_c.txt
-rwxrwxrwx 1 balldav balldav 1836589193 2010-03-19 23:22 NSRLFile_d.txt
root@balldav-laptop:/media/NSRL#

We then concatenate the four NSRLFile.txt files into one.

root@balldav-laptop:/media/NSRL# cat NSRLFile_a.txt NSRLFile_b.txt NSRLFile_c.txt NSRLFile_d.txt >> NSRLFile.txt
root@balldav-laptop:/media/NSRL#

The resulting file is 7GBs.

root@balldav-laptop:/media/NSRL# ls -l NSRLFile.txt
-rwxrwxrwx 1 balldav balldav 7304199549 2010-06-30 12:18 NSRLFile.txt
root@balldav-laptop:/media/NSRL#

I had heard that there were duplicate hashes across ISOs but I didn’t see any dupes.

root@balldav-laptop:/media/NSRL# cat NSRLFile.txt | wc -l
56974174
root@balldav-laptop:/media/NSRL# cat NSRLFile.txt | uniq | wc -l
56974174
root@balldav-laptop:/media/NSRL#

Before using the NSRL hash database an index file must be created. This allows for binary rather than sequential searches of the data and saves a lot of time.

root@balldav-laptop:/media/NSRL# hfind -i nsrl-md5 /media/NSRL/NSRLFile.txt
Index Created
root@balldav-laptop:/media/NSRL#

The resulting index file is 860MB.

root@balldav-laptop:/media/NSRL# ls -l *.idx
-rwxrwxrwx 1 balldav balldav 860119447 2010-06-30 14:46 NSRLFile.txt-md5.idx
root@balldav-laptop:/media/NSRL#

If we view the resulting .idx file we see the following fields:

root@balldav-laptop:/media/NSRL# cat NSRLFile.txt-md5.idx | more
00000000000000000000000000000000000000000|nsrl
00000238B43AFAF52EB6F9780D25173C|0000002187066656
000003160F7A0B3987C913B62902E379|0000000442813774
000003A20F4478A192448F93094B1984|0000006278182666
[snip]

The first entry is the md5 hash value and the second is the byte offset for the entry in NSRLfile.txt. The byte offset is in decimal. Convert this number to Hex then load up NSRLfile.txt in your favorite Hex Editor and jump to the byte offset to confirm.

root@balldav-laptop:~# echo “ibase=10; obase=16; 0000002187066656” | bc
825BFD20
root@balldav-laptop:~#

So 0000002187066656 decimal is 0x825BFD20 in hexadecimal.

Load the NSRLfile.txt into the Bless Hex Editor. From the menu choose Search/Goto Offset, enter 0x825BFD20 and click “Go to Offset”. I’ve highlighted the entry for the first hash value above which references a file called MOBSYNC.CH_

Bless Hex Editor

The corresponding NSRLFile.txt entries for MOBSYNC.CH_ look like the following:

root@balldav-laptop:/# cat NSRLFile.txt | grep 00000238B43AFAF52EB6F9780D25173C
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,10346,”358″,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,15989,”358″,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,1672,”WIN2000″,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,2553,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,2563,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,2619,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,2633,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,2939,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,2964,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,3291,”WIN2000″,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,4968,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,4987,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,5417,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,5924,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,6524,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,8081,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,8097,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,8106,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,8113,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,8122,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,8264,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,8276,”WIN”,””
“4CDE504FCD0E0038156ABA9F7F60E6EB6266BF4D”,”00000238B43AFAF52EB6F9780D25173C”,”051E410B”,”MOBSYNC.CH_”,11312,8291,”WIN”,””
root@balldav-laptop:/#

From left to right the different fields are as follows:

sha-1 hash,
md5 hash,
crc32,
filename,
filesize,
productcode,
opsystemcode,
specialcode.

Since the NSRL hashset is a mix of good and bad hash values a ‘specialcode’ value of “M” refers to the hash of a malicious file.

Sample of opsystemcodes are WIN, WIN95, WIN98, WINNT, WINXP, WIN2000, NT4WKS, Linux, Solaris, Sun and MAC OS 10.3.9+. There are more.

Finally to look up a hash value in the hashset do the following:

root@balldav-laptop:/media/NSRL# hfind NSRLFile.txt 00000238B43AFAF52EB6F9780D25173C
00000238B43AFAF52EB6F9780D25173C MOBSYNC.CH_
root@balldav-laptop:/media/NSRL#

Other command line switches include -q and -f.

The -q option simply outputs a ‘1’ if a match is found and a ‘0’ if no match found. Good for scripting.

root@balldav-laptop:/media/NSRL# hfind -q NSRLFile.txt 00000238B43AFAF52EB6F9780D25173C
1
root@balldav-laptop:/media/NSRL#

The -f option followed by a flat file of hashes will process each entry one by one.

root@balldav-laptop:/media/NSRL# hfind NSRLFile.txt -f input_file
00000238B43AFAF52EB6F9780D25173C MOBSYNC.CH_
000003160F7A0B3987C913B62902E379 iso8859-4.h
000003A20F4478A192448F93094B1984 libpanel-applet2-0_2.8.1-2_i386.cnr
root@balldav-laptop:/media/NSRL#

This is a very basic intro to the functionality in the ‘hfind’ utility in TSK.

Advertisements
Categories: Uncategorized Tags: , ,
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: