NTFSParser is running into an assertion failure

[florianbuehlmann]

Hi

I'm trying to get the "TestApplication" running but I'm stuck at one point:
In the line "foreach (FileRecord record in parser.GetRecords(true))" (Line 33) of the file "Program.cs" the parser should read all records. However it encounters the error:

NTFSLib.Objects.FileRecord.Parse on Line 69 throws an Assert failure, because the signature is not "FILE" but "\0\0\0\0".

It then crashes on Line 90 because res.USNSizeWords is zero.

This happens on a newly formatted NTFS volume as well as on existing volumes that work just fine in Windows.

Interestingly, the "NTFSWrapper" class seems to have no problem parsing the file system structure, as I can read all existing files (Program.cs after Lines 45).

Maybe a wrong cluster/offset is being read?

[LordMike]

I know there are some issues with the first parse of the mft. For example, if your mft is fragmented just rigt, the program will continue reading past one of the chunks and get garbage data.

I haven't had the time to make a separate ntfs parser just for the initial run of the mfts file record. (At the time we parse that, we have nothing to build the parsers I already have).

Sent from my iPhone

On 19. aug. 2016, at 09.56, florianbuehlmann [email protected] wrote:

Hi

I'm trying to get the "TestApplication" running but I'm stuck at one point:
In the line "foreach (FileRecord record in parser.GetRecords(true))" (Line 33) of the file "Program.cs" the parser should read all records. However it encounters the error:

NTFSLib.Objects.FileRecord.Parse on Line 69 throws an Assert failure, because the signature is not "FILE" but "\0\0\0\0".

This happens on a newly formatted NTFS volume as well as on existing volumes that work just fine in Windows.

Interestingly, the "NTFSWrapper" class seems to have no problem parsing the file system structure, as I can read all existing files (Program.cs after Lines 45).

Maybe a wrong cluster/offset is being read?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[florianbuehlmann]

I see... It's quite hard to find any information about "MFT fragmentation" at all.

I could reproduce the error from a very small USB thumb drive (256MB) formatted with different cluster sizes in windows 10.
Only with a cluster size of 64KB I could not experience the assert failure.

With that thumb drive of 256MB (NTFS formatted, empty) and a cluster size of 4096, I've seen that "FileRecordCount" is set to 256 (NTFSParser.cs line 151 in the loop).
However when looking at $Mft with a hex-editor, I could see that only the first 32 records have the "FILE" signature. The rest of the file is simply zeroed. Exactly after these 32 records the NTFSParser throws this assert failure.
So for some reason when formatting the thumb drive in windows, only the first 32 records are initialized. The NTFSParser class although assumes that "FileRecordCount = streamlength / bytesPrFileRecord" (NTFSParser.cs line 91), so assumes that the whole mft is initialized.

I couldn't find any difference between "Quick format" and "Full format" of the volume though.

[LordMike]

What your describing is exactly fragmentation of the mft file. What happens is that record 0 contains $mft and one of the attributes will tell you the fragments.

The parsers do not react on this. The code simply assumes that the mft is not fragmented. If you were to defrag the mft using tools from either sysinternals (contig) or piriform (defraggler).

In either tool you can defrag the mft.

Sent from my iPhone

On 19. aug. 2016, at 11.36, florianbuehlmann [email protected] wrote:

A quick fix that worked for me:

`
public IEnumerable GetRecords(bool skipUnused = false)
{
if (skipUnused && _usedRecords == null)
{
// Initiate _usedRecords
InitiateRecordBitarray();
}

while (true)
{
if (skipUnused && !_usedRecords[(int)CurrentMftRecordNumber])
{
bool foundFileRecord = false;

    // Skip to the next used record
    for (int i = (int)CurrentMftRecordNumber + 1; i < FileRecordCount; i++)
    {
        CurrentMftRecordNumber = (uint)i;

        if (_usedRecords[i])
            // Use this
            foundFileRecord = true;
            break;
    }

    if (!foundFileRecord) break;
}

FileRecord record = ParseNextRecord();

if (record == null)
    break;

yield return record;

}
}
`

The only thing I changed was addind the "foundFileRecord" bool variable.

I haven't thoroughly tested it, but it seems to find all records I had on my thumb drive.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[florianbuehlmann]

Thanks for your hints regarding the defragmentation of the MFT.
For my purposes defragmenting the MFT on a volume is not possible (I'm working on read-only devices).
This afternoon I had a look at your code but could not really understand what you meant.

I have seen that in NTFSParser the data fragments are being retrieved for the MFT (I assume the data fragments are the fragments you were talking about?), from which are then being read from in NTFSDiskStream (Depending on the read position the associated fragment is being loaded). So to me this looks like it's already parsing the fragments, but I'm probably missing an important step?

Fixing the MFT parsing doesn't look very easy indeed... :-)

Anyways, I fully understand you do not find time to fix it, as I probably don't find time either to have a deeper look at it, so don't worry.

[LordMike]

It isn't quite - that's why I haven't jumped at it :)

You're entirely correct on NTFSDiskStream. If I remember correctly, I read the MFT itself as an NTFSDiskStream, but at the time I instantiate it, I don't have the full overview of the MFT fragments .. I don't have that overview, because to be able to read the MFT's file record (record 0), I need to use the NTFSDiskStream ... (we're now in a chicken/egg problem here).

One way to solve it would be to simply read record 0 using the NtfsFileRecord (iirc) which can take a byte array. In most cases (98%) the MFT is not so fragmented, so all its details are in that one 1024-byte record (or 4096-byte record). In the last 2%, the MFT is very fragmented, and file record 0 will refer to a number of other records - and that's where it gets complicated.

I've been thinking about the latter solution, but it's nontrivial.

[florianbuehlmann]

I think I understand what you are saying but what I observe is still not clear to me.
I did a new test today with a ~1TB drive:
Interestingly, the $Mft file is exactely the same when writing the member _mftStream to a file and comparing it with an extraction I made from an forensics application I know it is works correctly:

_mftStream = new NtfsDiskStream(_diskStream, false, fragments, BytesPrCluster, compressionClusterCount, (long)dataAttribs[0].NonResidentHeader.ContentSize); var fileStream = File.Create("C:\\D_MFT_NTFSLIB"); _mftStream.Seek(0, SeekOrigin.Begin); _mftStream.CopyTo(fileStream); fileStream.Close();

In CMD then:

fc D_MFT D_MFT_NTFSLIB
Comparing files D_MFT and D_MFT_NTFSLIB
FC: no differences encountered

Also I checked that given MFT is really fragmented using "PassMark Fragger":
https://i.imgsafe.org/9d09713e87.png

Stepping through the DataFragments in the code also gives the same LCN and offsets as "PassMark Fragger" says.

Printing GetRecords() FileRecords also outputs valid records in the same number of records as I expect them to be (Also verified file names for plausibility).

From what I understand what you're saying should not work, I should get only the first fragment and only the records of that fragment and then run in an exception when I try to read a file record from the second fragment. However I still get both fragments content and all FileRecords.

For verification purposes, I also copied the contents of _mftStream to a byte buffer, parsing a new FileRecord from that byte buffer and creating a new Stream using that new FileRecord. Contents were the same though.

This is really driving me crazy and probably above my heads.

[LordMike]

Heh. You're following along very nicely :)

Try fragmenting the file "a lot".. The goal here is for the fragments part (the $DATA attribute) for the $MFT record to become larger than the ~800 bytes that are allowed. If that happens, the attribute is siphoned out into a separate NTFS file record somewhere else. The issue should then appear (iirc).

If that's the case, then it's clear to me that the issue happens with external attributes (or non-resident attributes) not being read by the constructors of the MFT parsers. Fragments are okay as long as they're all described in that 1024-byte file record that constitutes $MFT.

[florianbuehlmann]

Okay, here are my observations:

I have managed to fragment the $Mft of a 250MB volume to 404 fragments (Harder than it sounds, since Fragger doesn't support fragmenting the MFT, so I created a few large and a lot of small files on the volume): Screenshot.

With that fragmented MFT, i noticed that I'm getting a different error: Screenshot. I assume this is the error you are talking about?

After this, I noticed that the MFT FileRecord has an additional attribute AttributeList: Screenshot

From that base, I extracted the fragments and appended it to the existing NtfsDiskStream:
NTFSParser
NtfsDiskStream

What I changed:

NTFSParser
Added Lines 103 - 122. These lines basically extract the AttributeList's list-items and parse the fragments from them. They are then added to the existing NtfsDiskStream using the newly-added addFragments method. The code also ensures no fragment is added twice, so only the necessary fragments are added.
Since in the NtfsDiskStream constructor we already give the full details about length and compression, we only needed to add the missing fragments to the existing array (addFragments method).

NtfsDiskStream
I added the addFragments method there to add new fragments to lookup. For this to work, I removed the readonly modifier from the _fragments variable and also changed Stream to NtfsDiskStream in NTFSParser.

After that changes, I dont get the said exception anymore and can read the file tree.

Note: I didn't verify this with any compressed objects.

Maybe this could give you a start? Some code is maybe not written in your style, but you may adapt it.

Notabene: My original exception still exists (And I now realised it is probably unrelated to the MFT-fragmentation issue):
In the GetRecords method inside NTFSParser inside the used-records check (first if-clause), what happens when the last record is not used?

I'm getting the exception there, because the code assumes that the last record breaks the while(true) loop, but this last record doesn't necessarily need to be used.
I added the following code there after the for-loop:

// last record, not in use
if (!_usedRecords[CurrentMftRecordNumber]) { break; }

This at least doesn't produce any exceptions, not sure if some records are skipped though?

[LordMike]

I believe your first exception (reg. fragment being null) is what I was referring to. Iirc the fragment is found in a loop which returns null if it doesn't find the appropriate fragment. (A Debug.Assert should have been placed there - to make the exact issue more obvious).

I should note that the convenience features I've made are more an experiment than actual production ready code. This entire library is very low level, and it's possible to build higher level features out of it.

That being said - of course it has to work. If I find time, I might dedicate some to NTFS again. It's a fun project, digging around in filesystems and the way they work :). But I probably won't, as there are other usable NTFS parser in C# out there now (there weren't many when I started).

One that should work is DiscUtils. Depends on your end-goal ;).