mysterious discrepancy in the reported free space on two identical usb drives

Discussion:

(too old to reply)

Jonathan de Boyne Pollard

2010-02-18 04:52:59 UTC

</a> 
 
I have two identical WD 1tb passport usb drives filled with identical
data. [...] 

</blockquote>
... but not necessarily identical metadata.  That 28KiB difference
is a mere 28 deleted MFT records, for example.  Or it could be journal
entries, security descriptor records, or quite a number of other things. 

 
</body>
</html>

sobriquet

2010-02-18 10:46:58 UTC

Permalink

On 18 feb, 05:52, Jonathan de Boyne Pollard <J.deBoynePollard-

Post by Jonathan de Boyne Pollard
Loading Image...

I have two identical WD 1tb passport usb drives filled with identical data. [...]
... but not necessarily identical metadata. That 28KiB difference is a mere 28 deleted MFT records, for example. Or it could be journal entries, security descriptor records, or quite a number of other things.

But isn't this the kind of meta data that is supposed to disappear
when the recyclebin is emptied?
Is there any other way to clear any superfluous data somehow?
If two drives have the same data and one has more free space, surely,
somehow, it must be possible to free up
the same space on the other drive, because we're talking about two
identical drives and identical data that is supposed to be stored on
the drive.

I don't care about the minimal filespace that would be obtained this
way, I'm just interested in figuring out exactly what's responsible
for this discrepancy between the two drives in reported free space
while they contain identical data that has been stored on the drives
in an identical fashion.

Jonathan de Boyne Pollard

2010-02-18 20:17:26 UTC

Permalink

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<blockquote
cite="mid:70c04a46-ca9e-4c51-9766-***@z11g2000yqz.googlegroups.com"
type="cite">
<blockquote type="cite">
<blockquote type="cite">
<a
href="http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg">http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg</a> 
I have two identical WD 1tb passport usb drives filled with identical
data. [...] 

</blockquote>
... but not necessarily identical metadata.  That 28KiB
difference is a mere 28 deleted MFT records, for example.  Or it could
be journal entries, security descriptor records, or quite a number of
other things.
</blockquote>
But isn't this the kind of meta data that is supposed to disappear
when the recyclebin is emptied?
</blockquote>
Put simply: No.  Deleted MFT records are nothing to do with files in
the recycle bin, for example.
<blockquote
cite="mid:70c04a46-ca9e-4c51-9766-***@z11g2000yqz.googlegroups.com"
type="cite">
Is there any other way to clear any superfluous data somehow?
</blockquote>
Put simply: Short of drastic measures such as reformatting the
volume, no.  The MFT doesn't shrink in normal operation, for example. 
And the security descriptor stream is only compacted by <code>chkdsk</code>. 
(See MSKB 919241.)
<blockquote
cite="mid:70c04a46-ca9e-4c51-9766-***@z11g2000yqz.googlegroups.com"
type="cite">
we're talking about two identical drives and identical data that
is supposed to be stored on the drive.
</blockquote>
No, we're not.  As I said, the metadata are not necessarily
identical.
</body>
</html>

sobriquet

2010-02-18 22:29:49 UTC

Permalink

On 18 feb, 21:17, Jonathan de Boyne Pollard <J.deBoynePollard-

Post by Jonathan de Boyne Pollard
http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg
I have two identical WD 1tb passport usb drives filled with identical data. [...]
... but not necessarily identical metadata. That 28KiB difference is a mere 28 deleted MFT records, for example. Or it could be journal entries, security descriptor records, or quite a number of other things.
But isn't this the kind of meta data that is supposed to disappear when the recyclebin is emptied?
Put simply: No. Deleted MFT records are nothing to do with files in the recycle bin, for example.
Is there any other way to clear any superfluous data somehow?
Put simply: Short of drastic measures such as reformatting the volume, no. The MFT doesn't shrink in normal operation, for example. And the security descriptor stream is only compacted bychkdsk. (See MSKB 919241.)
we're talking about two identical drives and identical data that is supposed to be stored on the drive.
No, we're not. As I said, the metadata are not necessarily identical.

I see. Could a virus or malware somehow gain access to this space
where this metadata is stored to hide a copy of itself there?
Can I use a diskeditor like HxD or DiskExplorer for NTFS to view this
metadata somehow?

Arno

2010-02-18 23:04:24 UTC

Permalink

Post by sobriquet
On 18 feb, 21:17, Jonathan de Boyne Pollard <J.deBoynePollard-

Post by Jonathan de Boyne Pollard
http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg
I have two identical WD 1tb passport usb drives filled with identical data. [...]
... but not necessarily identical metadata.? That 28KiB difference is a mere 28 deleted MFT records, for example.? Or it could be journal entries, security descriptor records, or quite a number of other things.
But isn't this the kind of meta data that is supposed to disappear when the recyclebin is emptied?
Put simply: No.? Deleted MFT records are nothing to do with files in the recycle bin, for example.
Is there any other way to clear any superfluous data somehow?
Put simply: Short of drastic measures such as reformatting the volume, no.? The MFT doesn't shrink in normal operation, for example.? And the security descriptor stream is only compacted bychkdsk.? (See MSKB 919241.)
we're talking about two identical drives and identical data that is supposed to be stored on the drive.
No, we're not.? As I said, the metadata are not necessarily identical.

Very, very unlikely as this is only possible if the malware
has a very good unsderstanding of NTFS. This would be hard to
do and make the malware large, hence easy to detect. Malware
can hide in other places thogh, for example the partially
used clusters at file ends or brazenly in seemingly unused
space.

This is almost certainly not malware. Also the extra
space may well be used with the emtadata just being a bit
more compact on the one drive.

As I said, don't worry about this, a bit of uncertainity
in metadata size is expected in modern filesystems. If
you look at the actual size difference, you can understand
why nobody invested a lot of effort to optimize this.
It is just not worth the effort.

If you really want to make both drives the same size, the
only way I see is to format them and then put all files
on both using exactly the same procedure. This may still
not work, as the metadata is always slightly different,
for example in the timestamps.

If you really want to look at the metadata, good luck.
I expect analyzing these drives manually in detail might
take more than a month of time, possible much more.

Arno

--
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: ***@wagner.name
GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

Yousuf Khan

2010-02-19 04:34:11 UTC

Permalink

Post by Arno
If you really want to make both drives the same size, the
only way I see is to format them and then put all files
on both using exactly the same procedure. This may still
not work, as the metadata is always slightly different,
for example in the timestamps.

Something like a RAID-based mirroring system which copies data at a much
lower level than the filesystem.

Yousuf Khan

Arno

2010-02-19 14:38:36 UTC

Permalink

Post by Yousuf Khan

Something like a RAID-based mirroring system which copies data at a much
lower level than the filesystem.

Yes. Or making a copy with a sector imager. Don't use both
drives at the same time aftert this, as the GUIDs will
allso have been copied.

Arno

Jonathan de Boyne Pollard

2010-02-19 16:16:08 UTC

Permalink

Or making a copy with a sector imager. Don't use both drives at the
same time aftert this, as the GUIDs will also have been copied.

That's only the case if one copies the entire disc, rather than copies
just the contents of a specific volume (as tools such as DISKCOPY and
its equivalents do). When copying a specific volume, one only need
worry about volume serial numbers; and the failure mode that duplicate
volume serial numbers predominantly cause (failure to recognize
removable disc changes in some operating systems) won't occur with both
discs present simultaneously, and won't occur with non-removable media,
in any case.

Rod Speed

2010-02-19 00:03:45 UTC

Permalink

Post by sobriquet

Post by Jonathan de Boyne Pollard
http://img237.imageshack.us/img237/2852/passportdiscrepancy.jpg
I have two identical WD 1tb passport usb drives filled with identical data. [...]
... but not necessarily identical metadata. That 28KiB difference is
a mere 28 deleted MFT records, for example. Or it could be journal
entries, security descriptor records, or quite a number of other things.
But isn't this the kind of meta data that is supposed
to disappear when the recyclebin is emptied?
Put simply: No. Deleted MFT records are nothing to do with files in
the recycle bin, for example.
Is there any other way to clear any superfluous data somehow?
Put simply: Short of drastic measures such as reformatting the
volume, no. The MFT doesn't shrink in normal operation, for example.
And the security descriptor stream is only compacted bychkdsk. (See
MSKB 919241.)
we're talking about two identical drives and identical data that is supposed to be stored on the drive.
No, we're not. As I said, the metadata are not necessarily identical.

I see. Could a virus or malware somehow gain access to this space
where this metadata is stored to hide a copy of itself there?

Yes, it might be able to hide a copy of itself, but
there is no way to get it executed from there.

Post by sobriquet
Can I use a diskeditor like HxD or DiskExplorer for
NTFS to view this metadata somehow?

Yes, anything that can dump the contents of sectors you specify can do that.

Interpreting what you see tho is much harder.

Quite a bit of the detail of NTFS structures have never been formally documented.

Barry OGrady

2010-02-21 22:45:37 UTC

Permalink

On Thu, 18 Feb 2010 04:52:59 +0000, Jonathan de Boyne Pollard
<J.deBoynePollard-***@NTLWorld.COM> wrote:

<nothing>

Try again.

Barry
=====
Home page
http://members.iinet.net.au/~barry.og