Disk Defragmentation: A fast guide [Excepts from Serdar Yegulalp's Article]
For years, administrators have been hearing stories about whether or not they need to defrag their hard drives. How big of a problem is fragmentation really? How beneficial is defragmentation?
Defragmentation has become one of the big performance-sapping bogeymen of Windows systems. Like kudzu overgrowing the landscape, it can never be completely eliminated -- just kept at bay. At least, that's the conventional wisdom. Over time I've grown curious as to how much of the conventional wisdom about fragmentation?and defragging?is true. To that end, I set out to examine the subject with a fresh eye and find out just how much of a problem fragmentation is, as well as how much of a benefit defragmentation is. Let's begin by determining what fragmentation is. A file is stored in a file system as one or more allocation units, depending on the size of the file and the size of the allocation unit on the volume in question. As files get written, erased and rewritten, it may not be possible to write a file to a completely contiguous series of empty allocation units. One part of a file may be stored in one part of a disk, the rest of it somewhere else. In extreme cases, it may be scattered quite widely. This scattering is what is called fragmentation. The more fragmented a file, the more work the computer has to do to read it. Usually, this comes down to how fast the hard drive can seek to a specific sector and read the allocation units in question. If the computer has to read several fragmented files at once, the number of head movements and the amount of contention for disk access will go up?and things will slow down. Fragmentation became widely recognized as a problem in the early '90s. Hard drives were relatively small and slow (compared to today), and so fragmentation had a correspondingly big impact on system speed. To address the problem, Microsoft shipped DOS 6.0 with the command-line DEFRAG utility (licensed from Symantec). Some editions of Windows licensed a stripped-down version of what is still the most popular third-party defragmentation program for Windows: Executive Software's Diskeeper. Depending on their budget, as well as their inclination, most admins use either Diskeeper or Windows' native Disk Defragmenter tool. Either way, the prevailing sentiment is to defrag early and often.Most of the conventional wisdom regarding defragging evolved from the way storage worked in the early 90s. The file systems used on PCs then were FAT12 and FAT16, holdovers from the days of MS-DOS. When Windows 3.1 and Windows 95 appeared, the limitations of FAT12 and FAT16 came to the fore. Neither file system could support volumes more than 4 GB in size or file names longer than eight characters. Both were notoriously prone to errors and data loss and were particularly prone to fragmentation.
Dump the FAT and move to NTFS
Microsoft introduced FAT32 as a way around some of these issues. But the best long-term solution was to dump FAT entirely as a system-level file system and move to NTFS, a more robust file system that had been in use for Windows NT for some time. One of the many improvements NTFS provided was a reduced propensity for fragmentation. It doesn't eliminate fragmentation entirely, but it does guard against it much better than any version of FAT ever did. Today, no Windows system comes shipped loaded with anything less than Windows XP or Windows Server 2003, and the drives are almost always formatted with NTFS. So much of the impact of fragmentation is lessened by NTFS' handling of the problem.
Microsoft introduced FAT32 as a way around some of these issues. But the best long-term solution was to dump FAT entirely as a system-level file system and move to NTFS, a more robust file system that had been in use for Windows NT for some time. One of the many improvements NTFS provided was a reduced propensity for fragmentation. It doesn't eliminate fragmentation entirely, but it does guard against it much better than any version of FAT ever did. Today, no Windows system comes shipped loaded with anything less than Windows XP or Windows Server 2003, and the drives are almost always formatted with NTFS. So much of the impact of fragmentation is lessened by NTFS' handling of the problem.
New hard drives reduce need for defragging
Fragmentation is the demon that haunts hard disks for all users of Windows (and other operating systems). Defragmentation is the exorcist for that demon. After closely examining the subject, I've found that many of the things we held to be true about both fragmentation and defragmentation have become less and less true over time. The NTFS file system in Windows has helped alleviate many of the worst problems associated with fragmentation. However, it has not cured them.
Fragmentation is the demon that haunts hard disks for all users of Windows (and other operating systems). Defragmentation is the exorcist for that demon. After closely examining the subject, I've found that many of the things we held to be true about both fragmentation and defragmentation have become less and less true over time. The NTFS file system in Windows has helped alleviate many of the worst problems associated with fragmentation. However, it has not cured them.
File systems are not the only things that have changed in the last ten years or more. Hard drives have changed too.In 1996, a new Western Digital hard drive holding 1.6 gigabytes (what you could expect to fit on a decent-sized flash drive today) cost $399. That drive was in the same 3.5-inch form factor used for today's hard drives, so the size of the platters in the drive were pretty much the same. Today, a new Western Digital drive that costs $199 holds 500 GB. So for half the money you're getting more than 300 times more capacity. But there have been other changes. The platter sizes of drives have remained the same, but the rotational speeds of the platters have gone up?to as much as 15,000 RPM on high-end drives. But the biggest change in hard drive technology over the past decade has been in information density. With so much more information crammed into the same space, the drive heads move that much less to read that much more data. This in itself speeds up data transfers. Most hard drives now also come with an on-board cache of RAM, usually 16MB or more. Data can then be read from the disk into the drive's cache in one big chunk, and parceled out to the host computer as needed. This saves the hard drive from having to go back and dip repeatedly into the same parts of the drive for multiple bits of data. On top of everything else, the operating system does its own caching to further alleviate the affects of a fragmented file?or even fragmented NTFS metadata, which tends to be cached and held as needed. What does all this mean? The big disadvantage of fragmentation ? that it scatters data across a drive -- has been greatly offset by all of these changes. Furthermore, 50MB of data fragmented across a ten-year-old hard drive will have a far bigger impact on performance than the same 50MB fragmented to the same degree on a new hard drive. Now factor in the benefits of caching, faster and smaller head movements, modernized file system storage and OS-level caching, and it becomes clear why fragmentation isn't anywhere near as bad as it used to be. (The one exception to this scenario is if the file is scattered literally all the way across the surface of the drive, which is fairly unlikely.)
Larger and faster drives have minimized the impact of fragmentation. The Windows file system tends to fragment files all on its own to a small degree, but fragmentation starts for real when the drive starts to get full?as in over 70%. "As the drive fills up, the larger areas of free space become scarce and the file system has no choice but to splatter large files around the disk. As the drive gets really full (over 90%), the file system then starts to fragment the MFT and the Pagefile. Now you've got a full drive, with lots of fragmented files, making the job of the defragmenter nearly impossible because it also needs space to do its job.
A drive that is more than 80% full is not defragmentable. You can see that effect with huge drives, since they generally use smaller percentages of the drive's total free space. A side-effect is that the overall fragmentation is reduced, and the fact that these drives have faster seek times makes the effect even less noticeable.
The amount of available free space on the drive determines how effective defragmentation is. A defragmenter can only work when there is free space available to move things around. The less free space, the more hobbled the defragmenter will be at doing its job. In addition, more free space on a drive means that there is more room for the file system to write out files and directories without fragmentation in the first place.
When you set up a new hard drive, whether as a data drive or as a new system drive, defragment it as soon as the first wave of installing applications, copying data, etc. is done. This allows the defragmenter to do a fair amount of work while there is still plenty of free space on the drive for that work to be done. It also allows for some key file system structures to be written out in contiguous order, such as the swap or paging file, or the file the system uses to hibernate the computer. (Note: These files can be deleted and recreated if needed, but few people are in the habit of checking to see if they need this done, or doing so.)
Defragment early so that you could take advantage of the free space before your drive starts to fill up. This way, they could see a marginal improvement now, but, more importantly help, the defragmenter from getting log-jammed later on.
In short, many of the issues that crop up tend to be because of a lack of free space causing cumulative problems. With larger drive sizes, faster access times and smarter handling of storage, the total impact of fragmentation has been reduced. That being said, defragging still has a place.
Defragging more than once a week is pointless; the time spent doing the defrag far outweighs the benefits gained, especially if you do it that obsessively.
Fragmentation still exists and is a problem. I was not dismissing its impact wholesale. But its impact has been alleviated by advances in hard drive technology, operating system design and file system design, and its impact will continue to be reduced (but not eliminated) by further improvements in all of the above. It's still a good idea to defragment regularly, but there's little point in doing it obsessively when the real-world benefits might not be measurable in any reliable way. More than once a week for a workstation seems to cross the point of diminishing returns (although there are exceptions, which I'll go into). But the investment of time and system resources required to defrag once a day doesn't pay off except in the most incremental and difficult-to-assess fashion.
