Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

Defragging can be helpful, but not critical and not as important as it used to be

By Andrew Pollack on 02/16/2004 at 11:35 AM EST

I got an email question the other day and thought it'd make a great blog entry. This is, of course, all based on my experience and memory and not recent research so take it for what it is......

---

While defragmenting isn't a "Bad" thing, it isn't a necessary one either in most cases.

In the days of the Seagate ST-225 and ST-251, when we talked about random access times of 80ms and all drives were based on the MSDOS FAT system, having those big clunky stepper motor actuated read/write heads moving around took real time. Defgragging one of those did two things. First, it put all the parts of the file together sequentially on the drive, but second it put all the parts of the file sequentially on the physical platter itself within the drive. In those days, the logical sectors that the BIOS was writing to matched the physical locations on the disk. Writing to Platter-0, head-2 meant exactly that.

The typical drive/controller combination in say 1986 would have been a Seagate ST-225 20meg drive (a 5.25" 1/2 height drive with a stepper motor actuator) and a Western Digital WD-1002A-WA2 8 bit controller card. The drive was formatted in MFM format (17 sectors per track) at a 3:1 interleave (three spins of the drive would be required to sequentially read all the sectors on a track in order). Setting the interleave too low, like 1:1 if the card couldn't go that fast, slowed things down because if the card wasn't ready to read the next sector by the time it passed under the head, the platter had to spin all the way around again giving you effectively a 17:1 interleave. I recall seeing about 80k/second throughput on those. I owned a really sweet Adaptec 16 bit controller (don't recall the model number but 8000 seems famillar) which would format the drive (remember >g=C800:5 anyone?) in an RLL format (30% more sectors per track) and was fast enough to handle a 1:1 interleave. I remember seeing speeds of 800k/second with that controller and an expensive 80meg full height drive from Micropolis in my 80286. ** all this is from memory 15 years ago, so forgive me if I've gotten some details a bit off **

What's different now?

1. FAT vs. inode based file systems like NTFS

In a FAT file system, to read the 28th sector of a file, you had to read the fat first with its link to the first sector, then the first sector which had a link to the next sector, and so on and so on. That means to read the 28th sector you had 29 reads. That made a huge difference especially on files where you weren't reading the whole thing at once (a random access file). A fully defragged file, would have 17 sectors in a row all on one track of one platter -- which could be read in as little as one revolution of the platter (on a 1:1 interleave controller) or more commonly 3 revolutions (a 3:1 interleave was very common). Modern file systems don't work that way. The files are allocated via 'informational nodes' which contain maps to some or even all of the sectors of a file. Oh, and we also put way more data per sector on most of our drives now.

2. Drive speeds

Modern drives aren't the big clunky things they were in those days. Instead of 80ms random access seek time, we now see 8 milliseconds commonly. There hasn't been anything but a 1:1 interleave in years as far as I know, and the drives spin so fast that at 3.5" there is enough wobble at the ends of the platter to cause errors at the data densities now being achieved (that's why 3.5" is dead now, and all the newest stuff will be smaller and more dense). We no longer measure in k/second. Typically IDE configurations now run around 12 megabytes per second, and my new Serial ATA drive measures out at 30 megs per second in this machine. A 10,000 RPM SCSI drive with a good controller can double that and with RAID the speeds just ROCK.

3. Heads & Sectors aren't where you think they are any more

The ST-506 interface that defined MFM and RLL is still alive. We don't call it that, and we don't have a fat cable and a skinny cable to attach to each drive. We also don't have to "low level format" our hard drives. Now we have "IDE" which is the same thing really, but it puts the controller right on the drive. Since the controller is matched to the drive, no low level format is needed and the controller can be specially matched to enhanced features of the drive itself. Well, your PC's bios wasn't built to handle the number of sectors we now have on drives. It was built assuming a lot more heads though. So modern controllers "MAP" this for compatibility. They tell the outside world some number of heads and cylinders that give the pc the same number of sectors, but there's no direct link. That means, your defgramenting program cannot effectively know (on an IDE drive) exactly where the data "really" is. In theory, the drive's "Logical Map" will be optimized in some way so that you're not bouncing all over the drive if you put things in sequential order, but not necessarily.

Dangers of Defragging:

The good news is, it won't wear out your drive. That's a myth today. Those old drives were susceptible to it though. The metal band that wound and unwound to move the read/write heads when the stepper motor turned could literally get flexible. No, not anymore. The only danger today, is if things get moved around and aren't where the program expects them to be. Good defrag software should prevent that by getting between the OS call to read or write, and the actual drive. 32 bit operating systems do not allow software to write directly to the hardware (one of the biggest mistakes, imo, in early DOS).

Conclusion:

In part 1 we understand that we're not having sequential seek through a long line of sectors to get the files we want any more unless we need to read the whole file. In part 2 we see that those random seeks don't take as long, and in part 3 we see that we may not be saving as much time as we think on IDE drives. Combine this, and you see that the need for defragging is much lower than it used to be.

So when should you do it?

If you read a big file a lot, and you need to read it quickly and all the way from end to end -- its a good target for defragging. MP3 files don't count. They've big, and you read them end to end, but you rarely need to read them really fast -- just faster than your cd burner or your audio player needs them. The big files that get read from end to end frequently are your program files. For that reason, its probably a good idea to defrag after you install some big new software package. Also, put your big software stuff on your fastest drive. It matters. If you edit really really big files (like 100 meg graphics, movies, and such) you may want to have a defrag tool since you tend to read all of them into memory to work with at once. Also, If you are low on disk space, defragging can make a big "unfragmented space" to hold the next big thing you put down like a swap file. That can sometimes help as well.

Hope this helps!


There are  - loading -  comments....

You make a good case. Thanks, Andrew.By David Bailey on 06/23/2004 at 11:21 AM EDT
Now, let's hear from Executive Software .
Diskeeper - from Executive SoftwareBy Andrew Pollack on 06/23/2004 at 11:21 AM EDT
You know, that's a good idea. I'll invite them to respond here and see what they have to say. There is NO QUESTION in my mind that right now they make the best product for automatically keeping a drive defragmented, and there ARE cases as I've said above where it will help -- just not the critical need there used to be. In the interest of fairness, here is a link to their "Myths of Fragmentation" page. IMO it doesn't really address my issues, but its all I can find. http://consumer.execsoft.com/diskeeper/myths.asp?RId=50&flg=1
Sent this message to Executive SoftwareBy Andrew Pollack on 06/23/2004 at 11:21 AM EDT
Andrew Pollack/thenorth 02/17/2004 08:43 AM To sales@executive.com cc Subject We invite you to comment on our discussion of fragmentation in a modern environment.. Dear Executive Software; In a semi-regular column that I write, I was asked about the pros and cons of running defragmentation programs. I wrote a few paragraphs from my own experience, memory, and past research into the subject which held the conclusion that while sometimes helpful it is no longer as critical to do. Someone else made the comment that they'd love to hear from Executive Software on the subject and I agree. Perhaps we could all learn something if Executive Software would like to comment on any of my own findings or offer any corrections on misconceptions. For now, I have posted a link to your "Myths of Fragmentation" page. Here is a link to the article. You may post there of course, of you may respond to me directly and I would be happy to either post a link to your response, post your response, or summarize your response and post a link to it depending on its size, scope, and utility. http://www.thenorth.com/apblog2.nsf/0/CDCA6DE3DE7AC17C85256E3C005B2576 --------------------------------------------------------------------------------------------------------- *** Andrew Pollack, Northern Collaborative Technologies *** AIM: AndrewJayPollack *** http://www.thenorth.com *** andrewp@thenorth.com *** 207-829-2131 ---------------------------------------------------------------------------------------------------------
NTFS which run out of disk space usually have sparse MFTs...By George Chiesa on 06/23/2004 at 11:21 AM EDT
which can be remedied by reformatting the NTFS (or using boot time defraggers). IOW you want to make the NTFS tables efficient (not with zillions of continuation space). Never let an NTFS run out of space because even after freeing space, the MFTs are kept screwed


Other Recent Stories...

  1. 04/04/2020How many Ventilators for the price of those tanks the Pentagon didn't even want?This goes WAY beyond Trump or Obama. This is decades of poor planning and poor use of funds. Certainly it should have been addressed in the Trump, Obama, Bush, Clinton, Bush, and Reagan administrations -- all of which were well aware of the implications of a pandemic. I want a military prepared to help us, not just hurt other people. As an American I expect that with the ridiculous funding of our military might, we are prepared for damn near everything. Not just killing people and breaking things, but ...... 
  2. 01/28/2020Copyright Troll WarningThere's a copyright troll firm that has automated reverse-image searches and goes around looking for any posted images that they can make a quick copyright claim on. This is not quite a scam because it's technically legal, but it's run very much like a scam. This company works with a few "clients" that have vast repositories of copyrighted images. The trolls do a reverse web search on those images looking for hits. When they find one on a site that looks like someone they can scare, they work it like ...... 
  3. 03/26/2019Undestanding how OAUTH scopes will bring the concept of APPS to your Domino serverWhile a full description of OATH is way beyond what I can do in this quick blog entry, I wanted to talk a bit about how "SCOPES" interact with the already rich authorization model used by Domino. Thanks to the fantastic work by John Curtis and his team, the node.js integration with Domino is going to be getting a rich security model. What we know is that a user's authorizations will be respected through the node.js application to the Domino server -- including reader names, ACLs, Roles, and so on. The way ...... 
  4. 02/05/2019Toro Yard Equipment - Not really a premium brand as far as I am concerned 
  5. 10/08/2018Will you be at the NYC Launch Event for HCL Domino v10 -- Find me! 
  6. 09/04/2018With two big projects on hold, I suddenly find myself very available for new short and long term projects.  
  7. 07/13/2018Who is HCL and why is it a good thing that they are now the ones behind Notes and Domino? 
  8. 03/21/2018Domino Apps on IOS is a Game Changer. Quit holding back. 
  9. 02/15/2018Andrew’s Proposed Gun Laws 
  10. 05/05/2016Is the growing social-sourced economy the modern back door into socialism? 
Click here for more articles.....


pen icon Comment Entry
Subject
Your Name
Homepage
*Your Email
* Your email address is required, but not displayed.
 
Your thoughts....
 
Remember Me  

Please wait while your document is saved.