Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

The case for using on disk compression for your Notes Data directory

By Andrew Pollack on 04/22/2005 at 02:21 PM EDT

I did some very extensive testing & reviews back in the 90's when Stacker and its ilk were on the market. Microsoft bought out much of that technology and its now pat of the operating system. You can select a folder, a file, or a drive and elect to "compress" the data.

For the sake of this discussion, lets assume a compressed file will be 50% smaller than a normal one. That's a lower than normal compression ratio compared to zipping an NSF file, but on disk compression isn't quite as effective because of the way it works.

Taking just disk I/O into account, that means 50% less data traveling to and from the mechanical media itself -- the slowest part of the transfer. In a perfect world, that means you double the speed. In practice, there is overhead however. You have to assign processor time to the compression, and you have the overhead of the programming itself and in memory copying of data that must take place.
12 years or so ago when I did tests, I determined that on a 386 25mhz processor with an ide hard disk operating at 33mhz you crossed the threshold where processor time loss was lower than disk read/write gain. In other words, compression was faster.

Today there are a lot of different options for drives. Today's drives transfers about 7-10 times faster than those of 12 years ago. Processors, however, tend to run 20 or more times faster (its not just about clock speed, but leave it at that). On a multiprocessor or Hyperthreading machine even more. Add to that, that the processor utilization on most servers is the only resource that isn't being tapped out. Most of the time we have 20-75% free processor cycles.

My purely subjective testing thus far definitely supports my hypothesis that increased processor use is well worth it you can cut disk usage in half. I'm seeing end user noticeable performance increases using compressed NSF files on both servers and workstations. I have not done any objective empirical testing recently however.

In the early days of Stacker, my big worry was reliability, but the technology proved out, and has been extremely reliable. Yes, its true that a sector failure will wipe twice as much data this way. Has it been an issue for you? Today's file systems are all virtualized anyway, so its not like you're getting in the way of hardware calls to the drive on interrupt 0x13 like you were back then.


  • car icon

    Server Performance

    Are your servers underperforming? Just buying new boxes isn't the answer. If you want to get better performance from your existing servers, Contact Me.
  • There are  - loading -  comments....

    Very interesting...By Ben Rose on 04/23/2005 at 15:15 EDT
    Comment Loading
    Works for meBy Chris Linfoot on 04/28/2005 at 08:25 EDT
    Comment Loading
    But...By Chris Linfoot on 04/29/2005 at 07:20 EDT
    Comment Loading
    Check out your server CPU performance....By Andrew Pollack on 04/29/2005 at 07:42 EDT
    Comment Loading
    Pretty much where I was with thisBy Chris Linfoot on 04/29/2005 at 10:55 EDT
    Comment Loading


    Other Recent Stories...

    1. 09/02/2010Linux ext3 file system performance wierdnessI've had trouble off and on with a couple of Domino servers on linux. The server goes pear shaped and when I ssh in and look at the console, Domino is reporting drive errors. If you attempt to do anything on the OS at all, you quickly see that the whole file system has shifted into a "read-only" state. This is a bit like a car with a transmission problem shifting into "limp-home" mode. Needless to say, Domino doesn't like being unable to write to the disk. It has happened to me specifically with the most ...... 
    2. 08/31/2010A few tips for car sales people - if you ever want my business.I've been shopping for cars again. This one is a replacement for the spousemobile, so unlike the one for my daughter, this will be new. Having done a fair bit of shopping, here are some tips for you car sales people out there. #1. Do not bullshit me. I may know more than you think. It's just possible that I know how to build that car you're showing me. One lady insisted that I could ignore the EPA mileage sticker on a new Honda or Nissan (which happened to be what is on her lot) because (and I quote) ...... 
    3. 08/22/2010Creating a form generating tool that lets non-technical publshers create rich web formsI've been working on a tool for a client that wants to be able to create web forms to use on their site without learning any programming at all. At the same time, I don't trust any data from the browser side on a web site. I always assume that the browser side is open to hackers. That means any submitted form data has to be validated before it is accepted. Here's what I did... For publishing, I used the CKEditor, which is the web browser rich editor I've standardized on. For most web publishing uses, I use ...... 
    4. 08/13/2010That didn't take long. Oracle is suing Google over the use of Java. Is IBM next? 
    5. 07/20/2010IBM has invented the time machine - and not in a good way 
    6. 07/18/2010A lot of work, but something I've always wanted to do -- Fixing up a car for one of my kids 
    7. 07/13/2010Old Spice -- Their marketing team fully groks social media.  
    8. 06/22/2010Product Review: Plantronics Savi Office wireless headset 
    9. 06/16/2010Ed Brill has a blog. Peter O'Kelly has blog... 
    10. 06/15/2010How about Traveler for Desktop? 
    Click here for more articles.....


    pen icon Comment Entry
    Subject
    Your Name
    Homepage
    *Your Email
    * Your email address is required, but not displayed.
     
    Your thoughts....
     
    Remember Me  

    Please wait while your document is saved.