Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

DAOS (Attachements stored outside the NSF in a pool) after 1 week - The Vedict

By Andrew Pollack on 07/01/2008 at 08:46 AM EDT

It's a winner! After one week, the most important thing I can say about using DAOS on my mail file, is that I'd totally forgotten about it.

DAOS enabled servers use a file system data store as a shared pool for file attachments across many databases to prevent duplication, drastically reduce file storage space requirements, reduce disk fragmentation, and improve NSF file performance and reliability.

After one week of using DAOS, my mail file remains about half the size it was before, but otherwise I'm unable to tell the difference. My local replicas and replicas on other servers (including clusters) are not impacted. If anything, performance has improved.


There are  - loading -  comments....

re: DAOS (Attachements stored outside the NSF in a pool) after 1 week - The VedictBy Rob Wills on 07/01/2008 at 01:09 PM EDT
Cool. I would also expect the proportional reduction in size to increase on
servers with lots of mail files because of the duplicated attachments sent to
numerous recipients. This should significantly reduce the e-mail storage costs
of large corporations using Domino. Rob
re: DAOS (Attachements stored outside the NSF in a pool) after 1 week - The VedictBy Kevin Pettitt on 07/01/2008 at 02:24 PM EDT
The first question asked when I mentioned this on a client site last week was
"isn't this just a repackaged version of the old 'shared mail'?" I don't even
remember the reasoning anymore but it long ago became a matter of faith to
never implement that feature. So the concern is understandable. If someone
could talk about that it would be very helpful.

Following on from that point, a successful case study of DAOS on a very large
site would be very comforting. If I recall correctly, DAOS is implemented on an
individual server basis, so you could have a clustered pair of servers with
DAOS running on only one, right? The idea being that if DAOS goes flaky on that
box, the other remains viable, but of course there may be ways for the
flakiness to transfer over. Thoughts welcome.
re: DAOS (Attachements stored outside the NSF in a pool) after 1 week - The VedictBy Andy Broyles on 07/01/2008 at 02:30 PM EDT
I have not had an opportunity to explore 8.5beta...can DAOS be used outside of
mail for other Notes applications?
re: DAOS (Attachements stored outside the NSF in a pool) after 1 week - The VedictBy Martin on 07/02/2008 at 04:22 AM EDT
As I understand the article at
http://www.dominoblog.com/dominoblog/dblog.nsf/dx/reduce-storage-costs-with-domi
no-8.5 it is possible to use it just for mailboxes as the attachments are
stripped out at mail.box. But then it shouldn't work on cluster server or
replicas as there is no connection with mail.box.
re: DAOS (Attachements stored outside the NSF in a pool) after 1 week - The VedictBy Erik Brooks on 07/01/2008 at 07:38 PM EDT
The way it is advertised, DAOS works across any Domino NSF on the server.

This should be a *HUGE* win across the board, though there are a few
disadvantages:

- If you want to backup individual DBs you absolutely must back up your DBs
with a Domino-aware, api-level backup tool. If you want to simply backup
everything then that's easier: backup the NSFs, backup the filestore directory
and you should be good.

- It's only as robust as your filesystem. If your filesystem balks at tons of
files you should explore DAOS carefully.

- I've heard that you can technically "leave" Domino's security. I.E. a
plaintext file attached to a Domino doc can end up as plaintext in the
filestore directory, susceptible to anybody with read access to the directory.


Personally I'm excited about it from an app dev standpoint. I can offload gigs
of attachments out of NSFs (on very expensive and fast arrays) onto a more cost
effective SAN.


re: DAOS (Attachements stored outside the NSF in a pool) after 1 week - The VedictBy Sebastian Leen on 07/05/2008 at 08:16 PM EDT
What about backup? I use a standalone backup utility (not related to domino). I
guess it will not work since there is no way to see which atts belongs to which
DB thus there is no easy way to restore a DB with the related Atts.

Also, I was told it was unsecure, that if I go to the DAOS folder and I make a
double click on the cached file, I can see it!!!

I was also told that spams will cause problem with DAOS since internally a MIME
email can be stored as an internal att so DAOS will cache it... so I will have
millons of atts in the DAOS folder, not too good.

Also it's not easy to measure the savings since it removes the atts from the
NSF but puts them in the file system...
wow, LOTS of misinformaton in that replyBy Andrew Pollack on 07/05/2008 at 08:37 PM EDT
backup:
There is a backup api in Domino that will allow Domino aware backup
software to do the backups including the necessary files. This also will allow
better incremental backups for nsf files. Alternately, you can back up the
DAOS store with your backup files.

security:
I don't know who told you this, but these decisions aren't even finalized
yet. Files aren't going to be just dumped into the file system unsecured. I
can't talk about details because of NDA -- but there's plenty of history on
IBM's commitment to security. Its a subject being fully covered in the design
partner discussions and I'm sure IBM either has or will be making statements
fairly soon on this.

spams:
There are many ways to combat this. the first is of course better spam
protection. The second is that you can set the minimum size for DAOS to pick
out files. Spams don't send 100k attachments, they send (at most) small image
files. Also, if you're planning to use DAOS, you're going to need to take
responsibility for picking a file system that can handle it.

Out of curiosity, where are you hearing all this?

Also, the email address you've left here isn't valid. That makes me wonder,
frankly, where you come from.




re: wow, LOTS of misinformaton in that replyBy John possi on 07/07/2008 at 10:07 AM EDT
Thanks Andrew, I never put my email address since I don't want to receive spams.

Here are my findings:
1 Backup: So I will need to change my usual backup utility by another one
coded using the Domino API. OK, I got the answer.

2 Security: I was told by a colleage that did the tests... anyway I got the
answer.

3 SPAMS: I think you didn't catch the question. Suppose that I enable DAOS for
mail.box (as IBM recommends). Now I receive one spam email so it is created in
the mail.box. At this time the attachment is already put in the DAOS
repository... it's too late for my antispam! My antispam will delete the email,
OF COURSE, but the attachment will be left in the DAOS repository until the
next prune is performed (one month). The attachment is put in the DAOS
repository as soon as Domino saves the document in the mail.box... antispam
acts AFTER the document is put in the mail.box.

4 SPAM2 Reply of Reply: I agree that antispam usually don't contain
attachments but just few images. But the HTML portion of the email (that is,
the body itself) is usually large than 100 KB. This is not only for spams but
also for reply of reply... So you will think "Ok, but DAOS is just for
attachments, not for the body"... but this is WRONG: When you receive a MIME
email (an email from internet) holding a body having more than a give size (I
don't know but I think 100 KB is enough) then the HTML and "Plain text" portion
of the email are stored in internal attachment. You can make the test, send an
email containing lot of text and then when you receive it you will see that
there are two $file internal attachment: One for the HTML and another for the
"Plain text" representation. So it basically means that if my users receives
plenty of personal emails containing replies of replies, or spams then my DAOS
folder will become huge until next prune.

5 What about clusters? Suppose in one physical server I have my main server
and the cluster... will I have two DAOS folders? I guess so since DAOS is in a
per-server basis.

6 Will it really save space? Suppose that I have thousands of users that
receives SPAM and plenty of attachments... I enable DAOS for all DBs. Now the
size of the NSFs will of course be reduced but each MB I took from the NSF will
be left in my file system. So DAOS magic is that duplicate atts will not be
repeated so the storage is based on removing duplicate attachments... but on
the other side there will be millons of attachments that will be in the DAOS
folder that were deleted by the user in the NSF (because they were SPAM or
replies of replies) so until next prune is performed these attachments will be
there. Consequence: At a given time (e.g. one day before my prune day) the
storage using DAOS will be bigger than without using DAOS since the size of the
not-yet-removed attachments will be bigger than the size of the
duplicate-atts-savings. Of course prunning each day is not a choice.

Thanks.
Let me try to answer some of these....By Andrew Pollack on 07/07/2008 at 11:03 AM EDT
...Though I have to say, I'm tempted to not bother given your propensity for
false input on the comment stuff. You know who I am, you should have courtesy
to tell me who you are when we're talking. Email address is not published
here, but is required. By my own rules, I should be pulling your comments.
The only reason I haven't, is because they're valid and interesting so far.

1. Backup: I don't know what your usual backup utility is so I can't answer.
DAOS is an enterprise solution, not a home server solution. If you're
seriously backing up massive domino mail directories with a $59.00 tool, I'm
not sure why I'm bothering to talk to you. The other major market tools are
being approached to participate in the API and I suspect support will be fairly
rich. Remember, we're still in early beta with this thing.

2. Security: Your colleage is right for the version currently being tested. I
think if you read the release notes for the beta, this is discussed. I know
for a fact that it won't be released this way. I can't say what the actual
plan is wrt encryption on disk and so on, but perhaps one of the ibm people who
read here will comment.

3. Spams: Why is your primary mail server your front end anti-spam device?
Sounds pretty small time to me. Maybe your environment isn't well suited to
using DAOS. Its a feature, not a requirement. My anti-spam happends long
before any mail reaches my user mail servers, and I'm a tiny shop. I know of
no major sites beyond the SOHO where spam gets all the way to mail servers
before anti-spam is done.

4. I think you'll find this isn't strictly true, but I'm going to have to let
someone from IBM answer it for now if they're willing.

5. At least at this time, DAOS is per-server. Domino clusters are totally
independent of each other. You can take an ax to one and the other will be
fine. You can cluster across wide area networks if you want. This is a very
different idea than the pseudo clustering Microsoft does. I've heard
discussions around the idea of a cluster using a shared DAOS store at some
point in the future -- perhaps if its being stored on a high availability SAN
where redundancy is handled outside of Domino. At present this isn't what
they're doing. I don't know if it will ever happen. I guess demand will drive
it.

6. Will it really save space? Your mileage may vary. Look into IBM's test
data (which I won't quote because I don't track what is NDA and what isn't) and
you'll see that it saves a huge amount of space for a majority of sites. How
much will be dependent on the way mail is used in the environment and other
factors but I know of no cases where its been tried and there has not been a
sizable space savings.

From everything I can tell so far, DAOS is far far from what the old crappy
"Shared mail" thing was. The version in Beta (and probably the initial stuff
in 8.5) barely touches the future potential of what it can do.


Other Recent Stories...

  1. 05/05/2016Is the growing social-sourced economy the modern back door into socialism?Is the growing social-sourced economy the modern back door into socialism? I read a really insightful post a couple of days ago that suggested the use of social network funding sites like “Go Fund Me” and “Kickstarter” have come about and gained popularity in part because the existing economy in no longer serving its purpose for anyone who isn’t already wealthy. Have the traditional ways to get new ventures funded become closed to all but a few who aren’t already connected to them and so onerous as to make ...... 
  2. 04/20/2016Want to be whitelisted? Here are some sensible rules for web site advertisingAn increasing number of websites are now detecting when users have ad-blocking enabled, and refuse to show content unless you "whitelist" their site (disable your ad-blocking for them). I think that is a fair decision on their part, it's how they pay for the site. However, if you want me (and many others) to white list your site, there are some rules you should follow. If you violate these rules, I won't whitelist your site, I'll just find content elsewhere. 1. The total space taken up by advertisements ...... 
  3. 12/30/2015Fantastic new series on Syfy called “The Expanse” – for people who love traditional science fiction[] “The Expanse” is a new science fiction series being broadcast onthe Syfy channelthis winter. It’s closely based on a series of books by author James S. A. Corey beginning with “Leviathan Wakes”. There are 5 books in the “Expanse” series so far. If you’re a fan of the novels you’ll appreciate how closely the books are followed.TIP: The first five episodes are already available on Syfy.com. If you’re having trouble getting into the characters and plot, use those to get up to speed.The worlds created for ...... 
  4. 10/20/2015My suggestion is to stay away from PayAnywhere(dot)com  
  5. 08/07/2015Here is one for you VMWARE gurus - particularly if you run ESXi without fancy drive arrays 
  6. 08/06/2015The Killer of Orphans (Orphan Documents) 
  7. 06/02/2015Homeopathic Marketing: Traveler on my Android is now calling itself VERSE. Allow me to translate that for the IBM Notes community... 
  8. 03/17/2015A review of British Airways Premium Economy Service – How to destroy customer goodwill all at once 
  9. 02/26/2015There's a bug in how @TextToTime() and @ToTime() process date strings related to international standards and browser settings. 
  10. 01/21/2015Delivering two new presentations at Developer Camp (EntwicklerCamp) 2015 in Germany 
Click here for more articles.....


pen icon Comment Entry
Subject
Your Name
Homepage
*Your Email
* Your email address is required, but not displayed.
 
Your thoughts....
 
Remember Me  

Please wait while your document is saved.