Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

A weird java problem in Domino leads me to this challenging call for help. Anyone?

By Andrew Pollack on 08/08/2006 at 11:00 PM EDT

I have a client application out there that kicks off a number of threads. The agent is failing to open some databases -- in a consistent and predictable but otherwise nonsensical pattern. When we spotted the pattern, it looked like we had it figured out. Lets assume the databases are db01, db02, db03, ...... db50. The agent would consistently fail to open db03, db06, and so on -- but not in an every 3rd database pattern. Here is the pattern of skipped databases: 3,6,9, 14, 17, 20, 23, 25, 28, 31, 34, 36, 39, 42, 45, 47, and 50.

I've narrowed the code down to what you see below. Each database is spawned off on its own thread as its own process. The very first thing the thread does us call session.getDatabase(). As you see, I've wrapped that call in its own try/catch framework. No exception is thrown. The database object is not null, but neither is it open. This is consistent with failure to open the database when you've left the optional "create on file" to either false or not set it (as is the case here).

So when we saw the consistent debugging reports:

Unable to open db03
Unable to open db06
Unable to open db09

We thought we had it. Some stupid problem with those database ACL's. Nope, no cigar. We we tried searching JUST the failed databases, can you guess which ones failed? In searching just the db03, db06, and db09 databases -- two worked, one didn't. db03 and db06 worked fine and db09 failed.

In other words, regardless of which database order you use, the third, sixth, ninth, fourteenth, and so on are the ones which fail!

Consider these additional facts:

1. There is a variable number of concurrent threads. The startup process will kick off one new thread for each database until that limit is reached, at which time it will wait in a loop at low priority until the number of running threads drops (a database is finished) and kick off the next. Other than the file name of the database, the objects are identical.

2. Varying the maximum number of threads -- initially set at 5, dropped to 2, and set as high as 10 -- had no impact at all on the result.

3. The Domino server is 6.5.5 with fixpack 1 --running on win32.

Here's the important bits of code which run when the new process is started:

Database sdb = null;
try {
sdb = session.getDatabase( serverName, dbName ) ;
} catch(Exception e) {
if(debug) logItem(" X5 " + this.getName() + ": " + e.toString());

}
if (sdb != null) {

if ( sdb.isOpen() ) {

/* <--- code removed ---> */
} else {

logItem("Unable to open " + dbName);

} // end of check for database.isopen()

} else {

logItem("Unable to access " + serverName + "!!" + dbName);

} // endof check for null database


There is literally no code here being executed in any kind of unique way. This is the first bit of code a new db object runs when its thread is started. Each db object is on its own and is unaware of its order in the process or of any other processing threads. Nothing here suggests a pattern.


I believe there is some outside limiting factor at play here.


There are  - loading -  comments....

My own thoughts on this are...By Mikkel Heisterberg on 08/09/2006 at 01:05 AM EDT
Andrew,

what an intriguing problem!! :-)

Given that you ruled out the obvious issues with ACL and access in general I'm
not going to go there. I'm instead thinking of problems with allocating memory
or similar or other resource bottlenecks inside the underlying DLLs that might
trick your code. Are the problems only apparent when running on the server? I
guess it would be worth a try to run the code outside of the Domino server as a
strandalone program and/or to see how the code fares it you use the Corba
classes instead of the DLL-versions.

This may be obvious but you do remember to initialize the Domino environment
per thread right (by extending NotesThread or similar)?

Let me know.

/lekkim
*What if you try to create a session for each one of the threads and see what happens? By Gerald Mengisen on 08/09/2006 at 02:44 AM EDT
My own thoughts on this are...By Jan on 08/09/2006 at 04:12 AM EDT
"... startup process will kick off one new thread for each database until that
limit is reached, at which time it will wait in a loop at low priority until
the number of running threads drops (a database is finished) and kick off the
next. "

Just a thought, but what if it doesn't wait, but moves on and just returns an
empty object until the number of threads drops?

Another thought, how many concurrent agents are allowed to run on the server?
Are new threads handled by the server as a new agent session?

I have no idea what i'm talking about here. Just offloading my thougths ;-)
What happens if...By Richard Schwartz on 08/09/2006 at 07:55 AM EDT
Have you taken the threading out of the picture entirely by calling that same
code from the main loop instead of dispatching the threads? If so, what
happens?
Thanks for the comments. Solution follows....By Andrew Pollack on 08/09/2006 at 11:43 AM EDT
I made a change forcing the creation of a new session on each threaded
object. In addition, I added a 200ms wait time before terminating the thread
after allowing the local session and the database to recycle.

In addition, there was a rich text issue involved. You may have recalled my
earlier rant about non-summary data getting to a Java agent. The solution was
to write the data to rich text and parse it in the agent. The string in rich
text kept getting line feeds inserted into it after so many characters. Since
this client used databases named in a pattern, the spaces always hit in the
same place -- frequently in the middle of the database path name. I corrected
this by adding my own line feeds after each entry in the rich text and trimming
the entries in the java agent. THAT is ugly.


Other Recent Stories...

  1. 05/05/2016Is the growing social-sourced economy the modern back door into socialism?Is the growing social-sourced economy the modern back door into socialism? I read a really insightful post a couple of days ago that suggested the use of social network funding sites like “Go Fund Me” and “Kickstarter” have come about and gained popularity in part because the existing economy in no longer serving its purpose for anyone who isn’t already wealthy. Have the traditional ways to get new ventures funded become closed to all but a few who aren’t already connected to them and so onerous as to make ...... 
  2. 04/20/2016Want to be whitelisted? Here are some sensible rules for web site advertisingAn increasing number of websites are now detecting when users have ad-blocking enabled, and refuse to show content unless you "whitelist" their site (disable your ad-blocking for them). I think that is a fair decision on their part, it's how they pay for the site. However, if you want me (and many others) to white list your site, there are some rules you should follow. If you violate these rules, I won't whitelist your site, I'll just find content elsewhere. 1. The total space taken up by advertisements ...... 
  3. 12/30/2015Fantastic new series on Syfy called “The Expanse” – for people who love traditional science fiction[] “The Expanse” is a new science fiction series being broadcast onthe Syfy channelthis winter. It’s closely based on a series of books by author James S. A. Corey beginning with “Leviathan Wakes”. There are 5 books in the “Expanse” series so far. If you’re a fan of the novels you’ll appreciate how closely the books are followed.TIP: The first five episodes are already available on Syfy.com. If you’re having trouble getting into the characters and plot, use those to get up to speed.The worlds created for ...... 
  4. 10/20/2015My suggestion is to stay away from PayAnywhere(dot)com  
  5. 08/07/2015Here is one for you VMWARE gurus - particularly if you run ESXi without fancy drive arrays 
  6. 08/06/2015The Killer of Orphans (Orphan Documents) 
  7. 06/02/2015Homeopathic Marketing: Traveler on my Android is now calling itself VERSE. Allow me to translate that for the IBM Notes community... 
  8. 03/17/2015A review of British Airways Premium Economy Service – How to destroy customer goodwill all at once 
  9. 02/26/2015There's a bug in how @TextToTime() and @ToTime() process date strings related to international standards and browser settings. 
  10. 01/21/2015Delivering two new presentations at Developer Camp (EntwicklerCamp) 2015 in Germany 
Click here for more articles.....


pen icon Comment Entry
Subject
Your Name
Homepage
*Your Email
* Your email address is required, but not displayed.
 
Your thoughts....
 
Remember Me  

Please wait while your document is saved.