Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

A weird java problem in Domino leads me to this challenging call for help. Anyone?

By Andrew Pollack on 08/08/2006 at 11:00 PM EDT

I have a client application out there that kicks off a number of threads. The agent is failing to open some databases -- in a consistent and predictable but otherwise nonsensical pattern. When we spotted the pattern, it looked like we had it figured out. Lets assume the databases are db01, db02, db03, ...... db50. The agent would consistently fail to open db03, db06, and so on -- but not in an every 3rd database pattern. Here is the pattern of skipped databases: 3,6,9, 14, 17, 20, 23, 25, 28, 31, 34, 36, 39, 42, 45, 47, and 50.

I've narrowed the code down to what you see below. Each database is spawned off on its own thread as its own process. The very first thing the thread does us call session.getDatabase(). As you see, I've wrapped that call in its own try/catch framework. No exception is thrown. The database object is not null, but neither is it open. This is consistent with failure to open the database when you've left the optional "create on file" to either false or not set it (as is the case here).

So when we saw the consistent debugging reports:

Unable to open db03
Unable to open db06
Unable to open db09

We thought we had it. Some stupid problem with those database ACL's. Nope, no cigar. We we tried searching JUST the failed databases, can you guess which ones failed? In searching just the db03, db06, and db09 databases -- two worked, one didn't. db03 and db06 worked fine and db09 failed.

In other words, regardless of which database order you use, the third, sixth, ninth, fourteenth, and so on are the ones which fail!

Consider these additional facts:

1. There is a variable number of concurrent threads. The startup process will kick off one new thread for each database until that limit is reached, at which time it will wait in a loop at low priority until the number of running threads drops (a database is finished) and kick off the next. Other than the file name of the database, the objects are identical.

2. Varying the maximum number of threads -- initially set at 5, dropped to 2, and set as high as 10 -- had no impact at all on the result.

3. The Domino server is 6.5.5 with fixpack 1 --running on win32.

Here's the important bits of code which run when the new process is started:

Database sdb = null;
try {
sdb = session.getDatabase( serverName, dbName ) ;
} catch(Exception e) {
if(debug) logItem(" X5 " + this.getName() + ": " + e.toString());

}
if (sdb != null) {

if ( sdb.isOpen() ) {

/* <--- code removed ---> */
} else {

logItem("Unable to open " + dbName);

} // end of check for database.isopen()

} else {

logItem("Unable to access " + serverName + "!!" + dbName);

} // endof check for null database


There is literally no code here being executed in any kind of unique way. This is the first bit of code a new db object runs when its thread is started. Each db object is on its own and is unaware of its order in the process or of any other processing threads. Nothing here suggests a pattern.


I believe there is some outside limiting factor at play here.


There are  - loading -  comments....

My own thoughts on this are...By Mikkel Heisterberg on 08/09/2006 at 01:05 AM EDT
Andrew,

what an intriguing problem!! :-)

Given that you ruled out the obvious issues with ACL and access in general I'm
not going to go there. I'm instead thinking of problems with allocating memory
or similar or other resource bottlenecks inside the underlying DLLs that might
trick your code. Are the problems only apparent when running on the server? I
guess it would be worth a try to run the code outside of the Domino server as a
strandalone program and/or to see how the code fares it you use the Corba
classes instead of the DLL-versions.

This may be obvious but you do remember to initialize the Domino environment
per thread right (by extending NotesThread or similar)?

Let me know.

/lekkim
*What if you try to create a session for each one of the threads and see what happens? By Gerald Mengisen on 08/09/2006 at 02:44 AM EDT
My own thoughts on this are...By Jan on 08/09/2006 at 04:12 AM EDT
"... startup process will kick off one new thread for each database until that
limit is reached, at which time it will wait in a loop at low priority until
the number of running threads drops (a database is finished) and kick off the
next. "

Just a thought, but what if it doesn't wait, but moves on and just returns an
empty object until the number of threads drops?

Another thought, how many concurrent agents are allowed to run on the server?
Are new threads handled by the server as a new agent session?

I have no idea what i'm talking about here. Just offloading my thougths ;-)
What happens if...By Richard Schwartz on 08/09/2006 at 07:55 AM EDT
Have you taken the threading out of the picture entirely by calling that same
code from the main loop instead of dispatching the threads? If so, what
happens?
Thanks for the comments. Solution follows....By Andrew Pollack on 08/09/2006 at 11:43 AM EDT
I made a change forcing the creation of a new session on each threaded
object. In addition, I added a 200ms wait time before terminating the thread
after allowing the local session and the database to recycle.

In addition, there was a rich text issue involved. You may have recalled my
earlier rant about non-summary data getting to a Java agent. The solution was
to write the data to rich text and parse it in the agent. The string in rich
text kept getting line feeds inserted into it after so many characters. Since
this client used databases named in a pattern, the spaces always hit in the
same place -- frequently in the middle of the database path name. I corrected
this by adding my own line feeds after each entry in the rich text and trimming
the entries in the java agent. THAT is ugly.


Other Recent Stories...

  1. 10/08/2018Will you be at the NYC Launch Event for HCL Domino v10 -- Find me!Come find me in NYC on Wednesday at the Launch Event if you're there. I really do want to talk to ...... 
  2. 09/04/2018With two big projects on hold, I suddenly find myself very available for new short and long term projects. In twenty five years, I don't think I've ever written an entry like this, but if you need the kind of work I do now would be a great time to get in touch. Both of the big projects I had lined up for late summer and early fall have been placed on hold and will be that way for a while. With the kids now all off at college and careers, I'm open to more travel than such than I have been in decades, but unless something else comes along, I'll be here working on updates to Second Signal and other things that ...... 
  3. 07/13/2018Who is HCL and why is it a good thing that they are now the ones behind Notes and Domino?We need to address some biases here. IBM has made a deal under which the Notes & Domino software and intellectual property is now being developed and maintained by HCL America. HCL America is part of the very large "HCL Technologies" company that has grown from its roots in India to become an 8 Billion Dollar company with a global presence in the IT Industry. You could be excused for initially believing, as many people do when they hear this, that "they've outsourced the code to India where they'll milk it ...... 
  4. 03/21/2018Domino Apps on IOS is a Game Changer. Quit holding back. 
  5. 02/15/2018Andrew’s Proposed Gun Laws 
  6. 05/05/2016Is the growing social-sourced economy the modern back door into socialism? 
  7. 04/20/2016Want to be whitelisted? Here are some sensible rules for web site advertising 
  8. 12/30/2015Fantastic new series on Syfy called “The Expanse” – for people who love traditional science fiction 
  9. 10/20/2015My suggestion is to stay away from PayAnywhere(dot)com  
  10. 08/07/2015Here is one for you VMWARE gurus - particularly if you run ESXi without fancy drive arrays 
Click here for more articles.....


pen icon Comment Entry
Subject
Your Name
Homepage
*Your Email
* Your email address is required, but not displayed.
 
Your thoughts....
 
Remember Me  

Please wait while your document is saved.