Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

A weird java problem in Domino leads me to this challenging call for help. Anyone?

By Andrew Pollack on 08/08/2006 at 11:00 PM EDT

I have a client application out there that kicks off a number of threads. The agent is failing to open some databases -- in a consistent and predictable but otherwise nonsensical pattern. When we spotted the pattern, it looked like we had it figured out. Lets assume the databases are db01, db02, db03, ...... db50. The agent would consistently fail to open db03, db06, and so on -- but not in an every 3rd database pattern. Here is the pattern of skipped databases: 3,6,9, 14, 17, 20, 23, 25, 28, 31, 34, 36, 39, 42, 45, 47, and 50.

I've narrowed the code down to what you see below. Each database is spawned off on its own thread as its own process. The very first thing the thread does us call session.getDatabase(). As you see, I've wrapped that call in its own try/catch framework. No exception is thrown. The database object is not null, but neither is it open. This is consistent with failure to open the database when you've left the optional "create on file" to either false or not set it (as is the case here).

So when we saw the consistent debugging reports:

Unable to open db03
Unable to open db06
Unable to open db09

We thought we had it. Some stupid problem with those database ACL's. Nope, no cigar. We we tried searching JUST the failed databases, can you guess which ones failed? In searching just the db03, db06, and db09 databases -- two worked, one didn't. db03 and db06 worked fine and db09 failed.

In other words, regardless of which database order you use, the third, sixth, ninth, fourteenth, and so on are the ones which fail!

Consider these additional facts:

1. There is a variable number of concurrent threads. The startup process will kick off one new thread for each database until that limit is reached, at which time it will wait in a loop at low priority until the number of running threads drops (a database is finished) and kick off the next. Other than the file name of the database, the objects are identical.

2. Varying the maximum number of threads -- initially set at 5, dropped to 2, and set as high as 10 -- had no impact at all on the result.

3. The Domino server is 6.5.5 with fixpack 1 --running on win32.

Here's the important bits of code which run when the new process is started:

Database sdb = null;
try {
sdb = session.getDatabase( serverName, dbName ) ;
} catch(Exception e) {
if(debug) logItem(" X5 " + this.getName() + ": " + e.toString());

}
if (sdb != null) {

if ( sdb.isOpen() ) {

/* <--- code removed ---> */
} else {

logItem("Unable to open " + dbName);

} // end of check for database.isopen()

} else {

logItem("Unable to access " + serverName + "!!" + dbName);

} // endof check for null database


There is literally no code here being executed in any kind of unique way. This is the first bit of code a new db object runs when its thread is started. Each db object is on its own and is unaware of its order in the process or of any other processing threads. Nothing here suggests a pattern.


I believe there is some outside limiting factor at play here.


There are  - loading -  comments....

My own thoughts on this are...By Mikkel Heisterberg on 08/09/2006 at 01:05 AM EDT
Andrew,

what an intriguing problem!! :-)

Given that you ruled out the obvious issues with ACL and access in general I'm
not going to go there. I'm instead thinking of problems with allocating memory
or similar or other resource bottlenecks inside the underlying DLLs that might
trick your code. Are the problems only apparent when running on the server? I
guess it would be worth a try to run the code outside of the Domino server as a
strandalone program and/or to see how the code fares it you use the Corba
classes instead of the DLL-versions.

This may be obvious but you do remember to initialize the Domino environment
per thread right (by extending NotesThread or similar)?

Let me know.

/lekkim
*What if you try to create a session for each one of the threads and see what happens? By Gerald Mengisen on 08/09/2006 at 02:44 AM EDT
My own thoughts on this are...By Jan on 08/09/2006 at 04:12 AM EDT
"... startup process will kick off one new thread for each database until that
limit is reached, at which time it will wait in a loop at low priority until
the number of running threads drops (a database is finished) and kick off the
next. "

Just a thought, but what if it doesn't wait, but moves on and just returns an
empty object until the number of threads drops?

Another thought, how many concurrent agents are allowed to run on the server?
Are new threads handled by the server as a new agent session?

I have no idea what i'm talking about here. Just offloading my thougths ;-)
What happens if...By Richard Schwartz on 08/09/2006 at 07:55 AM EDT
Have you taken the threading out of the picture entirely by calling that same
code from the main loop instead of dispatching the threads? If so, what
happens?
Thanks for the comments. Solution follows....By Andrew Pollack on 08/09/2006 at 11:43 AM EDT
I made a change forcing the creation of a new session on each threaded
object. In addition, I added a 200ms wait time before terminating the thread
after allowing the local session and the database to recycle.

In addition, there was a rich text issue involved. You may have recalled my
earlier rant about non-summary data getting to a Java agent. The solution was
to write the data to rich text and parse it in the agent. The string in rich
text kept getting line feeds inserted into it after so many characters. Since
this client used databases named in a pattern, the spaces always hit in the
same place -- frequently in the middle of the database path name. I corrected
this by adding my own line feeds after each entry in the rich text and trimming
the entries in the java agent. THAT is ugly.


Other Recent Stories...

  1. 03/26/2019Undestanding how OAUTH scopes will bring the concept of APPS to your Domino serverWhile a full description of OATH is way beyond what I can do in this quick blog entry, I wanted to talk a bit about how "SCOPES" interact with the already rich authorization model used by Domino. Thanks to the fantastic work by John Curtis and his team, the node.js integration with Domino is going to be getting a rich security model. What we know is that a user's authorizations will be respected through the node.js application to the Domino server -- including reader names, ACLs, Roles, and so on. The way ...... 
  2. 02/05/2019Toro Yard Equipment - Not really a premium brand as far as I am concernedDear Toro Customer Service, I arm writing about the following machine: Toro Power Max 1120 OXEModel:38654S/N:31000#### Specifically, bearing part #:63-3450 This is the part ($15 online / $25 at the local dealer) that caused me to raise my objections on-line. This piece of garbage is supposed to be a bearing. It carries the shaft which drives both stages of the auger. The shaft passes through the bearing (which is what bearings do) after the auger drive pulley as the shaft goes through the back (engine ...... 
  3. 10/08/2018Will you be at the NYC Launch Event for HCL Domino v10 -- Find me!Come find me in NYC on Wednesday at the Launch Event if you're there. I really do want to talk to ...... 
  4. 09/04/2018With two big projects on hold, I suddenly find myself very available for new short and long term projects.  
  5. 07/13/2018Who is HCL and why is it a good thing that they are now the ones behind Notes and Domino? 
  6. 03/21/2018Domino Apps on IOS is a Game Changer. Quit holding back. 
  7. 02/15/2018Andrew’s Proposed Gun Laws 
  8. 05/05/2016Is the growing social-sourced economy the modern back door into socialism? 
  9. 04/20/2016Want to be whitelisted? Here are some sensible rules for web site advertising 
  10. 12/30/2015Fantastic new series on Syfy called “The Expanse” – for people who love traditional science fiction 
Click here for more articles.....


pen icon Comment Entry
Subject
Your Name
Homepage
*Your Email
* Your email address is required, but not displayed.
 
Your thoughts....
 
Remember Me  

Please wait while your document is saved.