Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

A weird java problem in Domino leads me to this challenging call for help. Anyone?

By Andrew Pollack on 08/08/2006 at 11:00 PM EDT

I have a client application out there that kicks off a number of threads. The agent is failing to open some databases -- in a consistent and predictable but otherwise nonsensical pattern. When we spotted the pattern, it looked like we had it figured out. Lets assume the databases are db01, db02, db03, ...... db50. The agent would consistently fail to open db03, db06, and so on -- but not in an every 3rd database pattern. Here is the pattern of skipped databases: 3,6,9, 14, 17, 20, 23, 25, 28, 31, 34, 36, 39, 42, 45, 47, and 50.

I've narrowed the code down to what you see below. Each database is spawned off on its own thread as its own process. The very first thing the thread does us call session.getDatabase(). As you see, I've wrapped that call in its own try/catch framework. No exception is thrown. The database object is not null, but neither is it open. This is consistent with failure to open the database when you've left the optional "create on file" to either false or not set it (as is the case here).

So when we saw the consistent debugging reports:

Unable to open db03
Unable to open db06
Unable to open db09

We thought we had it. Some stupid problem with those database ACL's. Nope, no cigar. We we tried searching JUST the failed databases, can you guess which ones failed? In searching just the db03, db06, and db09 databases -- two worked, one didn't. db03 and db06 worked fine and db09 failed.

In other words, regardless of which database order you use, the third, sixth, ninth, fourteenth, and so on are the ones which fail!

Consider these additional facts:

1. There is a variable number of concurrent threads. The startup process will kick off one new thread for each database until that limit is reached, at which time it will wait in a loop at low priority until the number of running threads drops (a database is finished) and kick off the next. Other than the file name of the database, the objects are identical.

2. Varying the maximum number of threads -- initially set at 5, dropped to 2, and set as high as 10 -- had no impact at all on the result.

3. The Domino server is 6.5.5 with fixpack 1 --running on win32.

Here's the important bits of code which run when the new process is started:

Database sdb = null;
try {
sdb = session.getDatabase( serverName, dbName ) ;
} catch(Exception e) {
if(debug) logItem(" X5 " + this.getName() + ": " + e.toString());

if (sdb != null) {

if ( sdb.isOpen() ) {

/* <--- code removed ---> */
} else {

logItem("Unable to open " + dbName);

} // end of check for database.isopen()

} else {

logItem("Unable to access " + serverName + "!!" + dbName);

} // endof check for null database

There is literally no code here being executed in any kind of unique way. This is the first bit of code a new db object runs when its thread is started. Each db object is on its own and is unaware of its order in the process or of any other processing threads. Nothing here suggests a pattern.

I believe there is some outside limiting factor at play here.

There are  - loading -  comments....

My own thoughts on this are...By Mikkel Heisterberg on 08/09/2006 at 01:05 AM EDT

what an intriguing problem!! :-)

Given that you ruled out the obvious issues with ACL and access in general I'm
not going to go there. I'm instead thinking of problems with allocating memory
or similar or other resource bottlenecks inside the underlying DLLs that might
trick your code. Are the problems only apparent when running on the server? I
guess it would be worth a try to run the code outside of the Domino server as a
strandalone program and/or to see how the code fares it you use the Corba
classes instead of the DLL-versions.

This may be obvious but you do remember to initialize the Domino environment
per thread right (by extending NotesThread or similar)?

Let me know.

*What if you try to create a session for each one of the threads and see what happens? By Gerald Mengisen on 08/09/2006 at 02:44 AM EDT
My own thoughts on this are...By Jan on 08/09/2006 at 04:12 AM EDT
"... startup process will kick off one new thread for each database until that
limit is reached, at which time it will wait in a loop at low priority until
the number of running threads drops (a database is finished) and kick off the
next. "

Just a thought, but what if it doesn't wait, but moves on and just returns an
empty object until the number of threads drops?

Another thought, how many concurrent agents are allowed to run on the server?
Are new threads handled by the server as a new agent session?

I have no idea what i'm talking about here. Just offloading my thougths ;-)
What happens if...By Richard Schwartz on 08/09/2006 at 07:55 AM EDT
Have you taken the threading out of the picture entirely by calling that same
code from the main loop instead of dispatching the threads? If so, what
Thanks for the comments. Solution follows....By Andrew Pollack on 08/09/2006 at 11:43 AM EDT
I made a change forcing the creation of a new session on each threaded
object. In addition, I added a 200ms wait time before terminating the thread
after allowing the local session and the database to recycle.

In addition, there was a rich text issue involved. You may have recalled my
earlier rant about non-summary data getting to a Java agent. The solution was
to write the data to rich text and parse it in the agent. The string in rich
text kept getting line feeds inserted into it after so many characters. Since
this client used databases named in a pattern, the spaces always hit in the
same place -- frequently in the middle of the database path name. I corrected
this by adding my own line feeds after each entry in the rich text and trimming
the entries in the java agent. THAT is ugly.

Other Recent Stories...

  1. 01/26/2023Better Running VirtualBox or VMWARE Virtual Machines on Windows 10+ Forgive me, Reader, for I have sinned. I has been nearly 3 years since my last blog entry. The truth is, I haven't had much to say that was worthy of more than a basic social media post -- until today. For my current work, I was assigned a new laptop. It's a real powerhouse machine with 14 processor cores and 64 gigs of ram. It should be perfect for running my development environment in a virtual machine, but it wasn't. VirtualBox was barely starting, and no matter how many features I turned off, it could ...... 
  2. 04/04/2020How many Ventilators for the price of those tanks the Pentagon didn't even want?This goes WAY beyond Trump or Obama. This is decades of poor planning and poor use of funds. Certainly it should have been addressed in the Trump, Obama, Bush, Clinton, Bush, and Reagan administrations -- all of which were well aware of the implications of a pandemic. I want a military prepared to help us, not just hurt other people. As an American I expect that with the ridiculous funding of our military might, we are prepared for damn near everything. Not just killing people and breaking things, but ...... 
  3. 01/28/2020Copyright Troll WarningThere's a copyright troll firm that has automated reverse-image searches and goes around looking for any posted images that they can make a quick copyright claim on. This is not quite a scam because it's technically legal, but it's run very much like a scam. This company works with a few "clients" that have vast repositories of copyrighted images. The trolls do a reverse web search on those images looking for hits. When they find one on a site that looks like someone they can scare, they work it like ...... 
  4. 03/26/2019Undestanding how OAUTH scopes will bring the concept of APPS to your Domino server 
  5. 02/05/2019Toro Yard Equipment - Not really a premium brand as far as I am concerned 
  6. 10/08/2018Will you be at the NYC Launch Event for HCL Domino v10 -- Find me! 
  7. 09/04/2018With two big projects on hold, I suddenly find myself very available for new short and long term projects.  
  8. 07/13/2018Who is HCL and why is it a good thing that they are now the ones behind Notes and Domino? 
  9. 03/21/2018Domino Apps on IOS is a Game Changer. Quit holding back. 
  10. 02/15/2018Andrew’s Proposed Gun Laws 
Click here for more articles.....

pen icon Comment Entry
Your Name
*Your Email
* Your email address is required, but not displayed.
Your thoughts....
Remember Me  

Please wait while your document is saved.