Breakthroughs are funny. You'd think it would be frustrating to find out that what you've been doing isn't the best way to do it, but its the revelation itself that is exciting and leads to amazing success. I've you've been following my blogs as I bang my head against the wall with multi-threaded Java application programming, you've been hearing me tell the good news -- performance is fantastic, the threading works, the stability works, etc...

So today as I started fine tuning (and yes, I'm now getting sub second search results) I realized that it was time to go back and re-read the chapters for greater depth. The first (hundred) times though, you're skimming and hunting for nuggets to "make it work" and when you do, you're tempted to call it done. That's just not my nature. I can't help tinkering until it "feels right", and it just hasn't quite gotten there yet. You've all been there, too. You're writing code and it works but something tells you that you're making it harder than it has to be. You're passing too much stuff around and you know it. There's an inherent yuckiness. Today's milestone is one of understanding -- and results in an actual on-paper new schema that like any real solution is 100 times more simple, compact, flexible, resilient, and should be faster too. The amazing thing is that it does all the same things. Its not rocket science and its not revolutionary. Its just finally understanding how to do the same things I'm already doing -- in the 'Zen' of the language.

So, tomorrow morning the compiler comes out again and gone are the home made semaphores, gone is passing instantiated classes when static methods will do much better, and gone are storms of threads starting and stopping on a constant ad-hoc basis in favor of task specific service thread groups. I even have in mind a little monitoring daemon that will vary the size of the thread pool assigned to different queues based on the queue sizes in proportion to one another and on live results being reported by the threads telling where's they're "wait()ing" the most -- imagine that, a self tuning application.

I can't wait to write the "How-To" on this stuff -- its so much simpler than the books make it out to be. Look for a new series of "how-to" writeups to start up on this blog site next week.

--Andrew