Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application

By Andrew Pollack on 02/20/2012 at 01:00 PM EST

I've waited a long time before trying to write anything serious in XPages. I've known about them, and even extolled the potential of them as far back as late 2007 when I wrote the very first public article about them. I said then, and continue to say that whether or not XPages really live up to their potential will depend entirely on how well they're implemented in the client. Until 8.5.3 I have not had any confidence that they were stable enough to be worth working in. Thanks in large part to the work of people in the community who have pushed it, report its problems, and created the useful documentation that IBM has opted to leave out, I decided it's time to put the effort in and see what I can create with it.

As I said four years ago, to me the primary value that XPages brings to the table is the ability to focus on the page presented to the user, rather than on the individual data notes that underlay it one at a time. Most applications require a user to work with more than one interrelated document in much the same way that a relational database handles related tables. Prior to XPages, there's been no natural mechanism to do that smoothly with Notes and Domino. This one piece of functionality SHOULD make developing applications with XPages so much easier than it has ever been before.

If only it were that simple. I'm really trying to be fair here -- I want to love XPages, regardless of my current state of disgust with some of the IBM executives responsible for the Notes client.

The Original Application

The goal is to take an existing web application and create a new web front end for it using XPages. The web application has a lot of complex embedded html and css, but under that it's actually made up of a few really simple forms. There's a primary "User" form, and then each "User" can have zero, one, or many "Objects" associated with them. There are two kinds of objects, but they both are fairly simple forms based items and each is stored in a single document. There is a one-to-many relationship of users to objects, and they're tied together based on the user id. The app has other things in it, but this simple explanation covers the majority of what it does. For scale, each kind of object is in its own database. There is a database of "User" documents, and two databases each with their own kind of "Object" documents in them.

Sounds like an absolutely ideal target for an XPages application, doesn't it?

The Goal

Since this is the first phase of the project, all I wanted to do was present on a single XPAGE a couple of editable fields on the "USER" document, then have a list of the existing "Objects" of one type below that, each editable in place, and a "New" object of that type below the list. So a primary document, a repeating, editable set of existing related object documents, and a new object document which can be submitted. I'm not in the least bit interested in what it looks like at this point. I'm going for a purely functional use case test. I believe that if I can make this use case work smoothly, I'll have about 90% of everything I want to be able to do with Notes data in an XPage figured out. I can use the design patterns I've learned there on other projects to my hearts content because really, they're all the same. Oh, there's other stuff I'll need to do -- call agents, do some obscure lookups, etc, but really I can do all that with client side javascript if need to.

Here's What I Found

First of all, yes - it can be done. Technically, the promise of working with the related documents through the repeat controls can be made to work quite nicely. In fact, the repeat control itself has become "reasonably" straightforward. The hardest part is figuring out what you can actually pass to it so it knows what to repeat, and where to plug in that value, values, list, or code result -- and in the case of a code result, what format and type it can be. You see, there is virtually no reasonable documentation provided with the product by IBM. There are resources out there to find, but almost nothing in the product itself. Personally, I suspect this is because if they include documentation within the product, they have to QA that documentation and they have to provide it translated in several languages. That means spending money on headcount. Far better to just dump it out there, then maybe get involved with a book or two or even better get other people to write their documentation.

Now, would it be SO HARD to include something on this pane that indicated what the value, values, or objects that this field will accept are, as I've done below? This isn't a new product any more. It's 4 years on the market and several software revisions down the road and we still have this problem. This complete lack of contextual assistance is one of the critical factors that makes XPages such a complete pain in the backside to learn. It's a problem that is repeated on virtually every property box that accepts a scripted result as an option. None of them actually tell you what kind of a result the field is looking for. Is this laziness, or is it because IBM is too cheap to hire someone to write and QA the translations that would be required?


That's just one example, but there are hundreds of others. Where there is mouse-over help, it is singularly not helpful. For example, mouse over the option labeled "DataContexts". It sounds promising, I wonder what that does? According to the mouse-over pop-up help it "Controls data context lists". What the hell does that mean? Where is the link that lets me explore that topic in the documentation provided? Oh, wait, there is none. No link, no documentation.

There are a bunch of tricks to doing what I wanted to do. Thanks to work done by Matt White, David Leedy, and others I was able to find the hidden secret settings you have to use make my little application do what I want. Mostly. I'm not going to document those for you here yet because frankly, I'm not that good at it yet and you're better off looking at Matt's and David's work directly. When I have a solid handle on it, I think I can add to the community by documenting things in my own way, but that needs to wait until I'm sure I'm giving out good information.

I ran into several other challenges along way:

Server Crash: If, after delving deep into the various settings, I found the option to "computewithform" when I load or save a document which I created based on that form and turned it on -- my server crashed immediately on submit of that form. When I say crash, I mean complete and total crash with no errors or warnings visible. I had to use nsd -kill to clear it from memory and restart it. The crash turned out to be because of a bad subform in use on the form that had become corrupted in a previous version of designer. Nothing else using that subform caused server to crash, however. The real issue for me with the server crash, is that I've got this massive "xpages engine" thing running on my server and I have virtually no insight as to what it's actually doing.

ComputeWithForm: It still fails to compute with form on save. It will on load, but not save. If its turned on, the document fails to save. No further error is generated anywhere. It just doesn't work. I have no idea yet why. So much for using existing functionality. I'll have to re-do any calculations at save time to make sure the document ends up saved with the right values on it. Again, the big problem here is a total lack of insight as to what's not working, why it's not working, or even the fact that it isn't working.

Re-posting data on refresh: As much as I like the idea that my interactive page can submit and refresh parts of the page during its use, if the user ever does do something silly like use their browser's "Reload" button, the results are unpredictable. Whatever the last update the user attempted was, will be repeated. That can result in blank documents, duplicate submits, replication conflicts, and a general deviation from expected behavior. So, while it looked good initially, now I've got to go and create a routine to call after submitting anything on the page, that does a "blank" post to a document that is discarded at the server rather than saved so that if the user does do a refresh that is the last thing that was posted. That's just dumb.

Overall impression:

XPages can be learned, and once learned it can be used to create good applications. I suppose we'll call that a win. If the goal was to create a new design paradigm that would attract new developers to the platform by providing an IDE that didn't require the years of background, long lists of workarounds, and a wealth of community knowledge just to build half decent applications -- then its an utter failure.

I'll probably continue to work at it, and will likely build new applications, as I need them, in XPages for as long as I stay with the Notes & Domino platform. I'm just going to have to settle in and accept that developing Notes and Domino applications is going to be just has painful and require as much or more community knowledge as it always has. The opportunity to do this in a way that could seriously attract new attention to what is still the most solid and functional back end platform available was just entirely missed when IBM insisted on using the Trilog application that most of the programming community had already rejected as the basis to do it.


There are  - loading -  comments....

re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Keith Strickland on 02/20/2012 at 03:59 PM EST
Andrew,

For more documentation on the components take a look at
http://www.openntf.org/xspext/xpages%20extension%20library%20documentation.nsf/x
pages-doc/Controls.html

This documents all the components by tag (well, most of them anyways) and lists
all the properties (and property type) of the component. It's very handy.

Also, as for a what you can feed a repeat control. Take a look at the
java.util.Collection API. Anything that implements Collection can be passed
along to a repeat control.

HTH
Keith
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Andrew Pollack on 02/20/2012 at 04:01 PM EST
Keith -- great points. What a shame it's not there on panel, or in "click for
more information" link on the panel -- or SOMEWHERE that a responsible software
company that wanted to sell product might put it.
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Jesper Kiaer on 02/20/2012 at 04:45 PM EST
I agree with yours views on XPages. As I have written about XPages before I do
think it is based on the wrong technology (JSF), instead of something light and
agile like Play Framework. At least IBM could have invested in superb
documentation, but they didn't. The Mastering XPages book is good, but late and
not part of the core product documentation.
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Ursus on 02/21/2012 at 02:53 AM EST
Couldn't agree with you more - I am currenly learning XPages (have been a Notes
admin and Programmer for about 15 years now) and am having a hell of a time. I
have been on training at IBM and have the Mastering XPages book! Ok, maybe its
just me but finding the correct switches somewhere on the panels has just
become unusable!
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By David Leedy on 02/21/2012 at 08:51 AM EST
Go to XPages101.net or my own XPages.TV for training help.
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By GarryL on 02/21/2012 at 04:07 AM EST
This is a bit worrying, really. We are looking at standardising our web
development tools and, having had notes for many years, we considering XPages
(we are moving from client based apps to web based).

It seems extremely odd, IBM having virtually zero documentation, especially
given the wealth of other well versed tools out there. I guess along with most
companies we would not have time to fight with the enviroment.

Hmmm...maybe time to look further afield then.
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Giulio Campobassi on 02/21/2012 at 05:58 AM EST
I believe Einstein said it best... "Any intelligent fool can make things bigger
and more complex... It takes a touch of genius - and a lot of courage to move
in the opposite direction.".. Seemed to work well for Steve Jobs...
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By David Leedy on 02/21/2012 at 08:53 AM EST
there's not great "documentation" from IBM, but there's more then you think.
There are many tutorials published now and a lot of documentation on the wiki.
I just saw a tutorial the other day about Mobile App development that was very
helpful to a problem I was having.
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Andrew Pollack on 02/21/2012 at 09:07 AM EST
David, lots of community built tutorials -- and some IBM created
documentation. Almost none in the product itself, and what contextual help
there is within the IDE is shockingly bad.

Why? I contend its a simple matter of IBM not being interested in spending the
time and money to hire technical writers, editors, and translators to provide
this material at a professional level within the product.
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Julian Buss on 02/21/2012 at 06:01 AM EST
I only partially agree. Yes, for a Lotus Notes developer it's quite a learning
curve to master XPages, but once you "got it", it's a very productive and nice
development environment (at least from my point of view).

Developers who already have deep knowledge of Java, JavaScript, Dojo etc. (that
means, any decent web developer) often find it easy to work with XPages, since
it uses all the standards they already know. So i think IBM went the right way
here.

Yes, there could be more context documentation. Yes, there could be more
documentation right in Domino Designer.
But I don't think that's the point. There is the "Mastering XPages" book out
there, there are plenty of web sites out there, there are plenty of other devs
out there willing to help, so I think there is enough learning support.

I think the point is that it's very hard for existing Lotus Notes developers to
dive into the world of standard web development. To learn stuff like Dojo and
Java and JavaScript. But frankly, that's the road you have to go at some point
in your live, since classic Notes development is dead.

And it's the same effort you would need to take if you want to learn any other
web development platform.
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Jason Hook on 02/21/2012 at 08:31 AM EST
Julian I'm broadly in agreement with you. It is a steep learning curve for
some but I think it rewards persistence. Encouragingly there's a developing
community of people who are giving freely of their expertise. I suffered a lot
of growing pains even despite having some background in java and JSP
development. Once over though I've grown to appreciate xPages and it will tale
me a long time to forget how painful traditional Domino development is.
Really?By Andrew Pollack on 02/21/2012 at 09:05 AM EST
Julian, I write in Java, Javascript, HTML, CSS, XML, C , vb.NET, Lotuscript,
and Formula Language. I have production applications at customer sites using
every one of those.

Yes, XPages USE more standards, but that doesn't mean they are used well, it
doesn't mean they are organized well, and it doesn't mean they are documented
well.
re: Really?By jason hook on 02/21/2012 at 09:24 AM EST
"I write in Java, Javascript, HTML, CSS, XML, C , vb.NET, Lotuscript, and
Formula Language. I have production applications at customer sites using every
one of those."

Me too (except for C) xPages hurt, but I'm over it now and I'm productive. I
wish it hadn't hurt some much but I hadn't had to learn so much in a while.

I don't think the client compares well with say visual studio 2010 but after a
time that seems less important.

There are too many little windows but two monitors helps (as does the ability
to consolidate the windows and pull them out of the main window).

In my own experience becoming fluent in the client and my new apps has helped.
It's still way better than the old way of writing Domino Web apps without in
the old stateless way.
re: Really?By Dwight Wilbanks on 02/22/2012 at 03:27 AM EST
Or say Visual Studio 6
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Toby Samples on 02/21/2012 at 09:17 AM EST
Julian,

I am in total agreement with you, I didn't have any Notes experience previous
to coming to Xpages, I do however have many years experience using asp.net and
was able to grasp the architecture pretty easily. I feel the learning curve is
a lot less for someone with J2EE or asp.net experience as it is just another
flavor of component based development, where building Notes Client apps is much
more of a paradigm shift. Yes the documentation isn't the best, But the
learning curve is doable thanks to the huge community efforts.
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Mike Dill on 02/21/2012 at 11:37 AM EST
Per Julian, "...classic Notes development is dead..." ????

Hmmmm. I work for the largest engineering company in the world, and we're
still have 6,000 business apps using "classic" Notes development, including
"classic" Domino web methods. They run great, require less time to develop than
when attempting to use XPages (which I have done as well), and are far more
stable.

Guess which books I am now going back to refer to for web development --
"Advanced Domino 5 Web Programming" by Rose Kelleher, "IBM Lotus Domino:
Classic Web Application Development Techniques" by Richard Ellis, and IBM's own
"Lotus Domino Release 5.0: A Developer's Handbook".

Guess some would say that I'm stuck in the old world. But you know what?? Our
business customers are extremely happy with the products.
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Henning Heinz on 02/21/2012 at 12:44 PM EST
In a closed source environment you cannot develop against the vendor of the
product. Classic Notes development, as it currently is, is doomed. It is not
allowed to grow so your possibilities will get frozen at a level that was
already considered old many years ago.
Ironically with the forthcoming "Classic" browser plug-in I expect the old
technology will even get a small push again.
And even more ironically is that the classic Notes applications are what is
keeping Notes in so many companies.
But in my opinion IBM has already moved on. What you currently get are the
results from the lights out teams. They are probably doing their best but at
that level they would need some kind of magic to get all the stuff fixed and
enhanced in a way that it attracts existing and new customers.
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Miguel Calvo on 02/21/2012 at 01:01 PM EST
I've been developing in Domino for 14 years, and I'm pretty active with XPages
right now. I agree that documentation should improve and hope IBM is reading
this thread, but the framework is really powerful to mimic ( and much more )
Notes Apps RAD capacity in web development once you have overcome the learning
curve.

@Mike I'm sure you can get the same results with Classic development than
with XPages ( a web app is just HTML and javascript ). You only need your own
framework, that many of us have developed during the ages. But the core
product, without your personal framework, is nowadays useless for novices.

Besides, if you want your framework to adapt to modern development,
including Ajax, REST consuming, mobile development, OAuth authentication and so
on you have to write that for yourself, and that's something you could find in
the core XPages framework and in the XPages Extension Library, that is
constantly getting improved by IBM.

Last month I delivered a "classic web Domino development and XPages" training
event. People ( new to Domino ) was frankly amazed by how difficult and in some
ways weird, was to create a web app with classic Domino web development.
Xpages, on the other hand, cheered them a lot.

Xpages is the way to go, modern and constantly evolving. Classic development,
still needed ( All of us do have customers with many applications non XPages
based ), but something that should be considered for maintenance only.
Lack of documentation is now just an excuseBy Tim Tripcony on 02/21/2012 at 12:42 PM EST
When I was initially learning XPages, I too bemoaned the lack of documentation.
But that was over three years ago; nowadays, if there's something I want to do,
and my F3 key doesn't tell me how, Google will.

Every resource that is out there now was created by someone who learned XPages
when those resources didn't exist. It is far easier now to learn XPages than it
was 3 years ago when Google couldn't just tell you how. But the plethora of
resources now available such as xpages101 and xpages.tv are evidence that a
lack of in-product documentation is just an excuse. The people who created
those resources learned the techniques they describe without the detailed
documentation we're used to from the LotusScript era. Dig in, experiment, and
reach out to others when you get stuck. There's been a big push over the last
month to get XPage developers using StackOverflow... if you encounter a hurdle
and post a question there, I highly doubt that you'll have to wait longer to
get an answer than it would take to sift through in-product documentation had
IBM bothered to explicitly spell out everything the platform can now do.

Most of all, stick with it. Every single XPage developer I know who is now
churning out awesome stuff had a period of about 3 weeks to 3 months during
which the adjustment was difficult, if not frustrating. But invariably their
persistence paid off. There was some "light bulb" moment when it just clicked,
and now they're building amazing applications. XPages aren't an incremental
addition to the platform... they're a complete revolution of what is possible.
You can't immediately transition from being an expert on the old stuff to being
an expert on this. You have to be a novice for a while. And that's gonna suck.
But it's SO worth it in the long run.
Lack of documentation is no excuseBy Andrew Pollack on 02/21/2012 at 02:22 PM EST
It's no excuse. It's something (else) you can work around, but no excuse. It
is one of the top 5 things that has hurt uptake of xpages as design platform
outside the microcosm of blog-connected community participants.

It's also a deplorable and pathetic measure taken by IBM to save money at the
cost of a quality product. Documentation and QA have been reduced to the
smallest perfect of development budget ever, and as a result we get products
that are buggy, regression bugs galore in each release, and documentation
driven by a community of interest rather than a vendor.
re: Lack of documentation is no excuseBy Mike Dill on 02/21/2012 at 04:44 PM EST
My personal two cents worth (which may be worth only a single penny to some of
you)..........Once SharePoint can handle the complete equivalent of what can be
done with Notes Rich Text, then it'll probably cause the demise of
Notes/Domino. Don't get me wrong -- I'm not FOR that. I believe Notes/Domino is
the best RAD platform on the market by a long shot. But since the purchase of
Lotus by IBM, IBM has done to the Lotus products the same thing they do to
every piece of software they are associated with. They "enhance" it until its
dead.

Yes, even as large of a Domino infrastructure as our company has in place (and
it's LARGE and has been in place since the early 1990's), we now have
SharePoint as well.

I will not like it when Notes/Domino is no longer in the market, but I will
very likely be retired by that point anyway.
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Per Henrik Lausten on 02/22/2012 at 03:48 AM EST
To be fair, IBM has provided documentation and learning resources for XPages -
primarily on the Lotus Notes and Domino Application Development wiki at
http://www-10.lotus.com/ldd/ddwiki.nsf.

I have recently written a blog post with links to the IBM documentation and
links to other very useful learning resoures on XPages:
http://per.lausten.dk/blog/2012/02/learning-xpages-available-resources.html
re: Delving into XPages: Wins, Losses, and abject FAILs -- What it has been like to start transforming an existing application By Andrew Pollack on 02/22/2012 at 10:45 AM EST
How much of that in the language you paid for with your domino server? How
well integrated with designer?


Other Recent Stories...

  1. 01/26/2023Better Running VirtualBox or VMWARE Virtual Machines on Windows 10+ Forgive me, Reader, for I have sinned. I has been nearly 3 years since my last blog entry. The truth is, I haven't had much to say that was worthy of more than a basic social media post -- until today. For my current work, I was assigned a new laptop. It's a real powerhouse machine with 14 processor cores and 64 gigs of ram. It should be perfect for running my development environment in a virtual machine, but it wasn't. VirtualBox was barely starting, and no matter how many features I turned off, it could ...... 
  2. 04/04/2020How many Ventilators for the price of those tanks the Pentagon didn't even want?This goes WAY beyond Trump or Obama. This is decades of poor planning and poor use of funds. Certainly it should have been addressed in the Trump, Obama, Bush, Clinton, Bush, and Reagan administrations -- all of which were well aware of the implications of a pandemic. I want a military prepared to help us, not just hurt other people. As an American I expect that with the ridiculous funding of our military might, we are prepared for damn near everything. Not just killing people and breaking things, but ...... 
  3. 01/28/2020Copyright Troll WarningThere's a copyright troll firm that has automated reverse-image searches and goes around looking for any posted images that they can make a quick copyright claim on. This is not quite a scam because it's technically legal, but it's run very much like a scam. This company works with a few "clients" that have vast repositories of copyrighted images. The trolls do a reverse web search on those images looking for hits. When they find one on a site that looks like someone they can scare, they work it like ...... 
  4. 03/26/2019Undestanding how OAUTH scopes will bring the concept of APPS to your Domino server 
  5. 02/05/2019Toro Yard Equipment - Not really a premium brand as far as I am concerned 
  6. 10/08/2018Will you be at the NYC Launch Event for HCL Domino v10 -- Find me! 
  7. 09/04/2018With two big projects on hold, I suddenly find myself very available for new short and long term projects.  
  8. 07/13/2018Who is HCL and why is it a good thing that they are now the ones behind Notes and Domino? 
  9. 03/21/2018Domino Apps on IOS is a Game Changer. Quit holding back. 
  10. 02/15/2018Andrew’s Proposed Gun Laws 
Click here for more articles.....


pen icon Comment Entry
Subject
Your Name
Homepage
*Your Email
* Your email address is required, but not displayed.
 
Your thoughts....
 
Remember Me  

Please wait while your document is saved.