Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

Professional Services

Second Signal

Presentations

Andrew's Blog

Support

Creating a Rules Engine class in Lotusscript while cutting process time by more than 1000%

By Andrew Pollack on 02/03/2006 at 08:55 AM EST

Its time for a technical deep dive. I haven't done one of these in a long time and I know some of you couldn't care less about firefighting, shirts, or my daughter's soccer team (which is kicking butt).

As anyone who reads this site regularly knows, I've been working on a service bureau application around timely pager notifications and Voice over IP. One of the key features of the application is the ability to allow each users to have multiple paging or text messaging devices (typically cell phones, alpha pagers, and regular email drops), and for each of those devices to be activated under different circumstances. A typical site using this will have roughly one hundred people, each with an average of 1.5 devices. Since different rules can apply to different devices, it gets rather complex picking the right devices to use in each circumstance.

The original code was never really intended to be more than a proof of concept. It was never intended to scale up at all and has proven to be as poorly performing as expected. What it did was to start with a constructed search query which picked out the "rules" matching the basic parameters of the query, then go through each one and test specific aspects of the rules to compile a list of just the ones which match. After finally picking out the matches, it had to look up the devices which are referenced by all those rules and send the text message to the devices. The process could take as long as a few minutes when running on a low end linux based server. It was also a bit buggy where dealing with time slots and needed attention even if I was to keep using it the same way.

I've been considering the best approaches to take, and had narrowed down to the following options:

a) Fix the bugs and try to optimize the existing code by increasing the complexity of the full text search query, thus relying less on follow-up processing by looping through the documents. I'd had some excellent results with this technique for other applications.

b) Fix the bugs, optimize a bit, then pre-build results for the more performance dependant calls. For example, a scheduled agent could pre-populate results for "tone activation" pages for each hour or half hour of the day. The results could be stored as records in folders or fields on profile documents. This is a refinement of the basic idea that you want to take as much of the work out of "live" processes and put it in scheduled ones so that the live ones appear faster. The downside to this is that changes the users make don't get picked up very quickly, and the number of these folders or profile documents you have to use increases pretty quickly the more you try to use this method.

c) Pull the data out of Notes NSF files, and use a relational database. For large scale applications this would be faster and more flexible, but performance at the smaller end of the scale is not too great. DB2NSF -- the new features in Domino 7 which integrate DB2 directly with the NSF data store would probably be a good choice, and this is still in strong consideration but for now it increases complexity more than I'd like.

No matter what I did, clearly the first thing was to find a better way of handling the time slots which were causing the most complexity problems. Going down this road ended up producing a result so fast that pre-storing the results or moving them to a relational datastore wasn't necessary. The only problem with it was that the processing is complex enough that it could get tricky to maintain. That's when I decided to make it a class of its own.

Here's what I came up with....

First, I created two views. The first is a simple lookup style view containing only device records. There are three columns; the device owner, the device name, and the address to send the messages for that device. A device is uniquely identified by a combination of the owner of the device and its name. One of the beautiful things about this method, however, is that this view does not need any sorted columns. As a result, the views' performance is excellent.

The second view contains the rules, and it is much more complex. It contains seven columns, as follows:

Column 1: The owner of this rule
Column 2: The device to which this rule applies
Column 3: The rule applies only when the priority is one of the following: Low, Medium, High, Urgent
Column 4: The rule applies only when the category of message is one of the following: (there are a several keyword options here)
Column 5: Active Time Slots -- There are 96 time slots in the day, each 15 minutes long
Column 6: Active Days -- Seven possible values (Mon, Tue, ...)
Column 7: Who can send messages to this user (can be other people or groups)

Column 5 is the most complex. Starting with two time values (start and stop), the formula explodes these into a list of all the time slots during which the rule is active. The way the math works, a time slot including 6:40pm would be identified as 18.5 -- the 18th hour long slot, half way through the time slot. Here's the formula:

Start :=@Round( ((@Hour(st) * 60 )+ (@Minute(st)) + 1) / 60; 0.25);
End:= @Round(((@Hour(et) * 60 )+ (@Minute(et))+ 1) / 60; 0.25);
c := @For( n := 0 ; n <= 24 ; n := n + 0.25 ;
t := t : @If( (n >= start ) & (n <= end) ; @Text(n) ;"" ) );
@TextToNumber(@Unique(@Trim(t)))

Of course, it would be much better to do this calculation on the record and store the result as a single field at save time rather than doing it as a view column each time the view is indexed. The column formula works while you're trying things out, but ultimately that is the direction I'll go. It could probably be much more efficiently stored by using the logical "OR" to mathematically combine all the slots into a large value or creating a single long string where each entry represents one slot. I haven't done much testing with it, but in theory that would be potentially faster. In practice, perhaps not such much.

Usually, you'd want to sort these columns and set each column to treat multiple values as a separate entry. In this case, that would potentially result in more than 100,000 entries for just one hundred users, and with 7 columns all sorted on all these entries the view indexing would be pretty intense. Instead, I did the opposite. Only the owner's name in the first column is sorted, and all of the columns are left to keep the multiple values on the same view entry. The obvious advantage here is that we're down to only 150 view entries, and a single column sort. The indexing is almost instant.

Here's where it all comes together

When the code runs to match a message that needs to get sent to the rules which apply to it, both views are quickly read into memory all at once. Once in memory, the values are simply compared in the order they came in. Because the application is designed to handle groups of around a hundred people, the time required to read each view into memory in its entirety is actually quite small. Remember, in Domino a view is itself just another kind of Note. You're only making a single read from the datastore when you read a view unless that view is very large. Its the performance equivalent of opening a single document.

The problem with this is that its a pretty complex task. You have to have an in-memory data structure to hold the unique layout of each view in memory, and the process of checking the values is itself tricky. Done carefully, its very manageable but you wouldn't want to have multiple copies of it floating around. That's where a CLASS comes in so handy.

I created a "RulesEngine" class, which when created immediately reads both views into its own in-memory structures. The class then has a method which can be called with a "Mask" which is really just a custom data type of the same kind used to store each record from the view. The values on the mask are compared with those from each in-memory entry to see which apply.

The code for this is below. Keep in mind its not fully tested yet -- only two of the mask values have been tested for accuracy so far so its likely there are still a few bugs in there. Also note that the final column -- who is allowed to send messages to the device this rule references -- is not tested. The simple reason for that is that if done by simply using that value as a readerNames field and running this code under the authority of the person sending the message, the rule will never appear in the list for users who cannot use it. That's real security, not just obscurity.

Ok, you've been warned, so here's the not fully tested code:

This code shows a simple test case for the class

Sub Initialize
Print Time$
Dim rulesEngine As New rulesengineclass
Dim v As Variant
Dim entrymask As rulesentrytype
entrymask.weekday = "Sun"
entrymask.activehour = 13
v = rulesengine.getDevicesByRulesEntryMask( entrymask)
Forall e In v
Print e
End Forall
Print Time$
End Sub

Here is the class itself, and the structure definitions it needs

Type rulesentrytype
owner As Variant
device As Variant
priority As Variant
type As Variant
activehour As Variant
weekday As Variant
allowedsenders As Variant
End Type

Type deviceentrytype
owner As Variant
device As Variant
address As Variant
End Type

Class rulesEngineClass

Public rulesindex List As rulesentrytype
Public deviceindex List As deviceentrytype
Public lastResultCount As Long
Public classlog As String

Public Function getDevicesByRulesEntryMask( entrymask As rulesentrytype) As Variant
Dim v As Variant
Dim returnlist List As String
Dim returncount As Long
v = getRulesByEntryMask(entrymask)
Dim idx As String
Dim rule As rulesentrytype
Dim device As deviceentrytype
Forall ruleindexentry In v
If Iselement(rulesindex(ruleindexentry)) Then
rule = rulesindex(ruleindexentry)
idx = ""
If Isarray( rule.owner) Then idx = rule.owner(0) Else idx = rule.owner
idx = idx + "~"
If Isarray(rule.device) Then idx = idx + rule.device(0) Else idx = idx + rule.device
If Not Iselement(returnlist(idx)) Then
If Iselement(deviceindex(idx)) Then
returncount = returncount + 1
device = deviceindex(idx)
If Isarray(device.address) Then returnlist(idx) = device.address(0) Else returnlist(idx) = device.address
End If
End If
End If
End Forall
getdevicesbyrulesentrymask = returnlist
lastresultcount = returncount
End Function

Public Function getRulesByEntryMask(entrymask As rulesentrytype) As Variant
On Error Goto errorhandle
' should handle priority, type, activehour, weekday, and allowed senders
Dim check As Boolean
Dim returnlist List As String
Dim returncount As Long
Dim trap As String
Forall rule In rulesindex
check = True
' check priority
If Not entrymask.priority = "" Then
If Isnull (Arraygetindex(rule.priority, entrymask.priority, 5)) Then check = False
If check = False Then trap = "Priority"
End If
' check type
If check = True And Not entrymask.type = "" Then
If Isnull (Arraygetindex(rule.type, entrymask.type, 5) ) Then
If Isnull (Arraygetindex(rule.type, "Any", 5)) Then check = False
End If
If check = False Then trap = "Type"
End If
' check send to list!
If check = True And Isarray(entrymask.owner) Then
check = False
If Arraygetindex(entrymask.owner, "All Members" , 5) >=0 Then check = True
Forall o In entrymask.owner
If check = False Then
If Arraygetindex(rule.owner, o , 5) >=0 Then check = True
End If
End Forall
If check = False Then trap = "To"
End If
' check weekday
If check = True And Not entrymask.weekday = "" Then
If Isnull(Arraygetindex(rule.weekday, entrymask.weekday)) Then check = False
If check = False Then trap = "Weekday"
End If
' check activehour
If (check = True) And (Not entrymask.activehour = "")Then
If Isnull(Arraygetindex(rule.activehour, entrymask.activehour) ) Then check = False
If check = False Then trap = "Hour"
End If
If check=True Then
returnlist(Cstr(returncount)) = Listtag(rule)
returncount = returncount + 1
End If
If Not trap = "" Then classlog = classlog & trap & Chr$(13) & Chr$(10)
End Forall
alldone:
getrulesbyentrymask = returnlist
lastresultcount = returncount
Exit Function
errorhandle:
classlog = "Compare error: " & Erl & Error$
Resume alldone
End Function

Public Sub new()

Dim session As New notessession
Dim thisdb As notesdatabase
Set thisdb = session.currentdatabase

Dim ruleschartview As notesview
Dim devicelookupview As notesview

Dim rulesentry As rulesentrytype

Dim deviceentry As deviceentrytype

Set ruleschartview = thisdb.getview("ruleschart")
Dim counter As Long
Dim vn As notesviewnavigator
Set vn = ruleschartview.CreateViewNav
Dim entry As notesviewentry
Set entry = vn.GetFirstDocument
Dim x As Integer
Dim sarray(0) As String
While Not entry Is Nothing
rulesentry.owner = entry.columnvalues(0)
If Not Isarray(rulesentry.owner) Then
sarray(0) = rulesentry.owner
rulesentry.owner = sarray
End If
rulesentry.device = entry.columnvalues(1)
rulesentry.priority = entry.columnvalues(2)
rulesentry.type = entry.columnvalues(3)
rulesentry.activehour = entry.columnvalues(4)
rulesentry.weekday = entry.columnvalues(5)
rulesentry.allowedsenders = entry.columnvalues(6)
rulesindex(Cstr(counter)) = rulesentry
counter = counter + 1
Set entry = vn.GetNextDocument(entry)
Wend
Set devicelookupview = thisdb.getview("devicelookup")
Set vn = devicelookupview.CreateViewNav
Set entry = vn.GetFirstDocument
counter = 0
Dim idx As String
While Not entry Is Nothing
idx = ""
deviceentry.owner = entry.columnvalues(0)
deviceentry.device = entry.ColumnValues(1)
deviceentry.address = entry.columnvalues(2)
If Isarray(deviceentry.owner) Then idx = deviceentry.owner(0) Else idx = deviceentry.owner
idx = idx + "~"
If Isarray(deviceentry.device) Then idx = idx + deviceentry.device(0) Else idx = idx + deviceentry.device
deviceindex( idx ) = deviceentry
counter = counter + 1
Set entry = vn.GetNextDocument(entry)
Wend
End Sub
End Class

There are - loading - comments....

My own thoughts on this are...By Bob Balaban on 02/03/2006 at 07:47 PM EST

kewl post.
How can you cut ANYTHING more than 100%?

Sorry. Was using Republi-math -- what I meant was...By Andrew Pollack on 02/03/2006 at 08:10 PM EST

It takes as little as 1/1000 the time to process.

Subject
Your Name
Homepage
*Your Email
	* Your email address is required, but not displayed.

Your thoughts....


	Remember Me

Andrew Pollack's Blog

Creating a Rules Engine class in Lotusscript while cutting process time by more than 1000%

Site Links

Useful Links

Other Recent Stories...