Esteban's Blog

/dev/rnd

Knowledge Agent incremental change-set processing and binary diff

Incremental change-set processing

One of the new features introduced in Drools 5.1M2 is incremental change-set processing support by Knowledge Agent. This feature allows the Agent to apply resource’s modifications to the kbase in an incremental way.

Before this implementation, every time a monitored resource changed the Agent discarded the current kbase, created a new one and then it applied all the definitions not of the modified resource, but of all the monitored resources. As you can see, this approach has big issues:

If only one rule inside a monitored resources was changed, the entire kbase had to be recreated by compiling all the monitored resources again.
Rules are statefull; It is not a good idea to remove a rule from kbase if it didn’t change.

Resource’s definitions isolation

The first improvement over the old design is to isolate the definitions of each resource. Doing this, every time a resource is modified, all its definitions are removed from the current kbase, the resource is then compiled and added to kbase. The definitions of the unmodified resources are never removed from kbase, and even better, the kbase is never dropped. All the changes are applied into the current kbase.

Every time the Agent processes the resources of a change-set, it creates a Map to all their definitions. When a resource changes, the Agent uses this Map in order to know which definitions should be removed from the kbase. After this, the new resource is compiled, the Map is updated and the generated definitions are added to kbase.
By default the Agent will not use this new feature. If you wan’t to enable it you must set “drools.agent.newInstance” property to false in the Agent’s configuration.


KnowledgeAgentConfiguration aconf = KnowledgeAgentFactory.newKnowledgeAgentConfiguration();
// Incremental kbase build
aconf.setProperty("drools.agent.newInstance", "false");
KnowledgeAgent kagent = KnowledgeAgentFactory.newKnowledgeAgent(
"Agent Name", kbase, aconf);

Maintaining a map of resource’s definition allows the Agent to limit the scope of a resource modification, but there are still some things to improve. What happens with those definition that didn’t change inside a modified resource? What if the resource has 200 rules, but only one is changed? As far as we know, the 200 rules are going to be removed from kbase, compiled and added again. We need a way to detect the modifications inside a resource.

Binary Diff of resources

As I mentioned, it would be good to have a way to compare a resource with its modified version and only modify the kbase with the added/modified/removed definitions. In order to achieve this, the Agent performs a diff with the original resource’s definitions and the modified version of them before add them to kbase. When processing a modified resource, the Agent compiles it and iterates over each definition to find the ones that were added, modified or removed. Only them will be added/removed from kbase. The definitions that didn’t change remain in kbase.
Please note that the modified resource has to be compiled before the diff could be performed.
According to the definition’s type, the diff implementation varies:
Rules: For rules, the Agent searches for modifications in its attributes, LHS and RHS.
Queries: queries are always replaced on kbase wether they are modified or not.
Functions: The body of functions is evaluated in order to detect changes. The old version of a modified function is removed from kbase and replaced with its new implementation.
Other definitions: All other definitions are always replaced in kbase (like if they were modified). We expect to add better support for definition’s modification detection in further versions.

Conclusion

Using the incremental change-set processing feature of Knowledge Agent we can avoid two major problems:
Instead of recompile all the resources when only one has changed, the Agent can just compile the one who was modified. The agent’s kbase is updated instead of recreated.
Thanks to the binary diff applied to the definitions inside a modified resource, only those who has really changed are added to kbase (Right now this is 100% true only for rules). The state of the unmodified rules inside the kbase is maintained.

Advertisements

3 comments on “Knowledge Agent incremental change-set processing and binary diff

  1. Pingback: Knowledge Agent: Incremental Change Set Build Implementation « Esteban's Blog

  2. Poornima
    November 6, 2013

    Hi ,

    We have tried out the same as per the steps mentioned in this blog. Despite , we find that the knowledgebase is not getting updated with the latest rules , Could you please offer some more insights .

    • esteban
      November 6, 2013

      What are you exactly doing? How are you modifying the rules? Do you see anything strange in your logs?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Information

This entry was posted on March 25, 2010 by in drools.
%d bloggers like this: