NetKernel + Kowari

| | Comments (2) | TrackBacks (0)

NetKernel has a great pipes/filters framework for composing services. Kowari is the only non-rdbms backed RDF triple-store with support for queries against datetime data types.

Sadly, building a web application or service in Kowari kinda sucks. But, theoretically, Kowari is embeddable. So I set out to verify this assertion and to gain more knowledge about the inner workings of NetKernel.

On my road to discovery, I encountered a few issues only now partly resolved.

First, Kowari attempting to load a resource (config file) from it's own jar file failed. A call to ClassLoader.getSystemResource() was replaced by me with this.getClass().getClassLoader().getResource() to get by the failure.

Oddly this thread describes the same problem under Tomcat. But the suggestion posted does not work under NetKernel, namely Thread.currentThread().getContextClassLoader().getResource(CONFIG_PATH). But my workaround does not work under Tomcat. This may be a NetKernel issue. Thus far I've chosen to not formally lay blame.

Second, the JSR 173 libraries used by Kowari also fail due to ClassLoader issues. Moving them to the 'endorsed' classpath directory for NetKernel resolves the issue. Note the FactoryFinder class in the JSR implementation uses something rather funky, namely:

        try {
            // Construct the name of the concrete class to instantiate
            Class clazz = Class.forName(FactoryFinder.class.getName()
                                        + "$ClassLoaderFinderConcrete");
            ClassLoaderFinder clf = (ClassLoaderFinder) clazz.newInstance();
            classLoader = clf.getContextClassLoader();
        } catch (LinkageError le) {
            // Assume that we are running JDK 1.1, use the current ClassLoader
            classLoader = FactoryFinder.class.getClassLoader();
        } catch (ClassNotFoundException x) {
            // This case should not normally happen.  MS IE can throw this
            // instead of a LinkageError the second time Class.forName() is
            // called so assume that we are running JDK 1.1 and use the
            // current ClassLoader
            classLoader = FactoryFinder.class.getClassLoader();
...

Again I'll refrain from laying blame as the BEA guys have obviously put some thought into the issue.

Third, Kowari has no obvious (pre-built) way to stream their version of a ResultSet, called Answer, to XML. There is a BEEP related package with an answer streamer, but if failes on an XML BlankNode (something one would find very common in the wild). A fix for this was quite simple, yet I'm left with the nagging impression there is a better way.

Fourth, NetKernel is really just that, a kernel for managing the interaction and scheduling between components executing in the vm, called modules. But I haven't found support for explicit life-cycle management. That is, there is no init(), start() or stop() type methods on a module. It seems to rely on finalizers to clean up resources. Thus, shutting down a NetKernel instance corrupts the Kowari database.

Hopefully answers to my queries in their respective forums lead to code updates and answers so I can continue this. Would be nice to start work on the backend for a recent side project.

[11-16-04 - See update]

0 TrackBacks

Listed below are links to blogs that reference this entry: NetKernel + Kowari.

TrackBack URL for this entry: http://www.manamplified.org/cgi-bin/mt-tb.cgi/247

2 Comments

Killing Kowari at any stage won't corrupt commited data. If you were doing a load or inserting data with autocommit set off then you will lose that data - this is correct behaviour. But previously commited data should be there no matter if you kill the process, turn off the power or whatever.

It's coincidental that you want to serialize an Answer to XML, as I was looking at something similar.

When moving an Answer over the network (using RMI - but it extends to any protocol) I broke it up into pages which I called an AnswerPage. These hold the same data as an Answer, only with a fixed number of rows (eg. 1024 rows). I used these to stream an Answer in chunks.

The coincidental part is that my serialization of AnswerPage (actually, AnswerPageImp) is a similar problem to serializing a whole Answer to XML. In my case I implemented AnswerPageImpl.writeExternal() to write out the number of columns, rows, the column names, the compression level, and the column data. At the moment I use the default serialization of the individual elements to serialize the data, but I need to change this to strings anyway, as the default serialization has extra unneeded data, and takes too long to get data out by reflection.

This would be trivial to modify to output XML instead. You'd just want to put a similar method into the AnswerImpl class.

Also, while Kowari will report that it wasn't shut down properly, it hasn't actually corrupted the database. The datastore goes to great pains to make sure that the data on disk is ALWAYS consistent. If you forcably kill the program, the worst that can happen is that you lose a recent write. Closing everything just gives the program the opportunity to truncate any unused portions at the end of the data files. If this gets missed for some reason, then on the next startup the file will get truncated correctly anyway, (and maybe even pick up any completed writes that didn't get properly registered). We just report a heap of errors to help us track down any code that doesn't close things like we were hoping for. There are no consequences. (We were talking about removing these warning messages just this morning)

Leave a comment