Chapeau

I released version 0.1.2 of SPQR this morning; it is available from gemcutter (as an installable gem package) or from fedorahosted.org (as source). This version contains many enhancements and fixes when compared to the versions described my previous SPQR posts. I've noticed that people are still installing older versions of SPQR and would like to encourage everyone who's interested in using SPQR to adopt the most recent version.

The 0.1.x series and the 0.0.x series are not API-compatible, but the incompatibility is for a good reason: it enables SPQR to expose normal Ruby methods (that is, those that don't use keyword arguments). To see what I mean, compare the "Hello, world" example from v0.1.2 with its counterpart in v0.0.4. (The former is also embedded below, but you might have to click through to the post to see it if you're reading this in a syndication feed.)

Here are some other enhancements to the code since the last time I mentioned SPQR:

• Many fixes and enhancements to SPQR/Rhubarb (simple object-graph persistence) integration; most glue methods are now automatically generated at runtime.
• Improved code generation from XML QMF schema files, including support for generating classes that are both exposed over QMF (with SPQR) and persistent (with Rhubarb).
• A stable and mostly-repeatable test suite. (I hope to write up the process of developing unit tests for QMF agents at some point in the near future.)
• Enhancements to the app skeleton, which can now specify the broker that manages the QMF bus.
• Many fixes and stability enhancements, most of which have come out of my experience with a substantial SPQR/Rhubarb application I'm developing.

The full list is here. Enjoy, and please don't hesitate to write with questions.

Here are two quick notes (and a bonus meta-note) about the quickly-evolving SPQR project:

1. SPQR now includes a gem target (courtesy of jeweler). Pull a recent version and sudo rake install to try it out. Once the project has stabilized a little more, I will publish gems to gemcutter.
2. I'm mirroring the fedorahosted SPQR repository on github; if you're already a github user, you might rather follow the repository there.

I'm pretty excited about SPQR, but this site won't become "all SPQR, all the time." Since interested hackers can easily follow the day-to-day project status on github or fedorahosted, I will reserve future SPQR-related blog posts for substantial announcements.

In a previous post, I introduced SPQR and presented a couple of examples of how one could use SPQR to publish Ruby objects over QMF. Sometimes, though, you aren't starting from an application -- instead, you're starting from an XML QMF schema document. SPQR includes spqr-gen, a tool designed to automatically generate a skeleton SPQR application from a QMF schema. In this post, we'll see an example of spqr-gen in action.

First, let's look at a simple QMF schema for a class that exposes one method, echo, which returns its argument (note that the code examples may not show up if you're viewing this in a feed reader):

Running spqr-gen on this example produces two files: agent-app.rb and examples/codegen/EchoAgent.rb. As you can see, these two files contain all of the boilerplate we need to start implementing these QMF methods --- and, by chance, the boilerplate methods actually have the behavior that our agent is meant to!

spqr-gen is included in the SPQR repository; the system as documented in the last two posts is tagged "introducing-spqr".

SPQR is a framework to make it almost painless to create QMF agents in the Ruby language, and thus to write Ruby applications that can be managed remotely. I built it to serve as infrastructure for some new development at work, but am pleased to announce that it is now functional enough to be useful for a range of applications, and you can download it from its git repository on fedorahosted.org and try it out. If you know what a "QMF agent" is, then I hope you're already interested in learning more --- please skip ahead to the examples to see it in action! If you're confused, fear not and read on.

Background

AMQP is an open standard for interprocess messaging --- and an excellent basis for distributed applications with even particularly demanding requirements. I'm most familiar with the Apache Qpid implementation of AMQP, since many of my colleagues at Red Hat are Qpid committers. Qpid is scalable, high-performance, and open-source --- it also features a variety of language bindings.

One great feature of Qpid that isn't in other AMQP implementations is the Qpid Management Framework. QMF is essentially a protocol so that applications can allow themselves to be managed remotely. In this way, it is similar to familiar remote procedure call mechanisms: a server application publishes an interface specification, and client programs are able to interact with that interface by invoking proxy methods that forward their arguments to the server machine and ferry return values back to the client. Unlike RPC, however, QMF is built upon Qpid messaging, and so offers low-latency, high throughput, and great resiliency.

In the QMF world, "server" applications --- those that can be managed --- are called agents, and "client" applications are called consoles. (Consoles and agents communicate by connecting to the same Qpid message broker.) The interface to an agent is called a schema; while schemas are instantiated programmatically in the applications that use them, there is also a well-defined format for recording schema information in XML.

In older versions of Qpid, it was only possible to develop QMF agents in C++; with more recent versions (including the one packaged in Fedora 12), it is possible to use Ruby as well. Since I am currently developing a rather substantial QMF agent application in Ruby, I wanted to have robust infrastructure that would allow me to avoid repeating myself, skip writing boilerplate as much as possible, and get on with the interesting work of my actual application. SPQR is built on top of the existing Ruby QMF engine and allows developers to publish instances of standard Ruby classes (with minimal and unobtrusive annotation) over QMF.

Examples

The first example we'll run through is (of course) a very simple application that publishes one method, which merely greets the user. Ours will be a little more interesting than classic "Hello, world!" programs, though, since we'll provide a personalized greeting (and publish our answer to another process). Here's what our application would look like in plain Ruby:

Note that we're using keyword-style arguments for the hello method, since we'd like to be able to support out-parameters (that is, not merely a single return value). Otherwise, this is pretty straightforward. Here's what it would look like if we used SPQR to publish Hello.hello over QMF:

The core of the class is basically the same (although we do collect statistics now, primarily to demonstrate SPQR's support for QMF statistics). The only differences are in the annotations we've added in order to enable SPQR to manage this class (viz., mixing in the SPQR::Manageable module), and to tell SPQR which methods to expose over QMF (and the types of their arguments). We've also added find and find_all methods to make it possible for QMF console applications to query for a particular Hello object (or for all of them). (Since the Hello class has no state, we simply make it a singleton class and the find methods just return the sole instance.)

The second example is a little more interesting: it shows the intersection of SPQR's capability to publish objects to QMF and my Rhubarb library's capability to store specially-declared Ruby classes in SQLite databases. (You'll need SQLite and its Ruby bindings installed to run this example.) This example is a simple logging agent that exposes two classes: LogService, which is a singleton class that can generate log records, and LogRecord, which is a class backed by a database table that models these records.

Conclusion

SPQR is still under development, and is only available via git for the moment. However, if you're interested in developing QMF agent applications in Ruby, it supports a useful subset of QMF and presents a clean, simple way to expose your applications over QMF. I welcome comments, bug reports, feature requests, and patches.

In my last post, I mentioned a problem (choosing between one of several templated classes based on information that won't be available until runtime) and also mentioned a workaround involving virtual dispatch, in which the templated class extends a non-templated class. This doesn't address all of the concerns I set out with regard to other solutions, but it is rather cleaner than any of them since it makes conditional dispatch implicit. Here's an example of this very simple workaround as applied to a trivial class:

Here’s an interesting problem involving C++ templates. Say you have a class that is parameterized on a term — we'll use std::bitset<N> as a running example — and you’d like to wrap this class in another class that hides the value of N from its clients.

Of course, it isn’t possible to instantiate a template based on a value that isn’t a compile-time constant, but it may be desirable to choose from one of several (statically-defined) values at run-time. In the std::bitset case, we might define “small,” “medium,” and “large” sizes for a representation class, and have the constructor for wrapper class choose the representation that is large enough to hold the set of bits it is created with. We’d like wrap bitset in such a way so that its size is not an explicit part of the type of the wrapper class (and thus clients can write code that handles wrapped-bitset objects of varying sizes without parameterizing their code). This is a problem, because the size of bitset is part of its type.

In general, this problem manifests whenever a wrapper class can make a run-time decision about which of some static set of template parameter values for its (hidden) representation class is most appropriate; or, still more generally, when it makes sense to choose a different type of representation based on a run-time value. Note that this problem isn't confined to classes parameterized on terms; a similar problem could also appear with classes parameterized on types, like STL containers. However, it is less immediately clear to me what a solution for classes parameterized on types would look like in the general case, since the type parameter is more likely to be exposed in the container's interface. (There are special cases, like choosing different types of containers as an internal representation depending on the characteristics of different collections of homogeneous data, that might make sense.)

There are a few approaches to wrapping a class in C++, but none of them are particularly suitable for solving this problem. Here’s why:

1. Public inheritance does not shield clients from the template value in the representation class, because it must be a template parameter to the wrapper class. However, public inheritance frees the wrapper class implementor from implementing (potentially) many proxy methods that simply call out to to the representation class.
2. Explicit delegation requires the wrapper class implementor to write proxy methods for each operation of interest from the representation class. However, the template parameter to the representation class is still part of its type. Therefore, there must be several fields in the wrapper class that hold pointers to instances of different instantiations of the representation class template (e.g. std::bitset<32>, std::bitset<128>, and std::bitset<2048>). In addition, each proxy method must include a conditional expression that determines which of these objects is the actual representation and a pointer dereference. (Having the wrapper class include three std::bitset values, rather than pointers, would defeat the purpose of having “small,” “medium,” and “large” representations.)
3. Private inheritance combines the problems of the first two approaches with the advantages of neither.

A general solution to this problem at the type-system level might involve constrained template parameters and union types. So you could say, e.g., std::bitset<N in {32,128,2048}> b = std::bitset<32>(); This solves our problem by encoding a range of possible sizes in the type of a bitset-valued variable rather than encoding a particular size. (My gut feeling — although I don’t have the turnstiles to prove it — is that this feature might be difficult or impossible to include in a sound, statically-checked type system, let alone to implement in a production compiler.) Sadly, extending the language is rarely a viable option for most software projects.

There are, of course, ways to solve the problem of needing a set of bits large enough to hold k bits, where k isn't known until runtime: use std::bit_vector or use std::vector<std::bitset<32> >, to give two simple examples. More generally, it would also be possible to use virtual functions in a base representation class that is extended by a templated class: then derived<32> and derived<128> could both be pointed to by a representation*. But I wonder, since I’m continually amazed by feats of template metaprogramming, what clean and efficient solutions to the general problem are possible?

Due to some hardware trouble with my main work machine, I'm presently working in a virtual machine on my personal computer. After a few dim trails, I found a pretty straightforward method to clone my work computer into a virtual machine image, so that I am able to work in the exact same environment I would have on my physical work computer. Here's how to do it:

1. Clone the drive using dd (the following example assumes your drive is /dev/sda and you have an external drive mounted at /media/removable:
   
2. Use qemu-img to convert the raw bits of the drive to an image in the appropriate format for the virtual machine monitor you want to use (QEMU or VMWare):
   
3. Create a new virtual machine that uses this drive image, using the interface for your preferred virtual machine monitor.

I was able to image and convert a 100 gb drive in around six hours. My drive was an LVM volume and the home partition was encrypted with LUKS; I was delighted to see that qemu-img handled these oddball features of my drive flawlessly. (I can't think of a technical reason why these wouldn't be supported, but I'm nonetheless inclined to be pleasantly surprised when things work as they should out of the box.)

I'm Will Benton and I'll be posting some short technical articles related to my work -- specifically, programming languages, broadly construed, and high-throughput, high-performance, and distributed computing.