Introducing SPQR

spqr
qpid
qmf
ruby
Published

November 21, 2009

SPQR is a framework to make it almost painless to create QMF agents in the Ruby language, and thus to write Ruby applications that can be managed remotely. I built it to serve as infrastructure for some new development at work, but am pleased to announce that it is now functional enough to be useful for a range of applications, and you can download it from its git repository on fedorahosted.org and try it out. If you know what a “QMF agent” is, then I hope you’re already interested in learning more — please skip ahead to the examples to see it in action! If you’re confused, fear not and read on.

Background

AMQP is an open standard for interprocess messaging — and an excellent basis for distributed applications with even particularly demanding requirements. I’m most familiar with the Apache Qpid implementation of AMQP, since many of my colleagues at Red Hat are Qpid committers. Qpid is scalable, high-performance, and open-source — it also features a variety of language bindings.

One great feature of Qpid that isn’t in other AMQP implementations is the Qpid Management Framework. QMF is essentially a protocol so that applications can allow themselves to be managed remotely. In this way, it is similar to familiar remote procedure call mechanisms: a server application publishes an interface specification, and client programs are able to interact with that interface by invoking proxy methods that forward their arguments to the server machine and ferry return values back to the client. Unlike RPC, however, QMF is built upon Qpid messaging, and so offers low-latency, high throughput, and great resiliency.

In the QMF world, “server” applications — those that can be managed — are called agents, and “client” applications are called consoles. (Consoles and agents communicate by connecting to the same Qpid message broker.) The interface to an agent is called a schema; while schemas are instantiated programmatically in the applications that use them, there is also a well-defined format for recording schema information in XML.

In older versions of Qpid, it was only possible to develop QMF agents in C++; with more recent versions (including the one packaged in Fedora 12), it is possible to use Ruby as well. Since I am currently developing a rather substantial QMF agent application in Ruby, I wanted to have robust infrastructure that would allow me to avoid repeating myself, skip writing boilerplate as much as possible, and get on with the interesting work of my actual application. SPQR is built on top of the existing Ruby QMF engine and allows developers to publish instances of standard Ruby classes (with minimal and unobtrusive annotation) over QMF.

Examples

The first example we’ll run through is (of course) a very simple application that publishes one method, which merely greets the user. Ours will be a little more interesting than classic “Hello, world!” programs, though, since we’ll provide a personalized greeting (and publish our answer to another process). Here’s what our application would look like in plain Ruby: 

class Hello
   def hello(args)
     args["result"] = "Hello, #{args['name']}!"
    end
end

Note that we’re using keyword-style arguments for the hello method, since we’d like to be able to support out-parameters (that is, not merely a single return value). Otherwise, this is pretty straightforward. Here’s what it would look like if we used SPQR to publish Hello.hello over QMF: 

require 'spqr/spqr'
require 'spqr/app'
require 'logger'
 
class Hello
   include SPQR::Manageable
   def hello(args)
     @people_greeted ||= 0
     @people_greeted = @people_greeted + 1
     args["result"] = "Hello, #{args['name']}!"
    end
 
   spqr_expose  do |args|
     args.declare , , 
     args.declare , , 
   end
 
   # This is for the service_name property
   def service_name
     @service_name = "HelloAgent"
   end
 
   spqr_package 
   spqr_class 
   spqr_statistic , 
   spqr_property , 
   
   # These should return the same object for the lifetime of the agent
   # app, since this example has no persistent objects.
   def Hello.find_all 
     @@hellos ||= [Hello.new]
   end
 
   def Hello.find_by_id(id)
     @@hellos ||= [Hello.new]
     @@hellos[0]
   end
end
 
app = SPQR::App.new( => )
app.register Hello
 
app.main

The core of the class is basically the same (although we do collect statistics now, primarily to demonstrate SPQR’s support for QMF statistics). The only differences are in the annotations we’ve added in order to enable SPQR to manage this class (viz., mixing in the SPQR::Manageable module), and to tell SPQR which methods to expose over QMF (and the types of their arguments). We’ve also added find and find_all methods to make it possible for QMF console applications to query for a particular Hello object (or for all of them). (Since the Hello class has no state, we simply make it a singleton class and the find methods just return the sole instance.)

The second example is a little more interesting: it shows the intersection of SPQR’s capability to publish objects to QMF and my Rhubarb library’s capability to store specially-declared Ruby classes in SQLite databases. (You’ll need SQLite and its Ruby bindings installed to run this example.) This example is a simple logging agent that exposes two classes: LogService, which is a singleton class that can generate log records, and LogRecord, which is a class backed by a database table that models these records. 

#!/usr/bin/env ruby
 
# This is a simple logging service that operates over QMF.  The API is
# pretty basic:
#   LogService is a singleton and supports the following methods:
#    * debug(msg)
#    * warn(msg)
#    * info(msg)
#    * error(msg)
#   each of which creates a log record of the given severity,
#   timestamped with the current time, and with msg as the log
#   message.
#
#   LogRecord corresponds to an individual log entry, and exposes the
#   following (read-only) properties:
#    * l_when (unsigned int), seconds since the epoch corresponding to
#      this log record's creation date
#    * severity (long string), a string representation of the severity
#    * msg (long string), the log message
#
# If you invoke logservice.rb with an argument, it will place the
# generated log records in that file, and they will persist between
# invocations.
 
require 'spqr/spqr'
require 'spqr/app'
require 'rhubarb/rhubarb'
 
class LogService
  include SPQR::Manageable
 
  [, , , ].each do |name|
    define_method name do |args|
      args['result'] = LogRecord.create(>Time.now.to_i, >"#{name.to_s.upcase}", >args['msg'].dup)
    end
    
    spqr_expose name do |args|
      args.declare , , 
      args.declare , , 
    end
  end
 
  def self.find_all
    @@singleton ||= LogService.new
    [@@singleton]
  end
 
  def self.find_by_id(i)
    @@singleton ||= LogService.new
  end
 
  spqr_package 
  spqr_class 
end
 
class LogRecord
  include SPQR::Manageable
  include Rhubarb::Persisting
  
  declare_column , 
  declare_column , 
  declare_column , 
  
  # XXX: rhubarb should create a find_all by default
  declare_query , "1"
 
  spqr_property , 
  spqr_property , 
  spqr_property , 
 
  spqr_package 
  spqr_class 
 
  def spqr_object_id
    row_id
  end
end
 
TABLE = ARGV[0] rescue ":memory:" 
DO_CREATE = (TABLE == ":memory:" or not File.exist?(TABLE))
 
Rhubarb::Persistence::open(TABLE)
 
LogRecord.create_table if DO_CREATE
 
app = SPQR::App.new( => )
app.register LogService, LogRecord
 
app.main

Conclusion

SPQR is still under development, and is only available via git for the moment. However, if you’re interested in developing QMF agent applications in Ruby, it supports a useful subset of QMF and presents a clean, simple way to expose your applications over QMF. I welcome comments, bug reports, feature requests, and patches.

​