Like many of my colleagues on the MRG Grid team, I've been at Condor Week 2010 this week. Later today, I'll give a talk about some of the configuration management software we've been developing; for more details, you can visit getwallaby.com.
]]>We might want to just store employees in a hash, indexed by employee ID. If the hash had an entry for a given ID, we'd know that we had already looked up the employee with that ID. If not, we'd know that we needed to look up the employee with that ID and then place the employee object in the hash. This solution is straightforward, but could prove tedious to implement in practice. Fortunately, Ruby provides a mechanism to make this essentially transparent.
When you create a Hash in Ruby, you can create it in one of several ways. You can just call the constructor Hash.new (or {}), in which case attempting to look up a key that is not in the table will return nil. You can provide a single default object as an argument to the constructor, which will be returned instead of nil as the value for keys not in the table. Finally, you can provide a block that describes how to calculate the value for a missing key. This last option is interesting for the memoization application, since it mimics a straightforward implementation of call-by-need.
The above code snippet shows an example of this technique. When we ask employees for an employee ID that it doesn't already contain, it will invoke the block with itself and the missing key as arguments. In executing the block, employees will insert a key-value pair in itself corresponding to the missing key and the result of the method call with that key as an argument.
There are some caveats here. First, you need to make sure that your arguments are hashable. It is also somewhat clunkier to deal with multiple arguments, since they must be in a list. Also, it must be safe to memoize the function: that is, as long as the hash exists, the memoized method must return the same value every time it is invoked with a given argument. This technique isn't perfect (and almost certainly isn't novel), but it is a pretty slick way to keep from repeatedly invoking expensive methods and depends on nothing more than Ruby's standard library.
]]>If you've been using a version of the Wallaby agent prior to 0.3.5, you'll want to note that the Wallaby classes are now in a different QMF package. In older versions, the QMF package name was mrg.grid.config; now, the QMF package name is com.redhat.grid.config. (The ruby module names implementing Wallaby still begin with Mrg::Grid, but this is unlikely to affect most users.)
If you've been using version 0.3.5 of the Wallaby agent, then you'll want to note that most of the "getter" methods have been replaced by QMF properties. This should simplify most console applications. We've also renamed some methods to make their functionality clearer. The main changes are as follows:
For full documentation of the Wallaby API, please see the API documentation link on getwallaby.com.
]]>There are several new features and enhancements in this release, including the ability to access the current Qpid user and context from within managed methods. The most exciting new functionality, though, is support for QMF events. In order to use QMF events, you'll need a fairly recent version of the Qpid and QMF libraries from their source repository (later than revision 929717 for nearly-complete support; revision 942861 fixes a minor bug that won't affect most users). Once you have those installed, though, SPQR makes it characteristically easy to make QMF events; see the example below for details:
As usual, I welcome your feedback.
]]>Problems with application configuration generally fall into several categories: value-gap problems, default value problems, type-safety problems for the configuration parameter values themselves, and type-safety problems of the programmatic interface to the configuration. We'll discuss each of these in turn.
Truth-value gaps are a problem of classical logic: we would like some way to reason about propositions that are apparently neither true nor false (e.g. Aristotle's example that "there will be a sea battle tomorrow or there will not be a sea battle tomorrow" is true), but we cannot encode an unknown or indeterminate truth value as "true" or "false." (The three-valued logic Ł3 of Jan Łukasiewicz is one such system that addresses this problem.)
We can generalize the problem of truth-value gaps to many situations that come up in programming, for example:
In Condor, these sorts of problems arising from the configuration subsystem are typically handled in an ad hoc manner. In some places, the mere definition of a configuration parameter is enough to enable a feature; in others, the parameter must be defined and its string representation must include something that corresponds to "true" (one example is "the first character is either 't' or 'T'"). The current configuration API also makes it inconvenient (but not impossible) to determine if a parameter is undefined or set to false; in any case, the burden for sensibly treating value gaps falls to the programmer. (I see an interface that does not depend on programmer discipline to be a win over one that does.)
Configuration parameters should have sensible defaults. Unfortunately, the process of assigning default values may not be straightforward. A default value may not merely be a value but it may be the result of evaluating a function at runtime. Alternatively, defaults may depend on context: parameter FOO may have one value in subsystem X on platform Y, but another in subsystem Z or on platform W. (All but the last of these variable names have no deeper significance.)
In Condor, these defaults are handled in two ways: for some parameters whose defaults that are consistent across subsystems and platforms, the default is specified in the generic configuration file shipped with Condor. (About 275 parameters are given default values in this file.) For other parameters that require context-sensitive defaults, the default values are supplied as an extra parameter to configuration API functions at each call site; these may also be conditionally compiled, so that, for example, whether thread-based parallelism is available is defined to be true within the condor_collector on all platforms except Windows, but false everywhere else.
A configuration subsystem that supported specifying rich defaults --- including immediate values as well as expressions that would evaluate to the correct default depending on the context in which they were evaluated --- would free application programmers from the tedious and error-prone work of encoding defaults in every call to param().
Popular discussion of types and type safety is fraught with handwaving, imprecision, and nonsense (e.g. the ridiculous and contradictory appellation "dynamically typed," which is often applied to untyped languages). For the remainder of this discussion, I will be using definitions adapted from Luca Cardelli's survey chapter on type systems: namely, a type is the upper bound on a range of values, and a typed language is one in which nontrivial types can be ascribed to variables.
It should be clear that imposing a sufficiently expressive type system on configuration variables is generally desirable. A sufficiently expressive type system for an application like Condor would include not merely the classic types of low-level languages (e.g. unsigned int, boolean, character string, etc.), but also types that encode application-specific information (e.g. email addresses, hostnames, Pascal-style ranges, typed dictionaries). However, since configuration variables in Condor may be defined in terms of macro-expanded values or (in the case of some parameters) as ClassAd expressions to be evaluated later, it may be difficult to typecheck configuration variables a priori.
Another type-safety problem deals with how (or how frequently) values are computed. Some configuration parameters have fixed values for the life of a process, but others (e.g. "the current time" or "the result of taking a random element from this list") may change every time that they are evaluated. The idea of using type systems to track how an expression is evaluated (and not merely the shape of its result) is not new: Lucassen and Gifford proposed effect systems for functional languages in 1988, and computer music languages like Max/MSP and SAOL have long distinguished between expressions that are evaluated once and expressions whose values could change at various intervals (like control signals, which can change hundreds of times per second, or audio signals, which vary tens of thousands of time per second). In the context of configuring Condor, it is sufficient to track whether a parameter's definition is to be evaluated (at least) once or every time its value is requested from the configuration subsystem. The values of parameter definitions that need only be evaluated once can be memoized; currently, Condor's configuration subsystem doesn't support this sort of type information and any memoization is performed on an ad hoc basis in application code.
One of the main concerns behind thinking about improving Condor's configuration API is to reduce the potential avenues for error by developers. Adding a typed interface to typed configuration data greatly simplifies application code that uses the configuration subsystem and eliminates ad hoc, per-call-site typechecking of configuration values. (It shouldn't be acceptable to support a high-level configuration API in 2010 that might require clients to manually coerce a user-supplied string into a value of some other type!) A typed interface could also automatically handle memoization of non-varying configuration variables, and provide methods to inspect whether and where parameters are defined, rather than merely a value (that might represent a value gap).
As I mentioned earlier, these issues are the subject of active design and development. Currently, some of our notes on our solution to these problems in Condor are in a ticket on the Condor wiki; ideally, an implementation will be part of a developer release of Condor in the next few months. Of course, I welcome feedback on these ideas and on the concerns that they are meant to address.
]]>
Like many of my colleagues on the MRG Grid team, I've been at Condor Week 2010 this week. Later today, I'll give a talk about some of the configuration management software we've been developing; for more details, you can visit getwallaby.com.
]]>I couldn't find one I liked, so I wrote my own. I'm pleased to announce the immediate availablity of capricious (as source or as a RubyGem). capricious is a framework to manage randomness for simulations and tests; it includes an interface for pseudorandom number generators, a family of probability distribution simulators (currently uniform, normal, exponential, Erlang, and Poisson, but it is easy to add more), and a class that can track aggregate information (e.g. min, max, mean and variance) about a stream of samples in constant space. capricious also includes two pseudorandom number generators: a simple linear-feedback shift register and one of George Marsaglia's multiply-with-carry generators. Each of the distribution simulators is parameterized on a generator class as well as on a seed and any distribution parameters, so it is easy to evaluate new generators for different applications.
I will write more about some of the interesting technical details behind capricious soon; in the meantime, I welcome feedback, contributions, or bug reports.
]]>The 0.1.x series and the 0.0.x series are not API-compatible, but the incompatibility is for a good reason: it enables SPQR to expose normal Ruby methods (that is, those that don't use keyword arguments). To see what I mean, compare the "Hello, world" example from v0.1.2 with its counterpart in v0.0.4. (The former is also embedded below, but you might have to click through to the post to see it if you're reading this in a syndication feed.)
Here are some other enhancements to the code since the last time I mentioned SPQR:
The full list is here. Enjoy, and please don't hesitate to write with questions.
]]>I'm pretty excited about SPQR, but this site won't become "all SPQR, all the time." Since interested hackers can easily follow the day-to-day project status on github or fedorahosted, I will reserve future SPQR-related blog posts for substantial announcements.
]]>First, let's look at a simple QMF schema for a class that exposes one method, echo, which returns its argument (note that the code examples may not show up if you're viewing this in a feed reader):
Running spqr-gen on this example produces two files: agent-app.rb and examples/codegen/EchoAgent.rb. As you can see, these two files contain all of the boilerplate we need to start implementing these QMF methods --- and, by chance, the boilerplate methods actually have the behavior that our agent is meant to!
spqr-gen is included in the SPQR repository; the system as documented in the last two posts is tagged "introducing-spqr".
]]>AMQP is an open standard for interprocess messaging --- and an excellent basis for distributed applications with even particularly demanding requirements. I'm most familiar with the Apache Qpid implementation of AMQP, since many of my colleagues at Red Hat are Qpid committers. Qpid is scalable, high-performance, and open-source --- it also features a variety of language bindings.
One great feature of Qpid that isn't in other AMQP implementations is the Qpid Management Framework. QMF is essentially a protocol so that applications can allow themselves to be managed remotely. In this way, it is similar to familiar remote procedure call mechanisms: a server application publishes an interface specification, and client programs are able to interact with that interface by invoking proxy methods that forward their arguments to the server machine and ferry return values back to the client. Unlike RPC, however, QMF is built upon Qpid messaging, and so offers low-latency, high throughput, and great resiliency.
In the QMF world, "server" applications --- those that can be managed --- are called agents, and "client" applications are called consoles. (Consoles and agents communicate by connecting to the same Qpid message broker.) The interface to an agent is called a schema; while schemas are instantiated programmatically in the applications that use them, there is also a well-defined format for recording schema information in XML.
In older versions of Qpid, it was only possible to develop QMF agents in C++; with more recent versions (including the one packaged in Fedora 12), it is possible to use Ruby as well. Since I am currently developing a rather substantial QMF agent application in Ruby, I wanted to have robust infrastructure that would allow me to avoid repeating myself, skip writing boilerplate as much as possible, and get on with the interesting work of my actual application. SPQR is built on top of the existing Ruby QMF engine and allows developers to publish instances of standard Ruby classes (with minimal and unobtrusive annotation) over QMF.
The first example we'll run through is (of course) a very simple application that publishes one method, which merely greets the user. Ours will be a little more interesting than classic "Hello, world!" programs, though, since we'll provide a personalized greeting (and publish our answer to another process). Here's what our application would look like in plain Ruby:
Note that we're using keyword-style arguments for the hello method, since we'd like to be able to support out-parameters (that is, not merely a single return value). Otherwise, this is pretty straightforward. Here's what it would look like if we used SPQR to publish Hello.hello over QMF:
The core of the class is basically the same (although we do collect statistics now, primarily to demonstrate SPQR's support for QMF statistics). The only differences are in the annotations we've added in order to enable SPQR to manage this class (viz., mixing in the SPQR::Manageable module), and to tell SPQR which methods to expose over QMF (and the types of their arguments). We've also added find and find_all methods to make it possible for QMF console applications to query for a particular Hello object (or for all of them). (Since the Hello class has no state, we simply make it a singleton class and the find methods just return the sole instance.)
The second example is a little more interesting: it shows the intersection of SPQR's capability to publish objects to QMF and my Rhubarb library's capability to store specially-declared Ruby classes in SQLite databases. (You'll need SQLite and its Ruby bindings installed to run this example.) This example is a simple logging agent that exposes two classes: LogService, which is a singleton class that can generate log records, and LogRecord, which is a class backed by a database table that models these records.
SPQR is still under development, and is only available via git for the moment. However, if you're interested in developing QMF agent applications in Ruby, it supports a useful subset of QMF and presents a clean, simple way to expose your applications over QMF. I welcome comments, bug reports, feature requests, and patches.
]]>Of course, it isn’t possible to instantiate a template based on a value that isn’t a compile-time constant, but it may be desirable to choose from one of several (statically-defined) values at run-time. In the std::bitset case, we might define “small,” “medium,” and “large” sizes for a representation class, and have the constructor for wrapper class choose the representation that is large enough to hold the set of bits it is created with. We’d like wrap bitset in such a way so that its size is not an explicit part of the type of the wrapper class (and thus clients can write code that handles wrapped-bitset objects of varying sizes without parameterizing their code). This is a problem, because the size of bitset is part of its type.
In general, this problem manifests whenever a wrapper class can make a run-time decision about which of some static set of template parameter values for its (hidden) representation class is most appropriate; or, still more generally, when it makes sense to choose a different type of representation based on a run-time value. Note that this problem isn't confined to classes parameterized on terms; a similar problem could also appear with classes parameterized on types, like STL containers. However, it is less immediately clear to me what a solution for classes parameterized on types would look like in the general case, since the type parameter is more likely to be exposed in the container's interface. (There are special cases, like choosing different types of containers as an internal representation depending on the characteristics of different collections of homogeneous data, that might make sense.)
There are a few approaches to wrapping a class in C++, but none of them are particularly suitable for solving this problem. Here’s why:
A general solution to this problem at the type-system level might involve constrained template parameters and union types. So you could say, e.g., std::bitset<N in {32,128,2048}> b = std::bitset<32>(); This solves our problem by encoding a range of possible sizes in the type of a bitset-valued variable rather than encoding a particular size. (My gut feeling — although I don’t have the turnstiles to prove it — is that this feature might be difficult or impossible to include in a sound, statically-checked type system, let alone to implement in a production compiler.) Sadly, extending the language is rarely a viable option for most software projects.
There are, of course, ways to solve the problem of needing a set of bits large enough to hold k bits, where k isn't known until runtime: use std::bit_vector or use std::vector<std::bitset<32> >, to give two simple examples. More generally, it would also be possible to use virtual functions in a base representation class that is extended by a templated class: then derived<32> and derived<128> could both be pointed to by a representation*. But I wonder, since I’m continually amazed by feats of template metaprogramming, what clean and efficient solutions to the general problem are possible?
]]>I was able to image and convert a 100 gb drive in around six hours. My drive was an LVM volume and the home partition was encrypted with LUKS; I was delighted to see that qemu-img handled these oddball features of my drive flawlessly. (I can't think of a technical reason why these wouldn't be supported, but I'm nonetheless inclined to be pleasantly surprised when things work as they should out of the box.)
]]>