Chapeau - Best practices for Wallaby’s default group

Recall that Wallaby applies partial configurations to groups of nodes. Groups can be either explicit — that is, a named subset of nodes created by the user, or special groups that are built-in to Wallaby; each node’s group memberships have a priority ordering, so that an individual node’s configuration can favor the partial configuration from one group over another. There are two kinds of special groups: the default group, which contains every node and a set of identity groups, each of which only contains a single node. In addition, Wallaby includes a skeleton group, which combines aspects of explicit and special groups: while it can be managed like an explicit group, all newly-created nodes become members of the skeleton group automatically. The default group is always the lowest-priority membership for a node and its identity group is always the highest-priority membership; a node’s skeleton group membership can be reprioritized or removed as necessary.

The default group is an appealing target for common pool configuration, since it is guaranteed to be applied to every node. However, because Wallaby’s configuration model is additive, it is likely not the best place for every configured feature or parameter setting that you might initially consider applying to the whole pool. For example, if a group’s partial configuration installs a feature, every node that is a member of that group will install that feature (there is no way to say “this group’s configuration removes feature F, if it happens to be installed already). Similarly, if a parameter is set on a group, every node that is a member of that group will have that parameter set; individual node configurations can override the value that the parameter is set to within the group, but there is no way to unset the parameter altogether. Therefore, if you need to enable a feature on almost every node, the default group is not the right place to install that feature. (Indeed, the default group is not the right place to put a putatively universal parameter setting or feature installation even if you can imagine a future exception to its universality.) The default group is also not a great place to put configuration that you expect to take priority over other possible group configurations, since it will always be at the lowest priority.

At this point, you may be asking yourself why you’d want to put any configuration in the default group. While I tend towards a minimal default group configuration myself, I absolutely see several use cases for the default group:

Setting configuration parameters that are actually uniform across the whole pool (and will require a new value, not the absence of a value, if they change). One such parameter is FILESYSTEM_DOMAIN, which you can set to an arbitrary string in order to identify machines that access the same shared filesystem. If you were to extend your pool with machines that couldn’t access that same filesystem, you’d provide a new value for this parameter.
Installing Wallaby features that you actually want installed on every node. I’d include features like Master, which ensures that the condor_master daemon is running (since Wallaby’s configuration daemon runs under the condor_master; a node can’t be configured if it isn’t running) and NodeAccess, which controls access to the pool (although the specific policy parameters required by NodeAccess may change in pool subsets).
Rapidly prototyping configurations for homogeneous or small, experimental pools. When you’re first using Wallaby, the default group is a convenient way to get things running. Similarly, if most of your nodes are execute nodes with the same policies, you may be able to put most of your configuration in the default group, especially if your submit and central manager configurations are generally a superset of your execute node configuration. Fortunately, Wallaby makes it straightforward to move configuration from the default group to an explicit group if you should require more flexibility.

I’m sure there are other cases in which using the default group makes good sense, but in general, you should strongly consider using an explicit group or the skeleton group for almost all of your nearly-universal configuration. If you haven’t used the skelton group before, read more about it!