Practical Testing

I was recently reading AEG’s L5R RPG forums, when I was reminded of a method for testing for value. I might as well get talk of CCGs out of the way for those less interested in applying the methodology to RPGs and whatnot.


A bit of wisdom that is passed around in the V:TES community is to look at what cards you discarded over the course of the game and remove or, at least, consider removing them from the deck. While I don’t think that many players actually use this – I don’t in any sort of rigorous sense, it is a relatively easy way to weed out (likely) suboptimal cards. The closest I come to applying this process is keeping in mind that virtually every Crocodile’s Tongue and almost every Diversion I put into decks ends up getting discarded, which is why I never play the former anymore and play the latter more as a joke. For instance, the latter shows up in my Striga deck, and we all know that Striga is so broken that even joke cards “work”.

So, great, a way to tune decks. But, then, I rarely tune decks, so I’m more interested in other applications of the same sort of thinking. What is the core concept? That what people actually play has value and what people don’t is lacking. Particularly profound? Not so much. Underutilized as a system for balancing games? Way underutilized.

Balancing games? … moving on to playtesting. There were all sorts of reasons to be frustrated by playtesting niche CCGs. Rather than rant about yesteryear, I’m focusing on a particular aspect of playtesting that should be more worriesome to game managers (if anything about playtesting is). The usual situation, in my experience, is that a small amount of a new set gets an inordinate amount of attention and much of the set gets no meaningful testing. Let’s think about the logic of this. Certain cards or groups of cards attract for various reasons, but the primary reason is power level. Sure, I tried repeatedly to get Walker Smith printed for B5, and I playtested Gerontocracy so much more than anyone else that I’m credited as its designer, but taking out individual interests, groupthink tends to lead to focusing on power cards. So, logically, cards that are ignored are, on average, below the power curve.

In my ideal playtest world, the playtest manager would force everyone to provide a list of all of the cards they didn’t want to test. Why not just force people to actually play every card? Because, really, playtesting is painful enough without the mindnumbing process of trying to generate interest in chaff, nevermind that the interest level will be so low that any testing is questionable, anyway. So, you get the lists. Because you allotted enough time and resources, because you are terribly clever, or just because you’ve gotten fed up with junk playtesting far too often, you start the playtest with the bottom of the set. Well, after you test any new mechanics or other stuff you expect to undergo major overhauls. Not that you actually play-test the crap. You brainstorm reasons why people aren’t interested in it. Throughout the playtest, you repeat the process so that there’s no cards left behind. In theory, every card will rise to the level of interesting enough to people that they do get tested. No, this doesn’t provide some sort of balance of power, since it is well-established that different players look for different qualities in cards, but it should reduce the variance in attractiveness of cards.


Whereas the idea of tracking what gets played and what doesn’t is underutilized in my experience with CCGs and RPGs, my friend who runs lots of boardgame events is a proponent of using this methodology with Dominion.

What we do, since Dominion plays fast, is take a look at what people buy and remove those cards for the next game, repeating until we stop playing. Now, I don’t like Dominion. I actually believe it’s insanely overrated. I am of these opinions because I find that Dominion lacks variety, i.e. within any given game, people mostly buy the same cards or, at least, the people who have any chance of winning buy the same cards. While it seems like the process we use to weed out obvious strategies has some effect, each individual game still sees rather little strategic variety. Not finding the variety between games all that compelling … at least it’s quick – tellingly, a common justification for why people should continue to play Magic even though so many of its games suck.


I consider there to be three classes in d20 Conan: barbarian; scholar, thief. I keep mentioning this even though it’s a waste of time to try to convince non-analytical people of anything. Borderer? Barbarian strictly better. Noble? Mechanically crippled. Nomad? Pretty much a borderer. Pirate? Not bad, but easily replaced by barbarian/thief. Soldier? The worst class yet the hardest to convince people that it sucks – skills matter in Conan (so do saves). Temptress? Just a scholar/thief or thief, depending upon what you care about.

Obviously, if I’m right, then the game is flawed. If the game is known to be flawed, it shouldn’t be printed (in some imaginary world where balance matters more than anything else), or if not caught until too late, then it should be repaired. RPGs are trickier than CCGs, of course, as they aren’t competitive and speak to different interests. Actually, one wonders whether the psychographic profile system of Timmy, Johnny, and Spike can be applied to RPGs. Given that I’m somewhat more of a Spike when it comes to CCGs and I’m a big powerlessgamer in RPGs, probably not. Anyway, while little can be done on the publisher’s end for Conan (don’t see a problem, not publishing the game anymore!) and everything can be done on our playgroup’s end (a feature of RPGs not shared with CCGs), I still find the exercise of determining balance, both in terms of mechanical balance and in terms of interest balance, based on what people decide to play and what they don’t a compelling one. Given that the playerbase, if accurately represented on the Conan forums, for this particular game is so out of step with my analysis, maybe my analysis is just off. But, probably not.

To me, Conan made no attempt at balance. Based on comments I see, such as “barbarian should be more powerful”, imbalance could have been entirely intentional. But, what of L5R 4e, where an obsession with balance seemed to take hold. I can see little value in trying to chart clans and families, after all, the differences mechanically are mostly trivial except for how you combine with schools. In particular, there’s no benefit to rating clans when there are hardly any mechanical benefits at the clan level, nevermind that people strongly attach to clans and the like in RPGs for flavor reasons. It’s really about schools or the combination of schools with families for doublestacking traits or avoiding getting saddled with a trait bonus that one doesn’t care about. It could be argued that Crane and Scorpion having multiple doublestack possibilities makes those clans stronger, but anyway, back to spotlighting schools.

Which schools will people eschew? Is it for power reasons? Of Scorpion schools, I tolerate Bayushi Bushi and none other for flavor reasons. Likely, many others have similar personal interests that are hard to account for. But, maybe the methodology can be tested. For instance, it might be quite interesting to see how many people play Mirumoto Bushi in 4e vs. a hoped for apples to apples comparison with 3e. Mirumoto Bushi in 3e is broken. In 4e, I only see a subpar and dull school. This is quite unlike how Akodo Bushi is quality in 3e and, arguably, the best bushi school in 4e. My sense, which is quite specious at this early date, is that few are interested in Mirumoto for HoR3 where I commonly adventured with Mirumoto in HoR2.

Another observation is that few players play Agasha Shugenja and many play Isawa Shugenja. This may be an intentional imbalance as the world calls for far more Isawa than Agasha. But, what of Asahina Shugenja? They were grossly powerful in 3e, I believe nerfed in 3r, and I don’t know what in 4e. I like the school, but I like defensive powers. The meaningful comparison, of course, being between Asahina and other shugenja schools. What of people’s inclinations when doing Different School? I see Daidoji Iron Warriors being an attractive option, especially if not terribly concerned with doublestacking. I don’t see Yoritomo Bushi being an attractive Different School (or school to begin with).

What of the Imperial schools? The minor clan schools? With cool RPGs, everything gets adherents based on flavor. So, there’s always going to be the Boars of the world. But, check out the ridiculously synergistic Badger/Badger build.

What won’t a meaningful number of people play? If something, that something was a waste of ink better used for coming up with something people will play. Of course, this comes from someone who thinks about Omoidasu builds, so whatever.

3 Responses to Practical Testing

  1. Andy says:

    I’ve got to admit that some of the design choices for schools are rather baffling. It seems like any Bushi school that doesn’t get its extra attack by the third rank is at a huge disadvantage compared to the others.
    I don’t know why they don’t just standardize that.
    I can see Akodo as being pretty powerful but I’m not sure why Yoritomo would be such a bad choice, +1 Strength isn’t particularly inspiring, but the school’s abilities seem at least interesting.

    Any idea where the rules are for off-hand weapons? The listing in the index has the wrong page number in it.

  2. iclee says:

    Lot of folks wanted to eliminate multiple attacks all together. I don’t think that ranking up in bushi schools is all that important if you want to do cool stuff (generally just rank 5 when all hell breaks loose), but if you want to kill, multiple attacks is too mighty.

    They don’t standardize because that loses flavor. Some bushi schools are supposed to be better at fighting. Bayushi Bushi is supposed to be more like a multiclass fighter/thief.

    Kama aren’t that great and get put into Knives skill oddly when they were (and are) peasant weapons. Yoritomo was kind of sucky in 3e, too, though it was cooler and there were good kata. Very glass cannon in style, with no defensive benefits and huge damage benefits. Not having the book in front of me, my recollection with 4e is that it’s mostly that the school doesn’t really do anything special. Maybe 4e really is about things like Intimidation and Temptation (social) and disarm/knockdown/feint (combat). If so, that’s kind of dull, but it does mean some of the abilities I scoff at aren’t as lame as they sound.

    Rules for offhand weapons exist. They are hard to find, being in some sidebar in some place counterintuitive. Try looking in the equipment section. I knew I was surprised to read in the forums that there was an inherent benefit to offhand weapons, so I hunted around for it. It’s incredibly lame, war fans are lame, Mirumoto tech is lame. Yes, I realize small gains in Armor TN are much more important than increases in TNtbH in 3e because the attack rolls are so much lower, but anyone should be gaining more than +5 per insight rank on attack rolls, easily counteracting two-hand benefits.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: