General Data: The Gathering and visualization - How can we do it?

[VincePendrell]

Since I'm posting in another thread today, I might as well start this possibly more important discussion here. As with all my posts, this will be fairly long, so brace yourselves! The important part is in bold, the rest is fluff.

When I first joined the Riptide Lab, one of the things that I was most interested in was comparing experiences and figuring out the coolest ways to improve the fun™ in my Cube, and looking for good ways to analyze and visualize data. After a 2 year hiatus, completing my masters and starting my career as a game developer, I'm back at cubing. But while CubeTutor became the de-facto service to store your Cube list, I am still using a Google spreadsheet for all my Cube needs. Not just because I feel CubeTutor is missing some important features, but also the feature request list and the needs of the average user/contributor seems to be very different from my own needs.

I need a better way to aggregate useful data about my cube to make informed decisions on what archetypes needs more support, how well my curve is balanced for different archetypes, how high of a pick a card is, how much a card contributes to an archetype win% (Ideally I would control for player skill as well, but I don't think it is feasible). Basically, I want better ways to distant myself emotionally from any card and figure out if they are or not impacting my format in a negative or positive way (I'm looking at you, foil signed Wurmcoil Engine!).

So, the greatest question is: How do YOU store and analyse data on your cube performance to make informed decisions? I feel like no one will be as big of a balance freak as I am, but I'm realy interested in getting some opinions and suggestions.

For collecting metrics for picks and deckbuilding, I've thought about these solutions:
• Taking pictures of picks and decks, entering the data manually somewhere.
• Asking each player to write/type their picks and decklists, then entering the data somewhere.
• Use a digital solution for drafting and deckbuilding, then gather data from it.
→ Ideally, collecting if a card is played or not in a given game would be nice, but beyond unpractical.

For collecting results and archetypes, you probably need the lists already, but you could:
• Just note down the archetypes and ther win-loss-tie (I did this before my hiatus)
• Take the win-loss-tie data and type together with the decklist entry.
→ In an ideal world, the coolest solution would be to just throw all the decklists somewhere and analyse data clusters to see which archetypes actually exist in your cube and design around that idea, but this is also beyond unpractical and would demand way too much data.

Right now, I have a spreadsheet tuned to handle a lot of calculations that I might want to do during cube construction (you can consult it here), but I haven't figured out how to gather, store, and visualize pick-orders, decklists and results. I've been thinking of maybe just taking pictures of card pools post-draft, taking pictures of decks, then possibly using Access or Google Spreadsheets to register decks and drafts. I've been using Periscope a lot for data visualization at work, so if I decide to go crazy with this idea, I'd probably get a server and setup a database there to play around with it, but I'm on the lookout for alternatives. So, what is your suggestion for collecting and visualizing data about your Cube?

 

[ColonelSanders]

For direct management, I have my entire cube in magic set editor, which makes it easy to sort/filter through it. However, my cube is almost completely custom cards, so this was a natural starting point. If you have a normal cube, moving it all into set editor could be a very time consuming process. Similarly, I have all decklists that have been played because of cockatrice (my playgroup has all graduated from college and is spread across the country). That being said, I haven't put any real effort into going through the decklists and keeping statistics on what is and isn't getting played.

In fact, my approach to balance is almost the opposite of that. My playgroup is relatively fixed and we all have a hand in making the cube, so after each round, I try to get discussion on how people felt specific cards/archtypes did and take notes. Additionally, when we play, there's often an odd number of people, so one person spectates through cockatrice. Since we use a single skype call, this definitely starts discussion on balance and viability of strategies, not just plays that are happening on a specific turn.

I would argue that fun, balance, and even playability of cards/flat power level are not directly correlated Metagaming can normally compensate for imbalance in the short run, and in my experience, is easy to spot (either people complain, or deckbuilding choices are suddenly shifted). There is definitely use in tracking wins/losses and how often archetypes appear, but I think there's a lot of value in just getting qualitative feedback from the people you play with regularly. Part of cubing is making sure they're having fun, not just seeing your environment produce the intended decks/archetypes. That being said, a lot of this relies on having a regular group of reasonably skilled players who can moderate their biases.

 

[VincePendrell]

I would argue that fun, balance, and even playability of cards/flat power level are not directly correlated Metagaming can normally compensate for imbalance in the short run, and in my experience, is easy to spot (either people complain, or deckbuilding choices are suddenly shifted). There is definitely use in tracking wins/losses and how often archetypes appear, but I think there's a lot of value in just getting qualitative feedback from the people you play with regularly. Part of cubing is making sure they're having fun, not just seeing your environment produce the intended decks/archetypes. That being said, a lot of this relies on having a regular group of reasonably skilled players who can moderate their biases.

I agree with you a lot. I've been keeping a test pool for my cube since inception, then I'm always talking to my players about what they think about cards, seeing how they play it, then comparing it to how it was played and etc. I remember spending a good amount of time talking to my players about The Abyss and Moat in particular, and I'm always on the lookout to see what players are trying to build to see if it is possible to make that more viable than your usual Esper Control. Usually this comes from a player saying "I don't think we can manage an Obzedat, Ghost Council", or from seeing someone forcing UB aggro or BG stax and failing. To me, qualitative data is both easy and cheap to gather, so I'm not on the market for solutions to that.

Also, becoming a game developer has taught me that the word fun is extremely useless and doesn't mean anything, so I'm very cautious whenever it is used in a non-ambiguous way. I mean, it is really fun for a lot of people to slam a Grave Titan into play and win the game, but the usual consensus is that the card is harmful for the cube environment as it limits card choices and makes picks automatic. I'm more interested in getting more precise data on the other cards that we don't talk about, and experiment with the thresholds of playable/unplayable of different cards and archetypes.

 

[Eric Chan]

Hooooooly crap, your Google sheet of your cube list makes mine look like baby crayon drawings on a wall. You're like the Nate Silver of cube analysis!

 

[ColonelSanders]

Also, becoming a game developer has taught me that the word fun is extremely useless and doesn't mean anything, so I'm very cautious whenever it is used in a non-ambiguous way. I mean, it is really fun for a lot of people to slam a Grave Titan into play and win the game, but the usual consensus is that the card is harmful for the cube environment as it limits card choices and makes picks automatic. I'm more interested in getting more precise data on the other cards that we don't talk about, and experiment with the thresholds of playable/unplayable of different cards and archetypes.

I completely agree that getting a consensus on what is fun from a large pool of players is effectively impossible. However, one of the advantages of cube is that your player pool is tiny in comparison to what wizards has to design for. Or maybe this is a perspective thing,; I only have 3 other regular players, so I can pretty effectively characterize what each of them finds fun in a game of magic and balance accordingly.

Anyhow, discussions of what fun is tend to chase themselves in circles. Your spreadsheet has pretty much everything magic set editor has for analysis built in, so I don't have any other great methods for visualizing cube data to suggest. I like the idea of recording frequency with which color combos and archetypes appear + their w/l ratio, but if you want to move beyond that to a card by card level of frequency, you will need a larger sample size than a small playgroup can feasibly produce for significant results. And even then, a list that ranks card play frequency may be a crude way of describing what's going on in your cube.

 

[VincePendrell]

I agree, and even though I think most of this data gathering is overkill, my OCD needs this I'll take a better look at the MagicSetEditor tools as well, to see what I could use there as well.

Anyhow, discussions of what fun is tend to chase themselves in circles.

I like using this as a starting guide to avoid that issue
http://www.jesperjuul.net/ludologist/the-words-of-game-design-the-terminology-of-ion-storm

Hooooooly crap, your Google sheet of your cube list makes mine look like baby crayon drawings on a wall. You're like the Nate Silver of cube analysis!

Thanks! I had to google who this guy is haha
I still can't elegantly view the curve for a deck that would use X amount of colors to make sure that my Guild cards are curving properly with the rest, so I still have a lot of work to do

 

[Grillo_Parlante]

There are a few signposts for fun that I tend to look for as regards magic in actual gameplay:

1. The ability to play towards, or build, a strategy
2. The sensation of doing something fun
3. The careful balancing of risk and reward

The fun question is hard, but important, since after you gather the data you have to have some methodology in place to analyze it.

As for data collection sources, I use the analytics from cube tutor (because I am too lazy to do something like vince has done), and than track decks, mostly via cube tutor drafts (it was really helpful to win that featured cube). I also found it helpful posting in the pack thread in forum games, as that gives you perspective into what people are actually thinking when they are assessing card and color strength.

I also really like some of the suggestions in this under the hood article.

This exert is something important to keep in mind, I think, when looking to make data backed decisions:

The preferences of your players and the exact packs they’ve opened do just as much to shape an individual draft as your environment does. Broken decks can lie dormant for months because nobody wanted them; people can fight over archetypes that turn out to be terrible or shallow but the lack of depth is masked by the fact that one colour was trying to support six drafters at once.

 

[Onderzeeboot]

I doubt you're not already doing this if you're a stats lover, but I often ask my players to write down their seat, their first and their last picks, whether those cards end up in their deck, and what deck type and deck colors they ended up with. This only works if you diligently remind players to write down that information (pre-printed registration slips help), but it's pretty nice information to gather, and relatively easy on your players. It's only six picks they have to write down, and if they stack their cards correctly and don't shuffle their picks halfway through the draft, they can even write everything down after the draft.

 

[Tyrole]

Great thread! Vince, like you, I enjoy google spreadsheets to work on the science end of the art/science that is cube design. Cubetutor is great, but there is so much more I want to do, and it didn't exist until after I had built my cube workbook.

Quick summary:

• I manage my mtg inventory in deckbox; as such, I have a series of formulas to extract card properties (casting cost, P/T, etc.) from deckbox csv exports for use in my cube google doc.
• I have played with card categories, but the work Vince has done blows mine away completely. Amazing job!
• There are still a number of manual things I do when updating my cube, due to a combination of laziness and parsing weird mtg info (e.g., phyrexian mana)
• Much of my effort is inspired by the (now ended?) PureMTGO articles by Matthew Watkins (http://puremtgo.com/articles/recent?uid= oraymw)
• My spreadsheet is a pretty big mess now, with a lot of to-dos. A number of my calculations are extremely inelegant due to taking the fastest path to make things work on a number of occasions. I would probably be best served to start from scratch, play with defining cell ranges, and use SQL to query them instead of excel formulas.

While I believe the "feel" of a cube is important, having statistics to inform decisions and ensure certain levels of balance is extremely useful. We talk a lot about the art of cube design on these forums, but I think there is value in statistical analysis of set designs [side bar: I wish wotc would release more of this information from their designs. It's not interesting to most people, but to some of us...].

To get back to Vince's original question of how we can measure the use and value of cards from drafting and playing: I really don't know. Even capturing the data is a ton of work, and once we have it, are we likely to have a statistically significant amount? I am lucky to get in 1 draft per month, so I know there's no way it would matter for me. There would be an occasionally useful insight, but probably nothing beyond what I am able to notice by basic questioning of my drafters and informal reviews of their decks.


I plan to post in more detail when I get around to starting my cube thread in the other forum, but here are some analysis pieces I have going:

 

[VincePendrell]

I've been absent from the thread for a long time, but I haven't cubed for a while. I guess that's why once I've been chosen as the user most likely to show up dead in a ditch somewhere

First things first: This video has been in my To-Do list for a couple of months now, but I guess there is some interesting stuff here about what I'd love to try and do with my cube:

It's a talk from the Game Developers Conference and the speaker discusses metagames for competitive games. What I love the most is that he makes a tool available for us to play with. Making matchup charts is so ewhat doable even in our average cube environment, I believe, as long as we keep our lists consistent until we get enough data for a rebalance (probably means skipping making changes for a couple of sets). Either way, I hope it gives everyone some ideas

@Tyrole: I had actually typed a massive answer at some point, but my laptop had a major crash and I lost it, but cool stuff you got there Keeping tabs on P/T sounds like something that I should do. Reminds me of how I usually see Flaying Tendrils as the last 3 cards in a pack, since everything in OGW and BFZ seems to be x/3.