How the #GamesNightViz Starter Kit went x10

Quick quiz. How long does it take to beat Final Fantasy Tactics?

If you just opened HowLongToBeat in another tab, congratulations - you've found the exact problem this whole project exists to solve. Okay, next question: what did critics think of it? That's a different website (Metacritic). Is the remaster on Steam? Different website again. When did it FIRST release, and on what? Wikipedia, probably. Four tabs, four websites, four slightly different answers depending on who you ask... and that's ONE game.

Now imagine you're trying to build a data visualization comparing a few hundred of them.

That's the wall everyone hits when they try to viz video games. And it's the reason the #GamesNightViz Starter Kit exists - one downloadable dataset where 124,000 games already come with their release dates, platforms, genres, critic scores, playtimes, and Steam reviews stitched together. You shouldn't need to understand the plumbing of the games industry just to make a fun viz about the games you love. You should be able to press start and play.

This post is the story of how that dataset got built. It took three attempts, one painful restart, and a LOT of lessons about matching messy data. Grab a snack, this is a long one.

Why doesn't this dataset already exist?

Here's the thing most people outside the industry don't realize: there is no single place that knows about every video game. Each aggregator grew up serving a different crowd, so each one sees a different slice of the picture.

So if you only pull from one source, you inherit that source's blind spots. And the overlap between them is much smaller than you'd expect. Out of the 123,979 games in the finished masterlist, only 62,159 appear in all three of IGDB, Steam, and Metacritic. Meanwhile 6,398 games exist in exactly ONE source. Remove any single aggregator and those games vanish from history entirely.

Venn diagram showing the overlap between IGDB, Steam and Metacritic coverage

No single source sees the whole industry. They each see their own neighborhood. Which means someone has to stitch the neighborhoods together into a map... and that apparently, was me. 😅

The grind years: one Excel sheet, several months, 15,000 games

The first version of this was in 2024, and was built the honest, brutal way. I used browse.ai (a tool that lets you train little robot assistants by clicking around a webpage → "see this table? grab it, then go to the next page and do it again") to scrape game listings, then hand-merged everything in Excel. Copy, paste, eyeball, dedupe, fix, repeat. When two sites disagreed about a release date, I was the tiebreaker. When the same game appeared as "FFVII" on one site and "Final Fantasy VII" on another, I was the matching algorithm.

A few months of that produced about 15,000 games. And look, I'm genuinely proud of that sheet. It powered real #GamesNightViz entries and every row in it had been touched by human hands. But the ceiling was obvious. Fifteen thousand games sounds like a lot until you learn IGDB alone lists over 370,000. Worse, the scraping had leaned heavily on console-focused sources, so PC gaming (you know, only the biggest platform on earth) was a giant blind spot. Every month, new games released faster than I could manually add them.

The manual approach didn't scale. Time to bring in help.

Attempt #1: modding the old save file

When AI-assisted coding tools got good enough, the obvious first idea was the conservative one: keep the trusty 15k Excel sheet as the foundation, and bolt new data onto it from four sources - IGDB, Metacritic, Wikipedia, and Steam.

Seems pretty reasonable, right? Don't throw away months of manual work. Just... extend it.

Except the matching logic we built around it had a fatal flaw. To confidently link a legacy row to its counterparts, the pipeline effectively wanted agreement from every source - the game had to show up in IGDB, AND Metacritic, AND Wikipedia, AND Steam before it got the full treatment. Each individual check felt sensible ("more corroboration = more confidence!"). Stacked together, they formed a door that almost nothing could walk through.

Think about what that rule actually demands. A beloved Japanese console RPG from 1997 that never came to PC? No Steam page → blocked. A small indie darling with 50,000 glowing Steam reviews that critics never bothered scoring? No Metacritic page → blocked. We already know the sources each see different neighborhoods, so demanding a game exist in ALL the neighborhoods at once is basically a filter for "famous recent multi-platform hit."

Here's the number that killed the approach: of the 124k games in today's list, only 57,289 exist in all four of those sources. The all-gates-must-pass rule would have thrown away more than HALF the catalog. Games were falling through the cracks not because they were obscure, but because our logic punished them for the sources' blind spots.

Two Venn diagrams comparing approach 1 (only the tiny overlap of all four sources gets in) with approach 2 (any credible source gets you in)

The embarrassing part? We got a long way down this road before admitting it was the wrong road. We'd assigned permanent ID numbers to 15,355 legacy-derived games. Exported the workbook. The sunk cost was real and it was whispering "just patch it, you've come so far."

You know that saying about killing your babies? This was that moment. 😅

Attempt #2: New Game+

In gaming, New Game+ means restarting from scratch… but carrying your best gear from the previous run. That's exactly what we did.

The pivot decision: stop treating the legacy Excel as the foundation. Throw away its rows, throw away those 15,355 freshly-minted IDs (ouch), and rebuild the entire masterlist from the sources themselves. Flip the matching logic from "a game must appear everywhere" to "a game gets in if ANY credible source vouches for it." AND became OR. The tiny intersection became the big union.

But New Game+ means you keep the good loot, and the legacy sheet's genuinely irreplaceable treasure wasn't its rows - it was the roughly 19,000 links a human (hi 👋) had personally verified over those months of grinding. "THIS IGDB entry is the same game as THAT Metacritic page." That knowledge got salvaged into the new system as trusted seeds, so none of the manual labor actually died. The effort from months of Excel grinding are still in there, just invisible.

Then we rebuilt. Here's how, in plain terms.

Step 1: Pull everything. No, really… everything.

Instead of scraping bits and pieces, the new pipeline downloads each source's ENTIRE catalog through their official (or in Metacritic's case, unofficial-but-stable) data feeds: 369,995 games from IGDB, 195,477 apps from Steam, 162,830 titles from Metacritic, 150,926 entries from HowLongToBeat, 135,134 from Wikipedia's structured-data sibling Wikidata. Over a million records all up.

Two rules made this ethical.

First, politeness: every source has a speed limit, and the pipeline knocks a few times per second instead of hammering the door - nobody wants to be the reason a free community site adds a captcha.

Second, resumability: every record remembers when it was fetched, so if a download died overnight at record 80,000 (and it did, more than once), rerunning picks up at 80,001 instead of starting over. For the big jobs we ran several downloaders side by side, like opening more checkout lanes at the supermarket with one careful librarian doing all the actual filing, so nothing gets written over anything else.

And one more house rule that saved us over and over: each source's data lands in its own separate tray. Steam's copy of a game never overwrites IGDB's copy. Nothing mixes until the matching step - so when sources disagree (constantly), we still have everyone's original testimony.

Step 2: Quality gates → who deserves a row?

Confession: 370,000 IGDB entries does NOT mean 370,000 real, vizzable games. There are duplicates, bundles, cosmetic DLC packs, "games" that are actually soundtracks... letting everything in would drown the good stuff in noise.

So every candidate faces a set of quality gates, and passing ANY ONE of them earns a row:

One voucher from anywhere credible and you're in. That's the OR-logic fix, baked right into the front door. Of 334,085 IGDB candidates, 120,647 passed at least one gate. Then a separate rescue sweep handled games IGDB never listed at all - a game with a Metacritic page but no IGDB entry still deserves to exist, which added another 6,728 rows from the other sources' leftovers.

The pipeline from five full catalogs through trays, gates and matching to the final masterlist

Step 3: The boss fight → matching

Now the hard part. The same game lives in five trays under five slightly different names, and something has to figure out that "Wizardry 8", "WIZARDRY 8™" and "Wizardry VIII" are one game while "Portal" and "Portal 2" are definitely not.

This is called fuzzy matching, and the plain-English version is: teach the computer to score how similar two game records look, the way you'd eyeball it. Every potential pair gets a scorecard. The title does most of the talking (60% of the score, after cleaning both titles up: lowercase everything, strip the ™ symbols, turn roman numerals into numbers, drop suffixes like "Deluxe Edition"). Release year adds 20%, but with a one-year grace window, because sources genuinely disagree on dates when a game launched in Japan first or hit consoles a year before PC. Platforms and developer chip in the last 20% as sanity checks.

Then the rules of engagement: two records only get married when they're a 92-out-of-100 lookalike AND the best candidate clearly beats the runner-up. Obvious lookalikes below 60 get auto-rejected. The genuinely ambiguous middle goes into a spreadsheet for a human (a.ka. me) to eyeball. Sound familiar? It's the exact job I used to do manually in Excel, except now I only see the hard cases instead of ALL the cases.

We still got burned, of course. My favorite bug: the first version of the matcher worked first-come-first-served, so whichever record claimed a match first KEPT it forever, even when the rightful owner showed up later with a better claim. We caught it when Half-Life: Alyx (the actual game) lost its own identity to Half-Life: Alyx - Final Hours (a documentary ABOUT the game) purely because the documentary got processed first. The fix: collect all the claims, then hand each match to the strongest claimant instead of the fastest one. That documentary is now a permanent regression test and if it ever steals the game's seat again, alarms go off.

The messy middle (or: everything else that went wrong)

A blog post that only shows the wins would be lying to you, so here's the blooper reel:

That last part matters more than it sounds: every human ruling gets recorded in a permanent decisions table that outranks the automation forever. The pipeline can rebuild itself a hundred times; it will never silently undo something I decided.

Final tally after all gates, matches, merges, and cleanups: 127,647 candidates walked in, 123,979 earned a row (each with a permanent GNV ID, numbered in release date order. Quite fitting that ID #1 is the oldest game in the list), 1,627 turned out to be editions of games already present and got folded into them, and 2,041 were shown the door.

Oh, and the scoreboard on the whole journey?

Timeline comparing months of manual work for 15k games against days of AI-assisted rebuilding for 124k

A few months of manual grinding: 15,000 games. Six days of AI-assisted rebuilding: 123,979. That's not a knock on the manual era… every one of those months taught me what "good" looks like, and the AI-assisted run only worked because a human kept steering (and hand-reviewing, and catching documentaries impersonating games). But man, the difference still shocks me.

The completionist run: backfilling and enrichment

Getting 124k rows was the main quest. Then came the part every completionist knows: going back for everything you missed.

Auditing the finished dataset surfaced an uncomfortable-looking number right away: only 17% of games had a critic score. Seventeen! My first instinct was that the pipeline was broken. So we checked and went through the unscored games source by source, looking for score data we'd failed to collect.

There wasn't any. The 17% is the truth. Around 88% of IGDB-listed games have literally zero critic reviews there, and most Metacritic-listed games have no critic reviews either. Critics review the famous stuff; the long tail of gaming history just never got scored by anyone. And honestly? That's not a data gap, that's a INSIGHT - my favorite finding hiding in this dataset is the invisible mountain of games nobody ever reviewed.

Bar chart of source coverage across the final masterlist, with the critic score callout

The genuinely hard backfill was playtime. HowLongToBeat has no official data feed at all, and politely crawling 124,000 individual pages would have taken weeks. So we layered it, cheapest first: start with the hand-verified HLTB links salvaged from the legacy sheet (thank you, past me), then match against a bulk snapshot of HLTB's catalog that a community member (Thanks Zaire Ali for saving my sanity) had already published on Kaggle.

So we used that as a baseline (since a local copy is free and instant) and only use live searches for whatever's left. Near-perfect title matches got accepted even when the release year disagreed (a 95%+ title twin with a mismatched year is almost always the same game with sloppy date metadata, not a different game). The ambiguous middle went to a review spreadsheet for human eyes. Result: playtime data for 70,248 games —> 57% coverage on a site with no API, without hammering anyone's servers.

Smaller side quests, quickly: about 23,000 Steam games had a review total but no positive/negative split (they'd been fetched by an older version of the collector, before it captured the split) —> one evening with eight parallel workers re-fetched all 23,380 rows with zero errors. And a batch of Wikidata entries that had no release date when we first asked were quietly re-asked, because "no date last month" doesn't mean "no date today."

What's next?

The masterlist now refreshes on a planned yearly cycle. That's the beauty of attempt #2 over the Excel era: the pipeline is a machine, not a document. Rerunning it pulls whatever's new, matches it against what exists, and appends existing GNV IDs never change, so your vizzes won't break when the data updates.

Now, would more frequent refreshes be better? Maybe! Should review counts update quarterly while the catalog updates yearly? Also maybe! This cadence is a starting point, not a hard lock… so if you have opinions, I genuinely want them.

TLDR (Too Long Don't Read)

So there you have it - the whole messy, restart-riddled journey from a hand-rolled Excel sheet to a 124k-game masterlist you can just download and viz.

Go make something with it: grab the data via the #GamesNightViz site, poke at the starter dashboard on Tableau Public, and viz what you love. Or if you’d would like to take a peek under the hood - I have the full codebase in Github.

And if you spot a missing game, a wrong date, or a smarter way to do any of this - buzz me. 😉