Show HN: A nom parser for the Starcraft 2 Protocol Replay format (github.com)
Been having a lot of fun reading an SC2Replay collection through nom parsers, serializing into Arrow files so that pola.rs can read them and perform data analysis with jupyter lab, plotly or interact with SQL operations, etc.
Looking for feedback and ideas on what to progress on.
For example, "through history, are my timings getting better?". etc. Also would love to have ideas on what libraries to use to perform forecasting.
57 comments
[ 3.2 ms ] story [ 144 ms ] threadThe DSL supports comparison operators, '&&', '||' and nested expressions, while preventing mixing of '&&' and '||' without making precedence clear using parentheses. This DSL should fit 'non-trivial', but it still should be simple enough to easily understand it.
(which both includes a full json parser for each parser)
Yes, please.
- T, Z
I'm not sure if the older patches are hot-downloaded on-demand when you load an old replay, I think they actually might be -- but I don't know.
You can also "resume" a replay from any point and you and a friend can take over the game from that point. I think Harstem (both a professional player and a big streamer/ YouTuber) has a few YouTube videos where he explores a famous game where one pro player "had clearly lost and had absolutely no chance to win" but the losing player stayed in the game for a really, really long time and people were talking about whether and to what degree that was "unsportsmanlike" (similar to not resigning in a clearly-lost chess game). Harstem would take the losing side at the point in time that was generally agreed to be "lost" and some of his viewers of various ability levels and IIRC he beat all of them.
I studied SC2 altogether far, far, too much from 2019-2022. Had a goal to get into masters, played tens of thousands of games, watched multiple YouTube videos every day, memorized and practiced dozens of build orders, but still got stuck at my peak in low diamond.
When you're playing that much and reviewing replays, you start wondering why you can't watch old replays with people coaching you and someone eventually ends up explaining the system to you.
Uhh, this is confusing as written. But there are cheating hacks which allow you to see extra data that is sent to your client in-game, such as the entire map and your's and your opponents production queues. And while I'm not sure if this is what you were asking: "sniffing" events from other clients which haven't been sent to the blizzard server is not possible, because they're delivered to your client via the blizzard server (it's not P2P).
> Also curious about the drone kill counter mod/overlay that Harstem uses, do you know of the name of that?
It's a combination of two things: 1) A customized UI layout for replays called "GameHeart" and 2) a custom SC2 game mod called "GameHeart" -- you can use the UI layout for ladder games which weren't created as custom games with the GameHeart mod, but for that replay you'll miss out on features like the drone kill counter and income graph vs. time for which the data is generated by the game mod.
With a relatively larger dataset you could come up with some real interesting statistics on individual players performance, use of certain units, the success of various strategies and build orders, etc.
It would be real fun to try to predict the outcome of a game based on the first 3 minutes or something.
If it's quite good it could be useful as a tool for people trying to get better (although various tooling and custom maps have been around for these types of purposes since the game came out and most people even pros seem to just grind ladder for practice).
I was also hoping it could be used to analyze a replay and indicate how "back and forth" the game was, or if it was more one-sided, etc. Or if the top 100 pros were part of the featurization, then you could potentially determine how the odds change given a certain game state depending on who the pro is.
SC2replaystats had 4,440,656 replays uploaded in 2020 and 3,390,117 in 2019 (the only years I can quickly find data for). Searching and downloading replays is now paywalled, probably to limit scraping.
> Wouldn't mind paying a bit for that...
Google probably paid Blizzard a good amount of money to help them develop AlphaStar and in the process got access to all the replays. But I doubt you'd be offering enough money to get executive-level attention for such a deal. Sc2replaystats might be more receptive to selling you a randomly selected subset of their total data, but you'd still have to offer enough cash to be clearly worth the time of going through the negotiation process. I doubt they'd sell you all of their replays, as that's essentially the business' moat.
Perhaps you could buy the sc2replaystats business though.
(Yes, I know none of these suggestions are practical for the vast majority of interested hackers)
----------
There are smaller free replay packs available with a quick Google search. I'm sure if you looked around or asked around either Harstem's SC2 discord server or the SC2AI discord server or SC2 subreddit, someone would be able to point you to a free vault offering on the order of 10,000's of replays.
However, the game is undergoing a period of large updates right now which are dramatically changing the details of the gameplay ... so really you'd want as many as possible starting from the most recent "balance patch" which went live just 6 days ago[0]. It will take the SC2 community quite a long time to establish known "best practices" for the current balance patch, and it's highly likely that it will be significantly modified again soon -- the most recent patch seems to be politically fragile at all levels of the game. Casual to professional players all have valid constructive criticism of it, so more changes soon are essentially inevitable.
0: https://news.blizzard.com/en-us/starcraft2/24009150/starcraf...
Feedback like "your baneling run-bys are most cost-effective with 7 banelings around the ten minute mark" is firstly unlikely to be accurate, and secondly unlikely to be useful to a pro because their play is so situational.
There also isn't enough data in the replay files to piece together the kind of specific information they would want.
SC2 has extremely high variance, even at the pro level. It's very hard to mine useful insights from game data. Even pro builds, which are supposedly standardized, have crazy variation: https://sc2.gg/reports/top-openings-2022/.
There are so many variations in games, and there are often adjustments on the fly or even sometimes mess ups.
It's not easy to work with proprietary formats, but they've both become pretty popular, so I would 100% recommend sinking more time into this project as long as it scratches your itch. Gamers are always looking for more stats and deeper insights
1. https://aoestats.io/
If you search Direct Strike Starcraft on youtube you should be able to see the gameplay itself.
For context, I have an open source video player designed for esports coaches. The main feature is that you can load in multiple video streams at once at synchronise them together. Mainly for FPS games like Valorant / Apex Legends (https://www.vodon.gg/ if you want to explore it).
I'm starting to get access to some streams of data from the games via coaches that use the tool. My very naïve approach was simply to load game events into the video timeline (so you could easily skip forwards to deaths, kills etc) but I hadn't thought about loading this into data analysis tools.
The game events themselves for Valorant seem like they'd be enough to almost construct an online replay from them as well, which could compliment the recorded gameplay (i.e. construct a dynamic map of where everybody is that could be brought up on screen).
It's a very cool space, if you'd like to chat more my email is in my profile.
I implemented a basic one in Rust a while back: https://github.com/ZephyrBlu/rust-parser
And a full one in Python with a few bells and whistles ages ago: https://github.com/ZephyrBlu/zephyrus-sc2-parser
Don't maintain either of them though :(, and the Rust one is super rough.
SC2 is a very interesting area for data analysis, but at the same time I found it very challenging. There is so much nuance and inconsistency across games it can be really hard to do accurately do things like categorize builds or measure build timings.
The area I ended up focusing on was builds, and I feel like I did some interesting stuff there: https://sc2.gg/reports/top-openings-2022/.
I found personal statistics less interesting than aggregate statistics. Even pro games are very volatile, ladder games even more so. Extremely hard to get reliable signal out of them if you're trying to track things across games. Even simple things like Collection Rate are poor indicators without significant categorization work (Matchup, build, opponent build, etc).
https://www.nature.com/articles/s41597-023-02510-7
Feel free to look into the tools on my GitHub (https://github.com/Kaszanas). Since this is mostly the topic of my PhD I guess I will be updating the dataset in the near future. You may want to try and test your parser against it.
further research for you would probably include running Logistic Regression on aggregated data from each of the replays to try and have a model that can discern between winners and losers and see which parameters are key in your data.
Example: https://www.researchgate.net/publication/363613604_Determina...
And even further embedding the games as timeseries data via various methods.
[0]: https://youtu.be/yBCe8SqGwK8
[1]: https://jku-vds-lab.at/publications/2022_embedding_structure...