Replay Crawler
Moderator: Forum Moderators
Replay Crawler
My apologies if this is the wrong forum for this topic; it seemed most fitting.
I played a lot of Battle for Wesnoth back in the 1.6 through 1.10 days, and met a handful of wonderful people through casual survival games. As a matter of fact, I ended up dating one of those people and we just celebrated our third anniversary.
We were waxing nostalgic the other day about our first games together, when we were strangers, and it occurred to me that I might be able to find that game by poking around http://replays.wesnoth.org/. After a couple minutes' idle clicking I threw up my hands and wrote a (very naive) web crawler to iterate through the HTML and pull down replays with our names in the descriptions.
A few questions!
1) I don't mean to clobber the replay server. Is there a more community-friendly way of finding some games with my friends from years ago?
2) Are replays ever deleted? Even with automated searching, my searches have only turned up fairly recent games (1.8 onward)
3) Anybody want the code?
I played a lot of Battle for Wesnoth back in the 1.6 through 1.10 days, and met a handful of wonderful people through casual survival games. As a matter of fact, I ended up dating one of those people and we just celebrated our third anniversary.
We were waxing nostalgic the other day about our first games together, when we were strangers, and it occurred to me that I might be able to find that game by poking around http://replays.wesnoth.org/. After a couple minutes' idle clicking I threw up my hands and wrote a (very naive) web crawler to iterate through the HTML and pull down replays with our names in the descriptions.
A few questions!
1) I don't mean to clobber the replay server. Is there a more community-friendly way of finding some games with my friends from years ago?
2) Are replays ever deleted? Even with automated searching, my searches have only turned up fairly recent games (1.8 onward)
3) Anybody want the code?
He wondered idly if they might be friends someday.
Re: Replay Crawler
Not really, but unless you are going to spam a hundred or more GET requests per second, it’s doubtful it will have any impact on the web server.Talkative wrote:1) I don't mean to clobber the replay server. Is there a more community-friendly way of finding some games with my friends from years ago?
They are not backed up, that much I can say. There have been a few hard disk crashes in the past on both our current and previous hosts and I wouldn’t rule out replays disappearing in the process. It’s also possible that one of the other system admins may have deleted old replays before if we ever ran out of space on the old host (which was a relatively common occurrence as far as I know). In general, I’d recommend people to assumeTalkative wrote:2) Are replays ever deleted? Even with automated searching, my searches have only turned up fairly recent games (1.8 onward)
replays.wesnoth.org
may be nuked or pruned at any time if the need arises.Author of the unofficial UtBS sequels Invasion from the Unknown and After the Storm.