Replay Crawler

Discussion of all aspects of the website, wiki, and forums, including support requests and new ideas.

Moderators: Forum Moderators, Developers

Post Reply
User avatar
Talkative
Posts: 4
Joined: October 22nd, 2008, 12:02 am

Replay Crawler

Post by Talkative » January 23rd, 2016, 7:59 am

My apologies if this is the wrong forum for this topic; it seemed most fitting.

I played a lot of Battle for Wesnoth back in the 1.6 through 1.10 days, and met a handful of wonderful people through casual survival games. As a matter of fact, I ended up dating one of those people and we just celebrated our third anniversary.

We were waxing nostalgic the other day about our first games together, when we were strangers, and it occurred to me that I might be able to find that game by poking around http://replays.wesnoth.org/. After a couple minutes' idle clicking I threw up my hands and wrote a (very naive) web crawler to iterate through the HTML and pull down replays with our names in the descriptions.

A few questions!

1) I don't mean to clobber the replay server. Is there a more community-friendly way of finding some games with my friends from years ago?
2) Are replays ever deleted? Even with automated searching, my searches have only turned up fairly recent games (1.8 onward)
3) Anybody want the code?
He wondered idly if they might be friends someday.

User avatar
Iris
Site Administrator
Posts: 6588
Joined: November 14th, 2006, 5:54 pm
Location: Chile
Contact:

Re: Replay Crawler

Post by Iris » January 23rd, 2016, 8:23 am

Talkative wrote:1) I don't mean to clobber the replay server. Is there a more community-friendly way of finding some games with my friends from years ago?
Not really, but unless you are going to spam a hundred or more GET requests per second, it’s doubtful it will have any impact on the web server.
Talkative wrote:2) Are replays ever deleted? Even with automated searching, my searches have only turned up fairly recent games (1.8 onward)
They are not backed up, that much I can say. There have been a few hard disk crashes in the past on both our current and previous hosts and I wouldn’t rule out replays disappearing in the process. It’s also possible that one of the other system admins may have deleted old replays before if we ever ran out of space on the old host (which was a relatively common occurrence as far as I know). In general, I’d recommend people to assume replays.wesnoth.org may be nuked or pruned at any time if the need arises.
Author of the unofficial UtBS sequels Invasion from the Unknown and After the Storm.

Post Reply