Possible Big Project - Reading data from a Web Page?

[n0dice]

Hey guys,

My latest project I'm considering is possibly making my own fantasy sports league. I was wondering if there is a way to automatically take a set of data (statistic) which is located on an external website and process it, a crawler of sorts. What I would be starting off with is NBA.com. An example of what I would need to "crawl" is here:
http://www.nba.com/games/20050407/NYKNJN/boxscore.html

Fortunately the URL of the games I would have to access is not dynamically created and is quite predictable. It's just http://nba.com/games/yyyymmdd/awayteamhometeam/boxscore.html.

What I would need to do is grab the data for each statistical catagory considered and (using php preferably) calculate it with the points system values in my database and display it on my webpage.

Any info would be appreciated,

Thanks.

 

[RTM]

Your best bet would be to check out tutorials on www.hotscripts.com (or similar PHP discussion sites.) Some keywords to look for would be "datafeed" combined with "extraction", "manipulation" etc.

One red flag. Unless the data provided in your example on the 3rd party site is explicity provided for your intended use, there is the risk of the site (nba.com) going after you for inappropriate (and/or excessive) use of their content. This is very easy for them to see, especially if your site is dynamically hitting huge amounts of their content pages on a regular basis.

That being said, they may not proactively monitor their logs for deeplinking to their data. There is always the risk that they change their directory naming convention... this could mean instant breakdown of your content. For example, if they change the format from MMDDYY in a static URL to ... some sort of encrypted date reference such as nnnnnn_xx or whatever, then ... you're stuck!

Best to find a reliable source of the data from a provider that explicitly offers it for external use.

Hope this helps a bit,

Rob

 

[n0dice]

Thanks for the reply.

Can I ask how much of a strain it would be putting on their server though? I mean that site has to get a million hits a day. I'm not looking to make any kind of profit off of my website either. As for the url, they've had the same format for a very long long time and it again it's just a casual site and therefore not a huge deal if it goes down.

Oh, and by deeplinking I presume you mean dynamically linking to the external data? What if I just downloaded that data into the database/a database file once a day? Thanks.

 

[dabb]

I have a site that dynamically pulls in data from several other web sites. I just wrote some crontab started C programs that fetch & parse the external pages and rewrites local HTML that is included by my pages.

In your case, you're only reading scores once a day. It's not like you're going to strain their servers with that and I wouldn't hesitate at all to move forward. Just be aware that unless you're reading a strictly text-only page, parsing can sometimes get a bit tricky as page formats may change frequently. You have to monitor closely and you should have your parsing program notify you if it encounters problems with parsing.

 

[n0dice]

For my purposes, this being a casual site, I'm not going to worry about them changing the template. They have redesigned the site a few times but this template has stayed identical for some time now. The actual scores are in a tiny frame-type thing like this one:
http://scores.nba.com/games/20050407/NYKNJN/boxscore.html

The page does have small images, but the data I need is in a specific text-only table and the images aren't big or disruptive.


As for how I'm going to do it, I have absolutely no clue. I can't find any relavant documentation, basically from that page I need to get the statistical data into my database. Can someone guide me towards the general direction of how I should be doing this? a function perhaps? It doesn't matter whether it's directly there or if I have to download it first.

 

[dabb]

Unless you know how to do some programming, it's not magically going to happen by itself. On my stuff which runs on a linux box, I just execute "lynx -dump http://www.blahblah.com > html.txt" then run a C program to parse out the data I need from the retrieved HTML file. At that point you can do whatever you want with the data. In your case, you probably want to throw the scores into some sort of MySQL database. If you're not a programmer, you will need to find somebody that will do the job for you.

Good luck!

 

[Mystic]

http://scroogle.org

an open source sraper that scrapes everything from googles results, maybe get there source, and check it out, add it to yours, scrape the nba page...

 

[primacomputer]

First off, you need to get the correct URL. That one just contains a script that displays the scores URL in a layer or something. The correct URL is:
http://scores.nba.com/games/20050407/NYKNJN/boxscore.html

You will need the raw HTML of this because some of the content is generated by script. You'll need that script to extract the content. I usually use wget for this.

Then you will need a script to extract the relevant data and convert it into a format you can use. There's no program or function that will do this for you. You'll have to do it yourself.

I've written a lot of programmes for downloading databases like this from web sites. It's not particularly hard, but you will need to know how to code in some language, preferably something that supports regular expressions.

 

[n0dice]

Thanks for the replies, I've been talking to some people elsewhere and have gotten some help but it basically involves parsing the entire massive table manually because the <td>'s don't have unique ID's using preg_match_all(). There are other sites out there that are dynamically getting the statistics into their website and I have to think they're doing it by a method easier then this. I've sent an e-mail to the administrator of one such site where I participate via the forums and have yet to get a reply. I'm going to put this project on hold for the time being, hopefully I can find a better source. I don't like the idea of spending all this time coding only to have NBA.com throw in a   somewhere.

 

[primacomputer]

Most commercial sites will simply pay for this. I once suggested a client have software to do something like this. In the end they paid US$5,000 to the sports association to get the scores in a nice script/web friendly format for a year.

The NBA isn't going to lay everything out there nice and neat for you to leach. Their site is particularly complicated, and I suspect deliberately so. That being said, there is hardly anything that can't be extracted with a a few regexs. I recon you could take this data from raw html to database format in less than 50 lines of code. The nice thing about regexs is that they generally don't get thrown off by an  .
You might want to look at some other site that presents the data in a format that is easier to grab.

 

[Mystic]

there must be a RSS feed out there somewere, it just needs finding

 

[n0dice]

Well I have an update, I've decided to go via the player profiles and simply do the daily updates by subtracting the new total in each catagory from the old one. I should have figured this earlier - the data is much cleaner here. My basic outline for the script is this. The following is something someone from another message board gave me to get me started, I believe I'm getting the hang of it but not quite sure.

<?php

$handle = fopen("http://www.nba.com/players/", "rb");
$contents = '';
while (!feof($handle)) {
$contents .= fread($handle, 8192);
}
fclose($handle);

preg_match_all("/<a[^>]+href=[^>]+playerfile\/([^\/]+)\/[^>]+>/i", $contents, $matches);
$size = count($matches[1]);


?>


Using preg_match_all with the proper regexs (which I'm working on figuring out) this script so goes to the main profile page and get all the players names and counts how many players there are. This works thus far.

After that, I suppose I'd like to extract it (ideally) in one foreach (). Something along the lines of foreach ($matches[1]) start another fopen() sequence (like the one in the beginning) with the url being http://www.nba.com/players/(player_name as extracted from the Array? I don't know how I'll get this one, if anyone has any input I'd appreciate it). After that it would be a matter of using regular expressions to parse the stats table and importing it all into the database.

Does this sound like a worthwhile plan? In particular, I'm iffy about how I'm going to execute my foreach() loop and get the player name into the url... Thanks a lot for your time guys.

 

[primacomputer]

Using preg_match_all with the proper regexs (which I'm working on figuring out) this script so goes to the main profile page and get all the players names and counts how many players there are. This works thus far.

That regex works fine for me but the code doesn't. Perhaps my PHP perl compatible regex system isn't as compatible as it should be?

After that, I suppose I'd like to extract it (ideally) in one foreach (). Something along the lines of foreach ($matches[1]) start another fopen() sequence (like the one in the beginning) with the url being http://www.nba.com/players/(player_name as extracted from the Array? I don't know how I'll get this one, if anyone has any input I'd appreciate it).

$url='http://www.nba.com/playerfile/'.$name.'/index.html';
Should give you a url you can fopen foreach $name of your array of names.

After that it would be a matter of using regular expressions to parse the stats table and importing it all into the database.

This is a lot more tricky. You'll first need to read each table. A regex to get the career totals would be:
"totals -->.*?(<table.*?</table>)"
You'll need to do one of those for each table you want to read in.

To get each of these into a database I would simply strip all tabs and newlines and replace TR and TD tags with tabs and newlines. This gives you a tab delimited data file that is very easy to work with.