Comments on: Scraping, scripting, hacking

By: SenHu

SenHu — Thu, 12 Nov 2009 15:49:15 +0000

Wonderful ideas – nice article. I have always felt that a web site needs to decide – is it serving data, user functionality or both ? Often times, the designers and owners of web sites confuse the two.

For example, a site serving stock market data will have tools that will show trends – but only the trends they think are important. If a user wants to check his own custom trend, he/she is out of luck.

I always recommend biterscripting ( http://www.biterscripting.com ) for mining data from web sites. That way one gets his raw data, and can process it in the way he sees fit.

Libraries are servers of data (raw information). In terms of whether (or how) they should develop APIs, it needs to be done rather carefully – eles they will end up being like these stock market web sites – where to get complete information on a particular stock in the format one desires – one has to go to several web sites to accomplish that.

Sen

By: Kris Willis

Kris Willis — Tue, 13 Oct 2009 14:57:40 +0000

For those of you that are interested in watching this talk, it is available on the Internet Archive website at the following address: http://www.archive.org/details/MashOopNorth_MikeEllis

Just waiting for it to download from their servers at an annoying ~250 kB/s

By: Aaron Tay

Aaron Tay — Tue, 07 Jul 2009 14:02:22 +0000

Wasn’t there , but the general idea seemed clear to me from the slides. Some interesting ideas there, wasn’t aware of YQ, Google doc trick etc

By: Jon Mills

Jon Mills — Tue, 07 Jul 2009 12:59:55 +0000

Mike,

The best scraping tool I’ve used is the BeautifulSoup library for Python (http://www.crummy.com/software/BeautifulSoup/). It works well with badly formed HTML. Takes a bit of getting used to – but its definitely worth a look if you don’t mind rolling your own scripts.

Jon.

By: Chris Keene

Chris Keene — Tue, 07 Jul 2009 12:45:30 +0000

Talk made perfect sense to me.

Useful stuff, Yahoo seems to come out with more and more good things (never quite sure what their business plan is). yql looked useful.

And agree with regex. Find it nearly impossible to write my own, hard to get my head in to the right way of approaching a problem.

With all mashups I think there is a split between the informal one-off data extracting/mashing and real Uni/Library services. To be of benefit to library users mashups need to be part of formal services (boring i know) and how we transform informal stuff with (for eg) yahoo pipes in to services which are users can use is something I’m not always clear on.

Chris