barnesdmd.co.uk

A brief little update on the script I’ve written which I blogged about the other day which reparses the partial feeds from the BBC news website and creates full feeds of these.

I’ve gone through the script tonight adding scrape caching (6 hour cache currently) and a few other tweaks, this has decreased the build time on it by around three quarters so quite a bit, although of
course this varies if articles have to be fetched etc.

Now that I’ve done this I can now add a little more filtering to the scraper to remove the occasional unwanted elements which creep up from time to time such as voting forms which I didn’t originally think about. Another thing which I want to deal with can be seen if you look at the source code on any BBC news article, there are very few closing tags for paragraphs so I’d like to deal with this within my parser at least.

Its been quite interesting actually now that I’ve released this, I added a log to the script to
allow me to see more detail what is being fetched and when (and there are quite a large number of people using it). I didn’t realise for instance that the useragent string pulled from a Bloglines request gives the number of subscribers which its catering for which is quite useful. Google reader however doesn’t appear to do this which is a pity but of no real significance at this stage.

Tags: BBC, RSS, Bloglines, Google Reader

Tags: bbc, bbc-full-feeds, rss

This entry was posted on Thursday, November 30th, 2006 at 2:03 am and is filed under Internet. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.