ScraperWiki Digger Gets HTTPS Security System

During the last week of rioting across the UK 8 riot vans were called out to quel the unrest just around the corner from where I live. With scenes of chaos and destruction filling the airwaves and clogging up twitter I begun thinking: Are your scrapers safe from looters? We don’t stock trainers or flat-screen TVs, but data is much more valuable. And we want to keep it safe.

Remember the furore over Firesheep? It made it easy to hack into other people’s Facebook accounts from Internet cafes. That’s just one example of why it is important that all traffic between a web application and the browser is encrypted. So we’re setting up our digger with an alarm system and steering lock.

We’ve just rolled out secure HTTP across ScraperWiki. You now have to access the website, APIs and views using https:// URLs. You will be able to tell from the corner of your browser, for example on Chrome it looks green with a padlock like this.

Let us know if you have any problems with the new setup – particularly from tools that use the API. We’ve set up redirects and done some testing, so everything should be fine. But there are sure to be niggles – do tell us.

P.S. This is all in preparation towards making private scrapers more generally available.

This entry was posted in developer and tagged , . Bookmark the permalink.

3 Responses to ScraperWiki Digger Gets HTTPS Security System

  1. Francis Irving says:

    To keep the OpenCorporates import scripts, and possibly others, working, we’ve changed it so that the API (at api.scraperwiki.com) is served on both http:// and https://.

    This is fine for public data anyway. We’ll probably revisit that when we have APIs for accessing private data.

  2. Thad Guidry says:

    Anytime line for private scrapers ? Which is your quick guess 3-6 months, or closer to 6 months-1 year ? I’m cool with either, I’m toying with the idea of just waiting on private scrapers & you hosting, or just actually trying to offline ScraperWiki, pull down the source code, and wire it up for myself.

  3. Francis Irving says:

    Much sooner – we can do one off private scrapers now from our admin interface. Will have it available with slicker interface for things like transfering scrapers to other users, and the ability to offer a free trial to beta testers, later this month. Self service will be much later in the year.

    Could you email us with what your requirements are?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s