Scheduling: A scrape a day keeps stale data away

We’ve just rolled out a change to the default frequency of new scrapers.

They used to default to running once a day. Now they default to not running at all.

We’ve made this change because people often make new scrapers that aren’t ready yet. These run every day and send annoying emails saying that they’re broken.

Now, when your scraper is ready, choose “edit schedule”, and then from the drop-down select how often you want it to run.

The maximum at the moment is once a day. We plan to offer more frequent runs with premium accounts. Meanwhile, do get in touch with us if you have a particular application which needs high frequency scraping. (See the FAQ “How often does my scraper run?“)

Of course, many scrapers are one offs that don’t need scheduling. If you’re happy with the run you did in the editor, then you don’t need to worry about the scraper running itself and altering your data later.

Unless you specifically scheduled it.

This entry was posted in developer and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s