Kickstarter scrape text

Note that using an API and scraping a website are different things. So it appears you are not using an API. This is important to note seeing as APIs are specifically designed to allow you to get data and will not attempt to block you from doing so; whereas often times websites will try to prevent you from scraping as it adds to their data bills but is not a "real user"...often times it will be against their terms of service.

However in the case that there is no API available, the website is actively trying to stop you from scraping, but you still want to go ahead... then you can still do it technically.

In your case you mentioned seeing a "Human or Robot" message. This means they were sometimes or always showing a "Captcha". You can get around this by using a "Captcha Solving" service. Basically companies have setup with a whole bunch of low paid freelancers clicking "I'm a human" all day. Just search Google and you will find a lot of services being offered.

I have found some of these services to be very good and often can reliably solve captchas in less than 30 seconds. Most of them will provide sample API code in multiple languages (including python) so that you can easily integrate your code.

Basically your code will flow like this:

1. Access the webpage you wish to crawl
2. Check if you hit a Captcha
3. Send the captcha data to the solving service API
4. Submit the captcha result receive to the webpage
5. Grant access to the webpage and start your scraping

Note you should make sure your web scraper is using cookies so that you can maintain your session throughout your scraping, this will hopefully minimize the number of Captchas you hit.

You may also need to use a proxy to vary your IP address so that you are not always scraping from the same IP.