The High-Throughput Proxy for Amazon provides tools to retrieve product information from Amazon with zip code specific geolocation.
See Distributed Proxy Service for further information about Charity Engine web crawling tools; see also Computing with Charity Engine for an overview of the complete Charity Engine ecosystem.
Contents
Making Requests
The High-Throughput Proxy for Amazon is a proxy service that accepts HTTP(S) requests via standard proxy protocol, just like the Distributed Proxy Service, with some additional configuration options and certain limitations.
This proxy only accepts requests to Amazon domains, and the zip code functionality is only supported for the United States of America (US) geolocation. Requests that do not require geolocation should continue to use the standard Distributed Proxy Service.
For support of other geolocations, please contact us.
HTTP endpoint
http://amazon.proxy.charityengine.services:80
Note: the vast majority of pages on Amazon require HTTPS; the HTTPS endpoint should be used in most cases.
HTTPS endpoint
https://amazon.proxy.charityengine.services:443
Specifying zip code
Country-specific geolocation uses the X-Proxy-Country HTTP header, as with the generic Distributed Proxy:
X-Proxy-Country: US
To specify the country and the zip code, extend the X-Proxy-Country header with a separator and the 5-digit zip code (US specific):
X-Proxy-Country: US##50001
If no zip code is specified, a random zip code will be chosen from all available proxy nodes.
Note that not all 5-digit numbers are valid US zip codes. The request will be rejected if an invalid zip code is specified.
When starting to use a zip code that has not been recently used in significant quantities, there is a warm up period where requests can take longer than usual while the proxy nodes corresponding to the specific zip code are located. The response times should normalize after several hundred requests for a zip code are completed.