WORK IN PROGRESS | Services are in active development and are subject to change.


In general usage, requests made via Charity Engine's Proxy Service are distributed across the entire network, to optimize efficiency. In some cases, dedicated resources may be desirable; related functionality is described below.

Access to dedicated resource pools is by special arrangement only.
(Please contact us for further information).


Contents


Resources

Resources may be allocated for exclusive use for proxy/crawling service, with a limit of 2GB data transfer per month per device. The total data transfer can be calculated as:

[#-of-devices] * 2 / 1000 TB of monthly traffic

Any nodes that have exhausted their limit will return an HTTP 509 Bandwidth Limit Exceeded status code and an additional header: X-Generated-By: CharityEngine 

Connecting to the dedicated proxy pool

The interface to this service is like any standard HTTP proxy. Systems using this service should be configured to route web traffic through Charity Engine as detailed below; any request that goes out will be made by devices in the dedicated pool allocated to the customer or project:

Proxy Configuration
Host: dedicated.proxy.charityengine.services
Port: 20000

[TO COME] Local endpoints are available, if needed, to provide lower latency. Note: localized endpoints can be used only to access proxy nodes from the corresponding region. 

To use a local endpoint, replace the generic host address above with:

  • USA: us.dedicated.proxy.charityengine.services 
  • Europe: eu.dedicated.proxy.charityengine.services 
  • Australia: au.dedicated.proxy.charityengine.services 
  • (*contact us if additional regions are required)

Basic usage

General usage is as outlined in the main Distributed Proxy Service documentation. However, an "X-Proxy-PersistentNodeId" header must be added to requests to allow targeting to specific devices in the dedicated pool, e.g.

$ curl -vkx dedicated.proxy.charityengine.services:20000 --proxy-header "X-Proxy-PersistentNodeId: b4b6ea85-b4bc-461f-bc14-63b7d37206e6" --proxy-user "USER_KEY:" https://www.target.domain

Note the following details when making requests:

  • Customer generates the PersistentNodeId when making a request
  • Node IDs are permanent once generated, but may expire (see Section 4 below).
  • Reaching the allocated limit of devices will generate an HTTP 429 Too Many Requests status code and an additional header: X-Generated-By: CharityEngine 

Ephemeral resources

Dedicated CE devices are online for an average of roughly 8hrs per day, and may not be available every day.

Devices that are unavailable at the time of the request will generate an HTTP 408 Request Timeout status code and an additional header: X-Generated-By: CharityEngine 

For any device which has been unavailable for >72 hours, the PersistentNodeID will become inactive and a new PersistentNodeID will need to be generated. The newly generated ID will be assigned to a new device in order to maintain the contracted pool size.

Expired IDs will generate an HTTP 410 Gone  status code and an additional header: X-Generated-By: CharityEngine

Billing

Billing is based on device-month, with a cap on data-transfer per-device/per-month. Charges vary by device geography. Contact us for details.

When PersistentNodeIDs are retired (per Section 4), replacement IDs will inherit the balance of the monthly data transfer allotment of the retired IDs in order to keep the agreed pool size and aggregate monthly transfer allotments stable.

  • No labels