Universal Scraper Enterprise

Introduction

Universal Scraper Enterprise provides your organization with unparalleled control over web data extraction. Our solution handles:

• Dynamic execution of custom JavaScript instructions
• Bypassing Cloudflare and captcha challenges
• Advanced header customization and proxy management
• Persistent session handling with detailed reporting

Server Endpoints

Root Endpoint

URL: /
Method: GET
Description: Serves this documentation page.

Scraping Job Endpoint

URL: /v1
Methods: GET, POST
Description: Initiates a scraping job and returns a jobId along with a status URL.

Job Status Endpoint

URL: /status/:jobId
Method: GET
Description: Retrieves the current status and results of your scraping job.

Request Parameters & Options

The /v1 endpoint accepts the following parameters:

• url (string): Target URL to scrape (e.g., "https://example.com").
• instructions (array/JSON string): Sequence of actions to perform. Example:
  Copy

  [ 
    { "wait_for": "#login" },
    { "fill": ["#username", "user123"] },
    { "fill": ["#password", "secret"] },
    { "click": "#submit" }
  ]

• useProxy (boolean): Enable proxy usage.
• proxies (array): List of proxy addresses.
• customHeaders (boolean): Enable custom HTTP headers.
• headers (object/JSON string): Custom headers (e.g., {"User-Agent": "MyAgent"}).
• requestMethod (string): HTTP method (GET, POST, etc.).
• postData (object/string): Data for POST requests.
• block_resources (array): Resource types to block (e.g., ["image", "stylesheet"]).
• js_render (boolean): Enable JavaScript rendering.
• json_response (boolean): Include additional reporting details.
• wait_param (number): Milliseconds to wait after page load.
• wait_for_param (string/boolean): CSS selector to wait for.
• session_id (string): ID for a persistent browser session.
• delete_session_id (boolean): End the session after job completion.
• country_code (string): Specify proxy region (e.g., "US").

JS Instruction Set

The instructions parameter defines the actions executed within the browser. Supported instructions include:

Basic Navigation & Waits

• wait_for: Wait for a specific element.
  Copy

  { "wait_for": "#main-content" }

• wait: Pause execution for a defined time in milliseconds.
  Copy

  { "wait": 3000 }

• wait_event: Wait for events such as networkidle or load.
  Copy

  { "wait_event": "networkidle" }

• navigate: Navigate to a URL or relative path.
  Copy

  { "navigate": "/checkout" }

User Interactions

• click: Simulate a mouse click.
  Copy

  { "click": "#login-button" }

• wait_for_and_click: Wait for an element then click it.
  Copy

  { "wait_for_and_click": "#submit-button" }

• fill: Fill an input field with text.
  Copy

  { "fill": ["#username", "user123"] }

• check: Ensure a checkbox is checked.
  Copy

  { "check": "#terms" }

• uncheck: Ensure a checkbox is unchecked.
  Copy

  { "uncheck": "#subscribe" }

• select_option: Choose an option from a dropdown.
  Copy

  { "select_option": ["#country", "US"] }

Scrolling & Evaluation

• scroll_x: Scroll horizontally.
  Copy

  { "scroll_x": 200 }

• scroll_y: Scroll vertically.
  Copy

  { "scroll_y": 400 }

• evaluate: Execute custom JavaScript code.
  Copy

  { "evaluate": "document.body.style.backgroundColor = 'black'" }

Captcha & Cloudflare Bypass

• solve_cloudflare: Bypass Cloudflare challenges.
  Copy

  { "solve_cloudflare": true }

• solve_captcha: Solve captchas via CapSolver.
  Copy

  { "solve_captcha": true }

Frame-specific Actions

• frame_click: Click an element within an iframe.
  Copy

  { "frame_click": ["iframe#loginFrame", "#loginButton"] }

• frame_wait_for: Wait for an element inside an iframe.
  Copy

  { "frame_wait_for": ["iframe#loginFrame", "#username"] }

• frame_fill: Fill an input field within an iframe.
  Copy

  { "frame_fill": ["iframe#loginFrame", "#username", "testuser"] }

• frame_check: Check a checkbox inside an iframe.
  Copy

  { "frame_check": ["iframe#loginFrame", "#rememberMe"] }

• frame_uncheck: Uncheck a checkbox inside an iframe.
  Copy

  { "frame_uncheck": ["iframe#loginFrame", "#newsletter"] }

• frame_select_option: Select an option inside an iframe.
  Copy

  { "frame_select_option": ["iframe#loginFrame", "#country", "US"] }

• frame_evaluate: Execute JavaScript within an iframe.
  Copy

  { "frame_evaluate": ["iframe#loginFrame", "document.body.style.backgroundColor = 'blue'"] }

Session & Proxy Management

Persistent browser sessions are maintained for 30 minutes. Provide a session_id to reuse an instance; use delete_session_id to terminate it. Proxies can be configured via the proxies parameter and refined with the country_code.

Integration Examples

The following helper functions demonstrate how to integrate with Universal Scraper Enterprise in PHP, Python, and Node.js. For each language, separate examples are provided for GET and POST requests.

<?php
function scrape(
    $targetUrl,
    $instructions = [],
    $proxies = [],
    $headers = null,
    $httpMethod = 'GET',
    $postData = false,
    $block_resources = [],
    $js_render = false,
    $json_response = false,
    $wait_param = 0,
    $wait_for_param = false,
    $session_id = null,
    $delete_session_id = false,
    $country_code = "any",
    $waitSeconds = 60,
    $maxAttempts = 20
) {
    if (empty($targetUrl) || !filter_var($targetUrl, FILTER_VALIDATE_URL)) {
        throw new Exception("Parameter 'targetUrl' must be a valid URL.");
    }
    if (!is_array($instructions)) {
        throw new Exception("Parameter 'instructions' must be an array.");
    }
    $baseUrl = "https://advanced-scraper.com/v1/";
    $statusBaseUrl = "https://advanced-scraper.com/status/";
    $params = [
        "url"               => $targetUrl,
        "instructions"      => json_encode($instructions),
        "useProxy"          => (!empty($proxies)) ? "true" : "false",
        "proxies"           => json_encode($proxies),
        "customHeaders"     => ($headers !== null) ? "true" : "false",
        "headers"           => ($headers !== null) ? json_encode($headers) : "null",
        "requestMethod"     => strtoupper($httpMethod),
        "postData"          => $postData ?? "false",
        "block_resources"   => json_encode($block_resources),
        "js_render"         => $js_render ? "true" : "false",
        "json_response"     => $json_response ? "true" : "false",
        "wait_param"        => $wait_param,
        "wait_for_param"    => $wait_for_param ? "true" : "false",
        "session_id"        => ($session_id !== null) ? $session_id : "null",
        "delete_session_id" => $delete_session_id ? "true" : "false",
        "country_code"      => $country_code
    ];
    $queryString = http_build_query($params);
    $requestUrl = $baseUrl . "?" . $queryString;
    $ch = curl_init($requestUrl);
    curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
    curl_setopt($ch, CURLOPT_POSTFIELDS, $queryString);
    curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type: application/x-www-form-urlencoded"));
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
    curl_setopt($ch, CURLOPT_TIMEOUT, 30);
    $response = curl_exec($ch);
    if (curl_errno($ch)) {
        $err = curl_error($ch);
        curl_close($ch);
        throw new Exception('Job creation error: ' . $err);
    }
    curl_close($ch);
    $data = json_decode($response, true);
    if (!$data || !isset($data['jobId'])) {
        throw new Exception("Job creation failed. Response: " . $response);
    }
    $jobId = $data['jobId'];
    $statusUrl = $statusBaseUrl . $jobId;
    $attempts = 0;
    $statusData = null;
    do {
        sleep($waitSeconds);
        $chStatus = curl_init($statusUrl);
        curl_setopt($chStatus, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($chStatus, CURLOPT_FOLLOWLOCATION, true);
        curl_setopt($chStatus, CURLOPT_CONNECTTIMEOUT, 10);
        curl_setopt($chStatus, CURLOPT_TIMEOUT, 30);
        $statusResponse = curl_exec($chStatus);
        if (curl_errno($chStatus)) {
            $errStatus = curl_error($chStatus);
            curl_close($chStatus);
            throw new Exception("Failed to fetch job status: " . $errStatus);
        }
        curl_close($chStatus);
        $statusData = json_decode($statusResponse, true);
        if (!$statusData || !isset($statusData['status'])) {
            throw new Exception("Invalid status response: " . $statusResponse);
        }
        $attempts++;
        if ($attempts >= $maxAttempts) {
            throw new Exception("Maximum polling attempts reached.");
        }
    } while ($statusData['status'] === 'processing');
    if ($statusData['status'] === 'completed') {
        return $statusData['result'];
    } else {
        throw new Exception("Job failed: " . ($statusData['error'] ?? "Unknown error."));
    }
}
?>
            

// Scrape a page via GET
$result = scrape(
    "https://example.com",
    [ { "wait_for": "#main-content" } ]
);
            

// Submit data via POST
$result = scrape(
    "https://example.com/api/submit",
    [],
    [],
    [ "Content-Type" => "application/json" ],
    "POST",
    '{"name":"John Doe","email":"john@example.com"}'
);
            

import time, requests, urllib.parse, json

def scrape(target_url, instructions=[], proxies=[], headers=None, http_method='GET', post_data=False,
           block_resources=[], js_render=False, json_response=False, wait_param=0, wait_for_param=False,
           session_id=None, delete_session_id=False, country_code="any", wait_seconds=60, max_attempts=20):
    parsed = urllib.parse.urlparse(target_url)
    if not all([parsed.scheme, parsed.netloc]):
        raise Exception("Parameter 'target_url' must be a valid URL.")
    if not isinstance(instructions, list):
        raise Exception("Parameter 'instructions' must be a list.")
    base_url = "https://advanced-scraper.com/v1/"
    status_base_url = "https://advanced-scraper.com/status/"
    params = {
        "url": target_url,
        "instructions": json.dumps(instructions),
        "useProxy": "true" if proxies else "false",
        "proxies": json.dumps(proxies),
        "customHeaders": "true" if headers is not None else "false",
        "headers": json.dumps(headers) if headers is not None else "null",
        "requestMethod": http_method.upper(),
        "postData": post_data if post_data else "false",
        "block_resources": json.dumps(block_resources),
        "js_render": "true" if js_render else "false",
        "json_response": "true" if json_response else "false",
        "wait_param": wait_param,
        "wait_for_param": "true" if wait_for_param else "false",
        "session_id": session_id if session_id is not None else "null",
        "delete_session_id": "true" if delete_session_id else "false",
        "country_code": country_code
    }
    query_string = urllib.parse.urlencode(params)
    request_url = base_url + "?" + query_string
    response = requests.post(request_url, data=query_string,
                             headers={"Content-Type": "application/x-www-form-urlencoded"}, timeout=30)
    if response.status_code != 200:
        raise Exception("Job creation error: " + response.text)
    data = response.json()
    if "jobId" not in data:
        raise Exception("Job creation failed. Response: " + response.text)
    job_id = data["jobId"]
    status_url = status_base_url + job_id
    attempts = 0
    while attempts < max_attempts:
        time.sleep(wait_seconds)
        status_response = requests.get(status_url, timeout=30)
        status_data = status_response.json()
        if "status" not in status_data:
            raise Exception("Invalid status response: " + status_response.text)
        if status_data["status"] != "processing":
            break
        attempts += 1
    if attempts >= max_attempts:
        raise Exception("Maximum polling attempts reached.")
    if status_data["status"] == "completed":
        return status_data["result"]
    else:
        raise Exception("Job failed: " + status_data.get("error", "Unknown error."))

# Helper function is ready.
            

# Scrape a page via GET
result = scrape("https://example.com", [{"wait_for": "#main-content"}])
print(result)
            

# Submit data via POST
result = scrape("https://example.com/api/submit", [],
                headers={"Content-Type": "application/json"},
                http_method="POST",
                post_data='{"name":"Jane Doe","email":"jane@example.com"}')
print(result)
            

const axios = require('axios');
const querystring = require('querystring');

async function scrape(targetUrl, instructions = [], proxies = [], headers = null, httpMethod = 'GET', postData = false,
                      block_resources = [], js_render = false, json_response = false, wait_param = 0, wait_for_param = false,
                      session_id = null, delete_session_id = false, country_code = "any", waitSeconds = 60, maxAttempts = 20) {
  try {
    new URL(targetUrl);
  } catch (e) {
    throw new Error("Parameter 'targetUrl' must be a valid URL.");
  }
  if (!Array.isArray(instructions)) {
    throw new Error("Parameter 'instructions' must be an array.");
  }
  const baseUrl = "https://advanced-scraper.com/v1/";
  const statusBaseUrl = "https://advanced-scraper.com/status/";
  const params = {
    url: targetUrl,
    instructions: JSON.stringify(instructions),
    useProxy: proxies.length > 0 ? "true" : "false",
    proxies: JSON.stringify(proxies),
    customHeaders: headers !== null ? "true" : "false",
    headers: headers !== null ? JSON.stringify(headers) : "null",
    requestMethod: httpMethod.toUpperCase(),
    postData: postData || "false",
    block_resources: JSON.stringify(block_resources),
    js_render: js_render ? "true" : "false",
    json_response: json_response ? "true" : "false",
    wait_param: wait_param,
    wait_for_param: wait_for_param ? "true" : "false",
    session_id: session_id !== null ? session_id : "null",
    delete_session_id: delete_session_id ? "true" : "false",
    country_code: country_code
  };
  const queryString = querystring.stringify(params);
  const requestUrl = `${baseUrl}?${queryString}`;
  let response;
  try {
    response = await axios.post(requestUrl, queryString, {
      headers: { "Content-Type": "application/x-www-form-urlencoded" },
      timeout: 30000
    });
  } catch (err) {
    throw new Error("Job creation error: " + err.message);
  }
  const data = response.data;
  if (!data || !data.jobId) {
    throw new Error("Job creation failed. Response: " + JSON.stringify(data));
  }
  const jobId = data.jobId;
  const statusUrl = `${statusBaseUrl}${jobId}`;
  let attempts = 0;
  let statusData;
  while (attempts < maxAttempts) {
    await new Promise(resolve => setTimeout(resolve, waitSeconds * 1000));
    try {
      const statusResponse = await axios.get(statusUrl, { timeout: 30000 });
      statusData = statusResponse.data;
    } catch (err) {
      throw new Error("Failed to fetch job status: " + err.message);
    }
    if (!statusData || !statusData.status) {
      throw new Error("Invalid status response: " + JSON.stringify(statusData));
    }
    if (statusData.status !== 'processing') {
      break;
    }
    attempts++;
  }
  if (attempts >= maxAttempts) {
    throw new Error("Maximum polling attempts reached.");
  }
  if (statusData.status === 'completed') {
    return statusData.result;
  } else {
    throw new Error("Job failed: " + (statusData.error || "Unknown error."));
  }
}

module.exports = { scrape };
            

// Scrape a page via GET
(async () => {
  try {
    const result = await scrape("https://example.com", [{ "wait_for": "#main-content" }]);
    console.log(result);
  } catch (err) {
    console.error(err);
  }
})();
            

// Submit data via POST
(async () => {
  try {
    const result = await scrape("https://example.com/api/submit", [], [],
      { "Content-Type": "application/json" },
      "POST", '{"name":"Alex Doe","email":"alex@example.com"}');
    console.log(result);
  } catch (err) {
    console.error(err);
  }
})();