New headless Chrome has been released and has a near-perfect browser fingerprint

Sun, 19 Feb 2023 00:00:00 +0000

The new headless Chrome has been released in November 2022. For the moment, it hasn’t become the standard headless mode. It’s still hidden behind a --headless=new flag. However, it will soon become the standard headless mode, and this is going to change a lot of things when it comes to bot detection, particularly detection based on browser fingerprint.

New headless mode version announcement

headless_mode_util.cc

How is it going to impact bot detection?

TL;DR: The new headless Chrome browser fingerprint is way more realistic than the first/old version of headless Chrome. Depending on the sophistication of your detection engine, it’s going to make it easier for bot developers to bypass detection, particularly detection based on browser fingerprinting signals.

As written in Chromium’s code, The new headless mode // is Chrome browser running without any visible UI. Thus, a lot of subtle differences that used to exist between the old headless Chrome and a genuine headful Chrome don’t exist anymore.

headless_mode_util.h

The new headless mode significantly impacts the browser fingerprint. Thus, If you were relying on techniques the absence of plugins with navigator.plugins.length = 0 as I presented in my previous blog posts ~6 years ago or the absence of window.chrome object, you won’t be able to detect the new version of headless Chrome anymore.

Let’s code two simple bots to observe the changes in browser fingerprints:

one with the old headless version;
one with the new headless version.

I use the following code that leverages the old headless Chrome with puppeteer and visit https://antoinevastel.com to collect the browser fingerprint of the bot. The fingerprint is collected using the fp-collect library. Note that due to my position at DataDome, this library hasn’t been updated with new signals in the last 3 years. However, we’ll see that even using relatively old signals, we already see significant differences in the fingerprint. Note that I remove the canvas value only for readability purposes.

// old-headless.js
const puppeteer = require('puppeteer');

async function run() {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    await page.setRequestInterception(true);
    page.on('request', interceptedRequest => {
        if (interceptedRequest.url().endsWith('collectfp')) {
            const payloadFp = JSON.parse(interceptedRequest.postData());
            delete payloadFp['canvas'];
            console.log(payloadFp);
        }
        
        interceptedRequest.continue();
    });


    await page.goto('https://antoinevastel.com/bots');
    browser.close();
}

(async () => {
    await run();
})();

We dump the value of the old fingerprint in old-fingerprint.json:

{
  plugins: [],
  mimeTypes: [],
  userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/111.0.5555.0 Safari/537.36',
  byteLength: 'unknown',
  gamut: [ true, false, false, true ],
  anyPointer: 'fine',
  anyHover: 'hover',
  appVersion: '5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/111.0.5555.0 Safari/537.36',
  appName: 'Netscape',
  appCodeName: 'Mozilla',
  onLine: true,
  cookieEnabled: true,
  doNotTrack: false,
  hardwareConcurrency: 4,
  platform: 'MacIntel',
  oscpu: 'unknown',
  timezone: -60,
  historyLength: 2,
  computedStyleBody: 'accent-coloralign-contentalign...-webkit-writing-mode',
  languages: [ 'en-US' ],
  language: 'en-US',
  indexedDB: true,
  openDatabase: true,
  screen: {
    wInnerHeight: 600,
    wOuterHeight: 0,
    wOuterWidth: 0,
    wInnerWidth: 800,
    wScreenX: 0,
    wPageXOffset: 0,
    wPageYOffset: 0,
    cWidth: 800,
    cHeight: 865,
    sWidth: 1280,
    sHeight: 800,
    sAvailWidth: 1280,
    sAvailHeight: 800,
    sColorDepth: 30,
    sPixelDepth: 30,
    wDevicePixelRatio: 1
  },
  touchScreen: [ 0, false, false ],
  videoCard: [
    'Google Inc. (Google)',
    'ANGLE (Google, Vulkan 1.3.0 (SwiftShader Device (Subzero) (0x0000C0DE)), SwiftShader driver)'
  ],
  multimediaDevices: { speakers: 1, micros: 1, webcams: 1 },
  productSub: '20030107',
  product: 'Gecko',
  navigatorPrototype: [
    'vendorSub~~~function get vendorSub() { [native code] }',
    '...'
    'valueOf~~~',
    '__proto__~~~',
    'toLocaleString~~~'
  ],
  etsl: 33,
  screenDesc: 'function get width() { [native code] }',
  phantomJS: [ false, false, false ],
  nightmareJS: false,
  selenium: [
    false, false, false,
    false, false, false,
    false, false, false,
    false, false, false,
    false, false, false,
    false, false
  ],
  webDriver: true,
  webDriverValue: true,
  fmget: false,
  domAutomation: false,
  errorsGenerated: [
    'azeaze is not defined',
    null,
    null,
    null,
    null,
    null,
    null,
    "Failed to construct 'WebSocket': The URL 'itsgonnafail' is invalid."
  ],
  resOverflow: {
    depth: 10434,
    errorMessage: 'Maximum call stack size exceeded',
    errorName: 'RangeError',
    errorStacklength: 744
  },
  accelerometerUsed: false,
  audio: {
    nt_vc_output: {
      'ac-baseLatency': 0.005804988662131519,
      'ac-outputLatency': 0,
      '...'
      'an-channelCountMode': 'max',
      'an-channelInterpretation': 'speakers'
    },
    pxi_output: 124.04347657808103
  },
  screenMediaQuery: true,
  hasChrome: false,
  detailChrome: 'unknown',
  permissions: { state: 'prompt', permission: 'denied' },
  iframeChrome: 'undefined',
  debugTool: false,
  battery: true,
  deviceMemory: 8,
  tpCanvas: { '0': 0, '1': 0, '2': 0, '3': 0 },
  sequentum: false,
  audioCodecs: {
    ogg: 'probably',
    mp3: 'probably',
    wav: 'probably',
    m4a: '',
    aac: ''
  },
  videoCodecs: {
    ogg: 'probably',
    h264: '',
    webm: 'probably',
    mpeg4v: '',
    mpeg4a: '',
    theora: ''
  },
  redPill: 0,
  redPill2: null,
  redPill3: '40,40,40,40,60,60,40,40,20,40,20,20,40,2460,',
  uuid: 'f4f3c47d-5d09-5678-2736-3709dc1c5ca8',
  url: 'https://antoinevastel.com/bots'
}

We do the same thing with the new headless Chrome:

// new-headless.js

const puppeteer = require('puppeteer');

async function run() {
    const browser = await puppeteer.launch({
        headless: 'new'
    });
    const page = await browser.newPage();

    await page.setRequestInterception(true);
    page.on('request', interceptedRequest => {
        if (interceptedRequest.url().endsWith('collectfp')) {
            const payloadFp = JSON.parse(interceptedRequest.postData());
            delete payloadFp['canvas'];
            console.log(payloadFp);
        }
        
        interceptedRequest.continue();
    });


    await page.goto('https://antoinevastel.com/bots');
    browser.close();
}

(async () => {
    await run();
})();

We store the fingerprint in new-fingerprint.json.

{
  plugins: [
    'PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
    'Chrome PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
    'Chromium PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
    'Microsoft Edge PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
    'WebKit built-in PDF::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format'
  ],
  mimeTypes: [
    'Portable Document Format~~application/pdf~~pdf',
    'Portable Document Format~~text/pdf~~pdf'
  ],
  userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36',
  byteLength: 'unknown',
  gamut: [ true, false, false, true ],
  anyPointer: 'fine',
  anyHover: 'hover',
  appVersion: '5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36',
  appName: 'Netscape',
  appCodeName: 'Mozilla',
  onLine: true,
  cookieEnabled: true,
  doNotTrack: false,
  hardwareConcurrency: 4,
  platform: 'MacIntel',
  oscpu: 'unknown',
  timezone: -60,
  historyLength: 2,
  computedStyleBody: 'accent-coloralign-contentalign-itemsalign-...-webkit-user-modify-webkit-writing-mode',
  languages: [ 'en-GB', 'en-US', 'en' ],
  language: 'en-GB',
  indexedDB: true,
  openDatabase: true,
  screen: {
    wInnerHeight: 600,
    wOuterHeight: 1371,
    wOuterWidth: 1200,
    wInnerWidth: 800,
    wScreenX: 1302,
    wPageXOffset: 0,
    wPageYOffset: 0,
    cWidth: 800,
    cHeight: 886,
    sWidth: 2560,
    sHeight: 1440,
    sAvailWidth: 2560,
    sAvailHeight: 1440,
    sColorDepth: 24,
    sPixelDepth: 24,
    wDevicePixelRatio: 1
  },
  touchScreen: [ 0, false, false ],
  videoCard: [
    'Google Inc. (Intel Inc.)',
    'ANGLE (Intel Inc., Intel(R) Iris(TM) Plus Graphics 640, OpenGL 4.1)'
  ],
  multimediaDevices: { speakers: 1, micros: 1, webcams: 1 },
  productSub: '20030107',
  product: 'Gecko',
  navigatorPrototype: [
    'vendorSub~~~function get vendorSub() { [native code] }',
    'productSub~~~function get productSub() { [native code] }',
    'vendor~~~function get vendor() { [native code] }',
    'maxTouchPoints~~~function get maxTouchPoints() { [native code] }',
    '...',
    '__proto__~~~',
    'toLocaleString~~~'
  ],
  etsl: 33,
  screenDesc: 'function get width() { [native code] }',
  phantomJS: [ false, false, false ],
  nightmareJS: false,
  selenium: [
    false, false, false,
    false, false, false,
    false, false, false,
    false, false, false,
    false, false, false,
    false, false
  ],
  webDriver: true,
  webDriverValue: true,
  fmget: false,
  domAutomation: false,
  errorsGenerated: [
    'azeaze is not defined',
    null,
    null,
    null,
    null,
    null,
    null,
    "Failed to construct 'WebSocket': The URL 'itsgonnafail' is invalid."
  ],
  resOverflow: {
    depth: 10434,
    errorMessage: 'Maximum call stack size exceeded',
    errorName: 'RangeError',
    errorStacklength: 744
  },
  accelerometerUsed: false,
  audio: {
    nt_vc_output: {
      'ac-baseLatency': 0.005804988662131519,
      'ac-outputLatency': 0,
      'ac-sinkId': '',
      '...'
      'an-channelCountMode': 'max',
      'an-channelInterpretation': 'speakers'
    },
    pxi_output: 124.04347657808103
  },
  screenMediaQuery: true,
  hasChrome: true,
  detailChrome: { properties: 'unknown', connect: 55, sendMessage: 59 },
  permissions: { state: 'prompt', permission: 'default' },
  iframeChrome: 'object',
  debugTool: false,
  battery: true,
  deviceMemory: 8,
  tpCanvas: { '0': 0, '1': 0, '2': 0, '3': 0 },
  sequentum: false,
  audioCodecs: {
    ogg: 'probably',
    mp3: 'probably',
    wav: 'probably',
    m4a: '',
    aac: ''
  },
  videoCodecs: {
    ogg: 'probably',
    h264: '',
    webm: 'probably',
    mpeg4v: '',
    mpeg4a: '',
    theora: ''
  },
  redPill: 0,
  redPill2: null,
  redPill3: '40,40,40,20,40,40,20,20,20,20,20,20,20,120,',
  uuid: '40d642c8-fa51-e16b-82c7-c7db771531e7',
  url: 'https://antoinevastel.com/bots'
}

Then, we can make a simple diffs of the two fingerprints using diff new-fingerprint.json old-fingerprint.json > diff-fingerprints.diff

We obtain the following diff:

2,13c2,4
<   plugins: [
<     'PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
<     'Chrome PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
<     'Chromium PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
<     'Microsoft Edge PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
<     'WebKit built-in PDF::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format'
<   ],
<   mimeTypes: [
<     'Portable Document Format~~application/pdf~~pdf',
<     'Portable Document Format~~text/pdf~~pdf'
<   ],
<   userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36',
---
>   plugins: [],
>   mimeTypes: [],
>   userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/111.0.5555.0 Safari/537.36',
18c9
<   appVersion: '5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36',
---
>   appVersion: '5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/111.0.5555.0 Safari/537.36',
30,31c21,22
<   languages: [ 'en-GB', 'en-US', 'en' ],
<   language: 'en-GB',
---
>   languages: [ 'en-US' ],
>   language: 'en-US',
36,37c27,28
<     wOuterHeight: 1371,
<     wOuterWidth: 1200,
---
>     wOuterHeight: 0,
>     wOuterWidth: 0,
39c30
<     wScreenX: 1302,
---
>     wScreenX: 0,
43,49c34,40
<     cHeight: 886,
<     sWidth: 2560,
<     sHeight: 1440,
<     sAvailWidth: 2560,
<     sAvailHeight: 1440,
<     sColorDepth: 24,
<     sPixelDepth: 24,
---
>     cHeight: 865,
>     sWidth: 1280,
>     sHeight: 800,
>     sAvailWidth: 1280,
>     sAvailHeight: 800,
>     sColorDepth: 30,
>     sPixelDepth: 30,
54,55c45,46
<     'Google Inc. (Intel Inc.)',
<     'ANGLE (Intel Inc., Intel(R) Iris(TM) Plus Graphics 640, OpenGL 4.1)'
---
>     'Google Inc. (Google)',
>     'ANGLE (Google, Vulkan 1.3.0 (SwiftShader Device (Subzero) (0x0000C0DE)), SwiftShader driver)'
198,201c189,192
<   hasChrome: true,
<   detailChrome: { properties: 'unknown', connect: 55, sendMessage: 59 },
<   permissions: { state: 'prompt', permission: 'default' },
<   iframeChrome: 'object',
---
>   hasChrome: false,
>   detailChrome: 'unknown',
>   permissions: { state: 'prompt', permission: 'denied' },
>   iframeChrome: 'undefined',
224,225c215,216
<   redPill3: '40,40,40,20,40,40,20,20,20,20,20,20,20,120,',
<   uuid: '40d642c8-fa51-e16b-82c7-c7db771531e7',
---
>   redPill3: '40,40,40,40,60,60,40,40,20,40,20,20,40,2460,',
>   uuid: 'f4f3c47d-5d09-5678-2736-3709dc1c5ca8',

Exploring browser fingerprint differences

As you can see, there are a lot of differences between the old and the new headless Chrome fingerprints. While some of them are caused by randomness, e.g. the redPill values, others are stable differences that persist over time.

The first difference we notice is the user-agent. By default, old headless Chrome indicated its presence in the user agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/111.0.5555.0 Safari/537.36. The new headless Chrome pretends to be a genuine headful browser and removed the HeadlessChrome substring from its user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36.

Similarly, when it comes to plugins, the old headless Chrome used to return no plugins with navigator.plugins, which is a technique that used to be exploited for detection when Headless Chrome got released 6 years ago, cf this blog post. The new headless Chrome returns the same plugins as a headful Chrome, and that’s the same for the mimeTypes obtained with navigator.mimeTypes:

[
    'PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
    'Chrome PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
    'Chromium PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
    'Microsoft Edge PDF Viewer::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format',
    'WebKit built-in PDF::Portable Document Format::internal-pdf-viewer::__application/pdf~pdf~Portable Document Format,text/pdf~pdf~Portable Document Format'
  ],
  mimeTypes: [
    'Portable Document Format~~application/pdf~~pdf',
    'Portable Document Format~~text/pdf~~pdf'
  ]

Because of that, it means that attackers need to apply fewer changes to their fingerprints, which decreases their chances of making a mistake when overriding missing/inconsistent native attributes and functions. For example, as I explained in a recent DataDome blog post about Puppeteer extra stealth, we used to detect that navigator.plugins had been forged because the way it was overridden differed from a native PluginArray object. Now that headless Chrome has a more realistic fingerprint, bot developers won’t need to apply as many JavaScript patches to their headless chrome bots, which will likely decrease their chance of introducing inconsistencies and getting caught.

The new headless Chrome also has a window.chrome object. Thus, similarly to navigator.plugins, it means that attackers won’t need to forge the window.chrome object anymore, which decreases their chances of introducing inconsistent changes in their fingerprint.

Among the other noticeable changes, we notice that information about the GPU looks more human in the latest headless Chrome version. While in the past, headless Chrome indicated that it leveraged SwiftShader, with WebGL renderer values that looked like ANGLE (Google, Vulkan 1.3.0 (SwiftShader Device (Subzero) (0x0000C0DE)), SwiftShader driver), now, it returns similar values as a genuine Chrome browser:

videoCard: [
    'Google Inc. (Intel Inc.)',
    'ANGLE (Intel Inc., Intel(R) Iris(TM) Plus Graphics 640, OpenGL 4.1)'
  ]

Of course, the new headless browser fingerprint still exhibits obvious bot attributes like navigator.webdriver = true. However, these attributes can easily be hidden using flags such as –disable-blink-features=AutomationControlled as discussed in this blog post. So you better not rely only on these signals for your detection. It may help you to catch the less sophisticated bots, but attackers are already aware of these fields and know how to hide them.

Does it mean it has become impossible to detect the new Headless Chrome?

No, the new headless Chrome can still be detected using JS browser fingerprinting techniques. It can be still detected independently of its instrumentation framework such as:

Selenium
Puppeteer
Playwright

However, the task has become more challenging than it used to be. As you can imagine, given my position at DataDome (a bot detection company), I’m not going to share any new detection signals as I used to do. Moreover keep in mind that when it comes to bot detection, browser fingerprinting is not the only signal. It’s important to leverage other signals such as:

Behavior (client-side and server-side)
Different kinds of reputations (IP, sessions, user)
Proxy detection, in particular, residential proxy detection
Contextual information: time of the day, country, etc
TLS fingerprinting.

Using Python3-nmap to fingerprint and cluster proxies

Sun, 20 Nov 2022 00:00:00 +0000

A proxy is a program that enables users to change their IP address by routing traffic through someone else’s infrastructure. Proxies are often used by attackers to distribute their attack and make it look like requests are coming from lot of different devices.

Categories of proxies

There exists different kinds of proxies:

Data-center: the proxy IP located in data-centers like AWS, Google, OVH. They tend to have a low latency due to their location in data-centers with good connectivity.
Residential: the IP belongs to an ISP, e.g. Comcast, Verizon, AT&T. The proxy is located on a real-user device. Thus, the latency tends to be higher than data-center proxies.
ISP: ISP proxies combine the best of data-center and residential proxies. The IP address of ISP proxies are registered under the name of well known ISPs like AT&T, but are located in data centers. Thus, these proxies combine a good IP reputation with a low latency connection.

How do proxy providers obtain proxies?

To obtain data-center proxies, proxy providers need to have physical or virtual machines in the cloud or in data-centers, and run a proxy agent like Squid proxy.

ISP proxies are similar to data center proxies, but proxy providers have to acquire IP blocks to ISPs beforehand. This way, it looks like traffic comes from an ISP IP address instead of a data-center IP. In both case, everything is clean and legal. All machines and IP addresses belong to the proxy provider.

When it comes to residential proxies, the situation may become a litle bit more shady. As I already discussed in this DataDome blog post, proxy providers obtained residential proxies using different. While some approaches are legal, while not being necessarly moral, others rely on malware and are clearly illegal.

Legal methods:

Mobile and software SDKs: some proxy providers offer an SDK to be included in desktop software or mobile applications. These SDKs enable developers to monetize their applications instead of/in addition to ads. While this approach is legal as most SDKs tend to explicitly ask for user permission, a few people probably realize what it implies.
Browser extensions: proxy providers also contact some owners of popular browser extensions so that they include their proxy code in the extension.

Illegal methods:

Malware: Researchers actively fingerprinted proxies to determine the type of devices on which proxies were running. Their analysis showed that some proxies of the proxy network they were using were linked to webcams and IoT devices probably part of a botnet (as it’s not possible to explicitly give consent on these kinds of devices to install an SDK).

Thus, I was curious to see if we could cluster proxies using ports open as a sort of cheap fingerprint. I had no particular expectations in mind, maybe see if some clusters of open ports would be linked:

to data-center or residential proxies;
bots conducting aggressive vulnerability scanning, since it could indicate compromised machines;

Scanning 10K proxies to detect open ports

I use the Avastel all infected IPs 7d blocklist to get a list of IPs recently identified as proxies. I randomly select a sample of 10,000 proxy IPs.

Then, I leverage the Python3 Nmap library to programmatically scan these proxy IPs.

To speed up the scanning process I did two things:

I leverage the ThreadPoolExecutor class to scan multiple IP addresses in parallel.
I scan the top 50 ports using the --top-ports in Nmap. This flag can be used through the scan_top_ports method in the python3-nmap library.

To decrease the chance of having our scan detected and blocked, I also use the Pn Nmap flag.

Disclaimer: We may still miss open ports even using these options. However, it’s still a good estimate for my informal blog post study.

The code below shows the program used for scanning. Results are saved to a JSON file for a later study.

import nmap3
import json
import random
from concurrent.futures import ThreadPoolExecutor, as_completed


def read_ips_list(file_path='', max_ips=100):
    ips = []
    with open(file_path, 'r') as f:
        for line in f.readlines():
            ips.append(line.split(';')[0])

    return random.sample(ips, max_ips)


def save_scan_results(file_path, ip_to_open_ports):
    with open(file_path, 'w+') as f:
        f.write(json.dumps(ip_to_open_ports))


def get_top_ports_open_ip(ip_address, num_ports=50):
    nmap = nmap3.Nmap()
    results = nmap.scan_top_ports(ip_address, default=num_ports, args='Pn')
    ip_address_res_nmap = list(results.keys())[0]
    try:
        open_ports = [port for port in results[ip_address_res_nmap]['ports'] if port['state'] != 'closed' and port['state'] != 'filtered']
        return ip_address, open_ports
    except KeyError:
        return ip_address, []


def main():
    # The list is https://raw.githubusercontent.com/antoinevastel/avastel-bot-ips-lists/master/avastel-ips-7d.txt
    # Version of Nov 1st 2022
    ip_addresses = read_ips_list('./ip-list.txt', 10000)
    ip_address_to_open_ports = dict()

    futures = []
    with ThreadPoolExecutor(max_workers=40) as executor:
        for idx, ip_address in enumerate(ip_addresses):
            future = executor.submit(get_top_ports_open_ip, ip_address)
            futures.append(future)

    tasks_completed = 0
    for future in as_completed(futures):
        tasks_completed += 1
        ip_address, open_ports = future.result()
        ip_address_to_open_ports[ip_address] = open_ports

    save_scan_results('./parallel-results-scan.json', ip_address_to_open_ports)

if __name__ == "__main__":
    main()

Analyzing the proxy IPs scan results

For each IP address scanned, the saved JSON file contains information about the list of ports open:

[
    {
        "protocol": "tcp",
        "portid": "80",
        "state": "open",
        "reason": "syn-ack",
        "reason_ttl": "0",
        "service": {
            "name": "http",
            "method": "table",
            "conf": "3"
        },
        "cpe": [],
        "scripts": []
    },
    {
        "protocol": "tcp",
        "portid": "443",
        "state": "open",
        "reason": "syn-ack",
        "reason_ttl": "0",
        "service": {
            "name": "https",
            "method": "table",
            "conf": "3"
        },
        "cpe": [],
        "scripts": []
    }
]

Thus, we can write a simple program that obtains, for each combination of open ports, the list of IP addresses that have this exact combination of ports open:

{
    "5060,443,1723,80,587,8080": [
        {
            "ip_address": "90.151.104.2",
            "autonomous_system": "Rostelecom"
        }
    ],
    "443,80,993,8080": [
        {
            "ip_address": "106.111.50.202",
            "autonomous_system": "Chinanet"
        }
    ],

    ...

    "5060,993,465,443,80,8080": [
        {
            "ip_address": "45.141.178.7",
            "autonomous_system": "AS-BLAZINGSEO"
        },
        {
            "ip_address": "122.117.6.228",
            "autonomous_system": "Data Communication Business Group"
        }
    ],
    "5060,993,995,465,443,8080": [
        {
            "ip_address": "110.77.149.50",
            "autonomous_system": "CAT TELECOM Public Company Ltd,CAT"
        }
    ],
    "5060,53,22,80,443,1723,21,8080": [
        {
            "ip_address": "218.24.16.198",
            "autonomous_system": "CHINA UNICOM China169 Backbone"
        }
    ],
    "5060,993,995,465,443,80,587,8080": [
        {
            "ip_address": "177.37.179.250",
            "autonomous_system": "BRISANET SERVICOS DE TELECOMUNICACOES LTDA"
        }
    ],
    "587,5060,443,8080": [
        {
            "ip_address": "202.88.37.56",
            "autonomous_system": "HENGTONG-IDC-LLC"
        },
        {
            "ip_address": "104.165.108.114",
            "autonomous_system": "EGIHOSTING"
        }
    ]
}

While we see that some combinations of open ports are only present on a single/few IP proxies, e.g. 587,5060,443,8080 or 5060,993,995,465,443,80,587,8080, others are way more present, e.g. 80, and 443,80. The most common combination of open ports is 8080,5060,443,80, present 3785 times among the 10,000 scanned IPs.

The top 10 combinations of ports are listed below:

ports open;count
443,5060,80,8080;3785
no_ports_open_detected;2584
443,5060,8080;508
5060,80,8080;496
443,5060,80;490
443,80,8080;434
5060,80;189
443,5060;189
5060,8080;164
80,8080;126

I checked if some of these clusters could be linked to specific proxy sources I use to build my list (honeypot, data-center proxies, residential proxies, free proxies). The two most frequent combination of open ports, ‘443,5060,80,8080’ and ‘443,5060,8080’, are mostly linked to data-center proxy providers I use to gather proxy IPs, as well as IP addresses that conducted vulnerability scaning on some of my honeypots). We don’t see a significant volume of clean residential IPs with this combination of open ports.

On the contrary, proxies with ports “5060,80” open seems mostly linked to IPs conducting vulnerability scanning and free proxies scraped online. This finding could indicate the presence of proxies acquired maliciously through the use of malware.

Conclusion and next steps

In this blog post, we showed how to:

create a simple Python program to automatically scan ports open on proxy IPs;
use the combination of open ports as a cheap fingerprint to cluster similar proxy IPs together.

We could go further, for example by making HTTP(s) requests to IPs that have ports often linked to HTTP(s) open such as 80, 443, or 8080. We could also ask nmap to extract more information like the OS and its version. These values could be used as additional bits of information to enrich the proxy fingerprint.

Instead of doing the scan yourself, you can also use services/APIs such as Shodan as they tend to provide these kinds of information as shown on these screenshot below for IP = 221.120.210.67:

However, it has a cost (in case you want to do it for a side project!).

Exploring the Avastel all infected IPs 7d blocklist using Git-python and Matplotlib

Sat, 24 Sep 2022 00:00:00 +0000

TL;DR

In this blog post, we use Git-Python and Matplotlib to explore the Avastel all infected IPs 7d blocklist. The main findings are the following:

There are ~70K distinct IP addresses in the list at a given time;
800,000 malicious IP addresses have been flagged in 8 months;
10.3% of the IPs have been included in the list for > 1 month;
The top 3 autonomous systems whose IP addresses are the most frequently flagged as proxies or used by bots are 1) Korea Telecom, 2) AS Coloam and 3) Chinanet;
We also observe a long tail of data-center and residential autonomous systems from which we originate a significant volume of malicious traffic, such as M247 Ltd, HKT Limited, and Telefonica Brasil.

The purpose of this blog post is two-fold:

Provide insights on the Avastel all infected IPs 7d blocklist. For example, we want to visualize the number of malicious IPs over time, study how long IPs tend to stay in the list, as well as identify the top autonomous systems included in the list;
Show you how you can automate this kind of analysis in Python using Git-python and Matplotlib.

If you’re more interested in the latter, you can jump directly to the last section of the blog post.

Feel free to read this other blog post if you want to learn more about the different IP blocklists I maintain.

Collect metrics automatically using Git-python

First, we create an analysis.py file that contains our Python code used to explore the IP blocklist. touch analysis.py

In the same directory, we clone the avastel-bot-ips-lists repository git clone https://github.com/antoinevastel/avastel-bot-ips-lists.git

You should have the following directory structure:

analysis.py
avastel-bot-ips-lists/

The first step of our analysis is to import git-python and create a repository object that points to the avastel-bot-ips-lists repository. Then, we can list all commits to iterate on them later and see the state of the list for each commit.

from git import Repo
from dateutil import parser

repo = Repo('./avastel-bot-ips-lists')
commits = list(repo.iter_commits('master'))

Then, we iterate on all commits, and for each of them we extract a few metrics, such as:

The date of the list;
The IPs present in the list;
To which autonomous system IP addresses belong to;
The cumulative number of IPs present in the list at a given date.

# A few variables to store metrics about the IP list:
ip_to_dates = dict()
ip_to_as = dict()
num_ips_list_stat = []
cum_num_ips_stat = []
ips_seen = set()

try:
    # We iterate on each commit in chronological order
    for idx, commit in enumerate(reversed(commits)):
        print(f'{idx} - Commit: {commit}')
        repo.git.checkout(commit)
        try:
            with open('./avastel-bot-ips-lists/avastel-ips-7d.txt', 'r') as block_list:
                # We parse the file header to extract information about the date 
                block_list.readline()
                header_date_list = block_list.readline()
                date_str = header_date_list.replace('# Last update: ', '').replace(' (Coordinated Universal Time)', '')
                date_parsed = parser.parse(date_str)

                header_num_ips_blocked = block_list.readline().split(': ')[1].replace('\n', '')
                num_ips_list_stat.append((date_parsed, int(header_num_ips_blocked)))
                block_list.readline()
                block_list.readline()

                for line in block_list:
                    ip_address, autonomous_system = line.replace('\n', '').split(';')
                    ips_seen.add(ip_address)
                    if ip_address not in ip_to_dates:
                        ip_to_dates[ip_address] = []
                        ip_to_as[ip_address] = autonomous_system

                    ip_to_dates[ip_address].append(date_parsed)
                
                cum_num_ips_stat.append((date_parsed, len(ips_seen)))
        except FileNotFoundError:
            # Can happen for the first commits of the repo since the avastel-ips-7d list didn't exist yet
            print('No file')
except Exception as e:
    print(e)
finally:
    # Checkout to master to leave the list repo in a "stable" state no matter what happens
    repo.git.checkout('master')

Then, for each IP address, we compute how long it stayed in the list.

duration_presence_list_ips = []
ip_to_time_in_list = dict()
for ip_address, dates in ip_to_dates.items():
    if len(dates) > 1:
        dates_deduplicate = set([str(d.date()) for d in dates])
        dates.sort()
        duration_presence_list_ips.append(len(dates_deduplicate) - 1)
        ip_to_time_in_list[ip_address] = len(dates_deduplicate) - 1

We also compute the number of distinct IP addresses in the list per autonomous system to make statistics about autonomous systems frequently linked to proxies and bot IPs.

autonomous_system_to_num_ips = dict()
for ip_address in ips_seen:
    if ip_to_as[ip_address] not in autonomous_system_to_num_ips:
        autonomous_system_to_num_ips[ip_to_as[ip_address]] = 0
    
    autonomous_system_to_num_ips[ip_to_as[ip_address]] += 1

Visualize IP address metrics using Matplotlib (Python)

Now that we computed several metrics related to IP addresses and autonomous systems, we can do some data visualization using Matplotlib. First, we import Matplotlib and define some constants to generate consistent graphs.

import matplotlib.pyplot as plt
import matplotlib.dates as mdates

# To have better-looking graphs
plt.style.use('seaborn')

font_size_label = 16
font_size_ticks = 16
font_size_legend = 13
pad_xlabel = 15
pad_ylabel = 15

Then, we define three functions to visualize our data. The first function, plot_time_series takes as input a list of (x, y) tuples representing the data of the time series we want to plot and generate a graph based on it. It also takes as input a y_label parameter to customize the y-axis.

def plot_time_series(graph_xy_data, y_label, output_path):
    _, ax1 = plt.subplots()

    ax1.set_ylabel(y_label, fontsize=font_size_label, labelpad=pad_xlabel)
    ax1.plot([x[0] for x in graph_xy_data],
                        [x[1] for x in graph_xy_data])

    ax1.xaxis.set_major_formatter(mdates.DateFormatter('%m/%Y'))
    ax1.tick_params(axis='both', which='major', labelsize=font_size_ticks)

    ax1.get_yaxis().set_major_formatter(plt.FuncFormatter(lambda x, loc: "{:,}".format(int(x))))

    plt.gcf().autofmt_xdate()
    plt.savefig(output_path, bbox_inches='tight')
    plt.close()

The second function, plot_cdf_duration_ips_list , generates a CDF (cumulative distribution function) graph based on an input list that contains numbers (here, the time an IP has been in the blocklist). Using a CDF graph enables us to better understand the distribution of the data.

def plot_cdf_duration_ips_list(duration_presence_list_ips):
    x = np.sort(duration_presence_list_ips)
    y = 1. * np.arange(len(duration_presence_list_ips)) / (len(duration_presence_list_ips) - 1)

    _, ax1 = plt.subplots()

    ax1.set_ylabel('Ratio of IP addresses', fontsize=font_size_label, labelpad=pad_xlabel)
    ax1.set_xlabel('Duration the IP has been in the list (days)', fontsize=font_size_label, labelpad=pad_xlabel)

    ax1.plot(x, y)

    plt.savefig('./results/cdf_duration_presence_list.png', bbox_inches='tight')
    plt.close()

The third function, plot_top_autonomous_systems, generates a bar chart that represents the number of IP addresses flagged as malicious per autonomous system. You can also configure the number of autonomous systems you want to display on the graph.

def plot_top_autonomous_systems(autonomous_system_to_num_ips, num_ases=10):
    sorted_autonomous_system_to_num_ips = sorted(autonomous_system_to_num_ips.items(), key=lambda item: item[1], reverse=True)
    x_pos = np.arange(num_ases)

    plt.bar(x_pos, [x[1] for x in sorted_autonomous_system_to_num_ips[:num_ases]])
    plt.xticks(x_pos, [x[0] for x in sorted_autonomous_system_to_num_ips[:num_ases]], fontsize=font_size_ticks)
    plt.ylabel('Number of IP addresses', fontsize=font_size_label, labelpad=pad_ylabel)
    plt.xlabel('Autonomous system', fontsize=font_size_label, labelpad=pad_xlabel)
    plt.xticks(fontsize=font_size_ticks - 2, rotation=90)

    axes = plt.gca()
    axes.get_yaxis().set_major_formatter(plt.FuncFormatter(lambda x, loc: "{:,}".format(int(x))))
    plt.savefig("./results/top_ases_list.png", bbox_inches='tight')
    plt.close()

IP blocklist insights

We first have a look at the number of IP addresses included in the list over time. We see that since May 2022, it has stabilized between 60K and 80K IP addresses. The differences we observe over time can be explained mainly by the following reasons:

There are more or fewer attacks on my honeypots;
I used a different number of proxy providers over time;
I adjusted my proxy budget.

When it comes to the cumulative number of IP addresses in the list over time, i.e. in total, how many distinct IPs have been present in the blocklist at some point, we see that around 800,000 malicious IP addresses have been flagged in 8 months, which represents 100,000 distinct IP addresses monthly.

We can look at the CDF of the duration IP addresses were included in the list to understand whether or not IP addresses remain in the list for a long time. What we observe on the CDF graph below, is that around 65% of the IPs that were in the list got flagged only once since they appear at most 7 days in the list, which is the minimum duration an IP can stay in the blocklist if it’s flagged only once. Disclaimer, this doesn’t necessarily mean these IPs ceased to be malicious. Unfortunately, this has more to do with my data collection protocol. We can only infer that I didn’t observe any more malicious activity from these IPs. However, we also see that some IPs remain on the list for a longer period:

10.3% of the IPs have been included in the list for > 1 month;
4.5% of the IPs have been included in the list for > 1 month.

These IP addresses tend to come from the following autonomous systems:

M247 Ltd;
QUICKPACKET;
HostRoyale Technologies Pvt Ltd;
Digital Energy Technologies Ltd. These are data-center autonomous systems commonly used by bots. However, be careful if you wanted to completely block them, some of them, such as M247 are also used by several popular VPN providers.

The top 3 autonomous systems whose IP addresses are the most frequently flagged as proxies or used by bots are 1) Korea Telecom, 2) AS Coloam and 3) Chinanet. However, we also observe a long tail of data-center and residential autonomous systems from which originates a significant volume of malicious traffic, such as M247 Ltd, HKT Limited, and Telefonica Brasil.

If you are interested in more fine-grained data, I also generated a list of the top 450K IPs that stayed the longest in the list.

Conclusion

In this blog post, we showed how you can use Git-python and Matplotlib to analyze the Avastel all infected IPs 7d blocklist, a blocklist that contains recent IP addresses that have been flagged as malicious/used by bots.

We showed that over the last 8 months, we gathered around 800,000 malicious IP addresses in this blocklist, which represents 100,000 distinct malicious IPs monthly. While the majority of these IP addresses remain on the list for less than a week, we showed that a minority of IP addresses (~4.5%) originating from a few data-center autonomous systems (M247 Ltd, QUICKPACKET, HostRoyale Technologies Pvt Ltd, Digital Energy Technologies Ltd.) stayed in the list for more more than three months. Our study of the top autonomous systems linked to IPs present in the blocklist shows that these IPs come both from data-center and residential autonomous systems.

Random fact: I asked Stable diffusion AI model to generate an image that represents an "IP address used as a proxy by bad bots":

img = generator.generate(
    "IP address used as a proxy by bad bots",
    num_steps=10,
    unconditional_guidance_scale=7.5,
    temperature=1,
    batch_size=4,
)
pil_img = Image.fromarray(img[0])
display(pil_img)

I’m not convinced by its proposal, seems like we’ll have to find other approaches to detect proxies and bad bots until AI improves!

NodeJS/JavaScript: execute at most N tasks at the same time without external dependencies

Sat, 26 Feb 2022 00:00:00 +0000

As I explained in a previous blog post, I try to rely less and less on external dependencies when using NodeJS (for security and pedagocgical reasons).

In this blog post, I show how to create a simple Pool class to handle a pool of tasks that can be run in parallel, all while limiting the task concurrency, and maximizing the throughput. If you want a production-ready NodeJS package to do this, you can use Bluebird and its Promise.map function with the concurrency parameter.

Precision regarding the problem to solve

Note that what we do in this article is more optimal than simply calling Promise.all in a loop. Indeed, many solutions posted on StackOverflow suggest doing something that looks like the following code snippet:

const tasks = [t1, t2, ..., tN];
const MAX_CONCURRENCY = 5;

const resultTasks = [];
for (let i = 0; i < tasks.length; i += MAX_CONCURRENCY) {
    // Get the MAX_CONCURRENCY tasks to run
    const tasksToRun = tasks.slice(i, i + MAX_CONCURRENCY);

    const tasksPromises = [];
    for (let task of tasksToRun) {
        tasksPromises.push(task());
    }

    // Wait for all MAX_CONCURRENCY tasks to finish
    const res = await Promise.all(tasksPromises);
    resultTasks.push(res);
}

In the example above, we run tasks by batch of MAX_CONCURRENCY tasks. However, if we run 5 tasks in parallel and one of the tasks finishes before the others, then, we now only have 4 tasks running in parallel, even though there are still tasks waiting to be run in the tasks queue.

Indeed, Promise.all waits for all promises to succeed before it resolves. It means we don’t start any new tasks before all the 5 tasks have been completed (for loop iteration is stopped until all tasks promises resolve). Thus, at a given time our only guarantee is that we have <= 5 tasks running.

What we want is a solution where we maximize throughput, all while controlling the concurrency. At a given time, we want to have MAX_CONCURRENCY tasks running in parallel, AND NOT <= MAX_CONCURRENCY tasks.

The solution we present in the next section is slightly complex since it handles more than just concurrency management. The idea is to have a generic solution that enables us to run any kind of task and handle success/error. This could be useful for example to write scrapers where the notion of success/error may differ depending on the site targeted. For example, some websites may return 403 in case of blocking, while others may return a response 200 with a different text. Thus, having a generic solution enables us to maximize reusability across different projects.

Task pool manager with control of max concurrency in NodeJS/JavaScript

class Pool {

    // Our pool of tasks takes only 1 parameter: the max concurrency,
    // i.e. the maximum number of tasks executed at the same time
    constructor(concurrency) {
        this.tasks = [];
        this.concurrency = concurrency;
    }

    addTask(task) {
        this.tasks.push(task)
    }

    // Private class method
    async _executeTasks(iterator) {
        const results = [];
        // We leverage a shared iterator to control the maximum concurrency
        // For more details you can read this SO post:
        // https://stackoverflow.com/questions/40639432/what-is-the-best-way-to-limit-concurrency-when-using-es6s-promise-all
        // I used it as a starting point for the solution presented in the blog post
        for (let [_, task] of iterator) {
            try {
                // Run the task and await for it
                const res = await task.run();
                try {
                    // We defined a isSuccessful function that enables us to have different definition of success 
                    // based on the tasks we run.
                    // Thus, we can run heterogeneous tasks at the same time.
                    if (task.isSuccessful(res)) {
                        // If it's successfull, we call a callback function onSuccess.
                        // This function can be used for example to save the result to a database, 
                        // print something, or do nothing if not needed.
                        await task.onSuccess(res);
                        results.push(res);
                    } else {
                        // In case of error, we call a callback function onError.
                        // This function can be used for example to add the failed task to a queue of tasks to retry later.
                        await task.onError(res);
                    }
                } catch (e) {
                    await task.onError(res);
                }
            } catch (e) {
                task.onError(e);
            }

        }
        return results;
    }

    async run() {
        const iterator = this.tasks.entries();
        const tasksWorkers = new Array(this.concurrency).fill(iterator).map(this._executeTasks);
        const res = await Promise.allSettled(tasksWorkers);
        const flattenedArrays = [];
        res.forEach((subArray) => {
            if (subArray.value) {
                subArray.value.forEach(elt => flattenedArrays.push(elt))
            }
        })

        // We reset the list of tasks after we have run them
        // so that we can add new tasks to the pool and run only the newly added tasks
        this.tasks = [];
        return flattenedArrays;
    }
}

To use our Pool class, we need to define a set of task objects that have the following properties:

name: a string used as an identifier of the task/for description purposes.
run: an asynchronous function to execute the task.
isSuccessful: an asynchronous function that takes as input the result of the task (returned by the run function) and that must return a boolean indicating whether or not the task should be considered as successful or not.
onSuccess: an asynchronous function that takes as input the result of the task (returned by the run function) and is executed in case the task has been successful (as defined by isSuccessful).
onError: an asynchronous function that takes as input the result of the task (returned by the run function) and is executed in case the task has not been successful (as defined by isSuccessful).

(async () => {
    const MAX_CONCURRENCY = 5;
    const NUM_TASKS = 100;
    const p = new Pool(MAX_CONCURRENCY);


    const failedTasks = [];
    const successfullTasks = [];
    // Ccreate NUM_TASKS tasks and add them to the pool
    for (let i = 0; i < NUM_TASKS; i++) {
        p.addTask({
            name: `Task ${i}`,
            async run() {
                // A simple run function that displays the index of the task, waits 1s (to better observe parallelism) and
                // randomly throw an exception (to test error management)
                console.log(`Running task ${i}`);
                await sleep(1000);

                if (Math.random() > 0.7) {
                    throw new Error('TaskError');
                }

                return i;
            },
            async onSuccess(res) {
                console.log(`Run task ${i} successfuly, res = ${res}`);
                successfullTasks.push(this);
            },
            async onError(err) {
                console.log(`Task ${i} failed with error = ${err}`);
                failedTasks.push(this);
            },
            async isSuccessful(res) {
                return typeof res === 'number';
            }
        })
    }

    const tasksResult = await p.run();
    // contains results of successful tasks
    console.log(tasksResult);

    console.log(`Num failed tasks: ${failedTasks.length}`);
    console.log(`Num successful tasks: ${successfullTasks.length}`);

})();

The GIF below shows the execution of the code above (100 tasks with MAX_CONCURRENCY = 5). We see that tasks are not all executed at once. At a given time, 5 tasks are being executed, and whenever a task finishes, another task is automatically run without having to wait for the 4 others to complete:

NodeJS: how to route HTTPS requests through an HTTP proxy without any external dependencies (optimized version)

Sat, 26 Feb 2022 00:00:00 +0000

In a previous blog post I showed how to make HTTPs requests through HTTP proxies in NodeJS using no external dependencies. It’s part of a series of articles where I present how to do frequent NodeJS/JavaScript tasks using no/less external dependencies.

However, if you run the code of the previous blog post on more than > 10 URLs, you may notice that it takes a lot of time to make only a few HTTPS requests. The main reason is that for each HTTPS request you make, the getURL function establishes a new connection with the proxy, which is highly inefficient.

We use the program below to measure the time it takes to make 50 HTTPS requests through an HTTP proxy:

(async () => {
    const urls = new Array(50).fill('https://api.ipify.org/?format=json');

    for (const [i, url] of urls.entries()) {
        console.log(`${i + 1}/${urls.length}`);
        try {
            const res = await getURL(url);
            console.log(res.body);
        } catch (e) {
            console.log(e)
        }
    }
})();

In total, it takes ~4 min 48s to make the 50 HTTPS requests (5.8 s/request):

time node non-optimized.js
1/50
138.121.240.87
...
50/50
115.93.182.68

node non-optimized.js  0.70s user 0.09s system 0% cpu 4:48.24 total

Optimizing code to make HTTPs requests through HTTP proxies

This section presents how we can improve the code of the getURL function to make it faster. The goal is to stop establishing a connection with the HTTP proxy at each request since that’s what heavily slows down our code.

We define a class called HttpsWithProxyClient since we have to store information between different HTTPS requests:

proxyAgent: an https.Agent instance to maintain the connection with the proxy.
lastRequestHostname: to store the last host on which we made an HTTPS request. Indeed, if we make an HTTPs request to a different host, we have to reset the connection with the proxy.

var http = require("http");
var https = require('https');

class HttpsWithProxyClient {
    constructor(proxyConf) {
        this.proxyConf = proxyConf;
        this.proxyAgent = null;
        this.lastRequestHostname = null;
    }

    // Async private method, used only internally by HttpsWithProxyClient
    // Handle the connection with the HTTP proxy
    // In case the connection is successful, it returns a new http.Agent with keepAlive activated
    async _connectToProxy(url) {
        return new Promise((resolve, reject) => {
            const headers = {
                'Proxy-Authorization': 'Basic ' + Buffer.from(this.proxyConf.proxy_username + ':' + this.proxyConf.proxy_password).toString('base64')
            }

            const urlParsed = new URL(url);
            http.request({ // establishing a tunnel
                host: this.proxyConf.proxy_host,
                port: this.proxyConf.proxy_port,
                method: 'CONNECT',
                path: `${urlParsed.hostname}:443`,
                headers
            }).on('connect', (res, socket) => {
                if (res.statusCode === 200) {
                    resolve(new https.Agent({ socket: socket, keepAlive: true }));
                } else {
                    reject('Could not connect to proxy!')
                }

            }).on('error', (err) => {
                reject(err.message);
            }).on('timeout', (err) => {
                reject(err.message);
            }).end();
        });
    }

    // Public asynchronous method used to make an HTTPs get request on a given URL through an HTTP proxy
    async getURL(url, headers) {
        return new Promise(async (resolve, reject) => {
            const urlParsed = new URL(url);
            headers = headers || {};
            
            // If there's no current valid connection established with the proxy
            // or if the host linked to the URL requested is different from the previous request
            // -> recreate a connection with the proxy
            if (!this.proxyAgent || this.lastRequestHostname !== urlParsed.hostname) {
                try {
                    this.proxyAgent = await this._connectToProxy(url);
                    this.lastRequestHostname = urlParsed.hostname;
                } catch (e) {
                    return reject(e);
                }
                
            }

            const req = https.get({
                host: urlParsed.hostname,
                path: urlParsed.pathname,
                agent: this.proxyAgent,
                headers: headers,
            }, (response) => {
                const chunks = [];
                response.on('data', (chunk) => {
                    chunks.push(chunk);
                });
                response.on('end', () => {
                    resolve({
                        body:Buffer.concat(chunks).toString(),
                        headers: response.headers,
                        status: response.statusCode
                    })
                });

                response.on("error", (err) => {
                    reject(err);
                })

                response.setTimeout(25000, () => {
                    reject('timeout')
                })
            });

            req.on('error', (err) => {
                this.proxyAgent = null;
                reject(err.message);
            })
        })
    }
}

We leverage the HttpsWithProxyClient class we just created to measure the performance improvement compare to the naive implementation of getURL defined in the previous blog post:

(async () => {
    const conf = {
        "proxy_username": 'xxxxx',
        "proxy_password": 'xxxxxx',
        "proxy_host": 'xxxxx.com',
        "proxy_port": 8000
    }

    // Side effect is even if you have a rotating proxy, the IP will be stable
    const httpsClient = new HttpsWithProxyClient(conf);
    const urls = new Array(50).fill('https://api.ipify.org/?format=json');

    for (const [i, url] of urls.entries()) {
        console.log(`${i + 1}/${urls.length}`);
        try {
            const res = await httpsClient.getURL(url);
            console.log(res.body);
        } catch (e) {
            console.log(e)
        }
    }
})();

time node optimized.js 
1/50
115.163.214.177
...
50/50
194.15.57.236

node optimized.js  0.26s user 0.13s system 0% cpu 57.807 total

The new optimized version takes 57s to make 50 requests, against 4 min 48 s for the non-optimized version. Execution time has been divided by ~5!

However, there is one significant fonctional difference between the 2 programs: since we persist the connection with the proxy, the IP address will remain the same as long as the connection persists, EVEN IF you use an auto-rotating proxy. If you want the best of both worlds, you could modify the HttpsWithProxyClient class so that it resets the proxy connection every N request to obtain a new IP address.

NodeJS: how to route HTTPS requests through an HTTP proxy without any external dependencies

Sat, 19 Feb 2022 00:00:00 +0000

Why I try to rely less on external dependencies

These last years there has been an increasing number of supply chain attacks, particularly in the NodeJS ecosystem, cf Supply Chain Compromises Through Node.js Packages, NPM library used by Facebook and others was compromised where attackers compromise heavily popular NodeJS packages hosted on NPM to compromise other packages/devices. The compromise can occur using different techniques, such as merging a malicious commit that appear benign or stealing the maintainer’s NPM account by conducting a credential stuffing/brute-force attack on his account and then modifying the package code. Since most NPM packages rely on a lot of dependencies (Ride Down Into JavaScript Dependency Hell), compromising one heavily used package can result in thousands of compromised packages. For this reason (and also for the sake of learning lower-level details of NodeJS) I try to rely less on external NPM packages for my side projects (I didn’t appreciate having to fix one of my projects at 1 am because of ua-parser-js being compromised).

Making HTTPs requests through proxy in NodeJS without any non-native dependencies

One of the tasks I commonly do is making HTTPS requests through an HTTP proxy. Until recently, I heavily relied on external dependencies such as Axios, node-fetch, or requests. When I decided to try to do it without any non-native dependencies I was surprised by how difficult it was for something that (I think) should be straightforward. When I searched for help on StackOverflow/Github, most of the response were basically: “Use this 3rd party library” (e.g. https://stackoverflow.com/a/21281075) After being bored of testing the different code snippets to check if they were working with HTTPS URLs (which was not always the case) as well as with the latest NodeJS version (at that time 17.5.0), I decided to publish a solution (to save myself, and maybe others, sometime in the future). It is inspired by the following responses/comments I found online:

// You should have a variable conf that contains information about your proxies:
// {
//     "proxy_username": 'username',
//     "proxy_password": 'password',
//     "proxy_host": 'proxyhost.com',
//     "proxy_port": 8888
// }

// The only 2 dependencies we use are http and https but both are native to NodeJS
var http = require("http");
var https = require('https');

async function getURL(url, headers) {
    return new Promise((resolve, reject) => {
        const urlParsed = new URL(url);
        headers = headers || {};
        headers['Proxy-Authorization'] = 'Basic ' + Buffer.from(conf.proxy_username + ':' + conf.proxy_password).toString('base64');
        http.request({
            host: conf.proxy_host,
            port: conf.proxy_port,
            method: 'CONNECT',
            path: `${urlParsed.hostname}:443`,
            headers
        }).on('connect', (res, socket) => {
            // To avoid leaking the header since it could enable someone to detect you!
            delete headers['Proxy-Authorization']
            if (res.statusCode === 200) {
                const agent = new https.Agent({ socket });
                var req = https.get({
                    host: urlParsed.hostname,
                    path: urlParsed.pathname,
                    agent,
                    headers: headers
                }, (response) => {
                    const chunks = [];
                    response.on('data', (chunk) => {
                        chunks.push(chunk);
                    });
                    response.on('end', () => {
                        resolve({
                            body: Buffer.concat(chunks).toString(),
                            headers: response.headers,
                            status: response.statusCode
                        })
                    });

                    response.on("error", (err) => {
                        reject(err);
                    })

                    response.setTimeout(15000, () => {
                        reject('Timeout')
                    })
                });

                req.on('error', (err) => {
                    reject(err.message);
                })
            } else {
                reject('Could not connect to proxy!')
            }

        }).on('error', (err) => {
            reject(err.message);
        }).end();
    })

}

The code snippet below shows how we can call the getURL function that makes an HTTPS request routed through a proxy. It also shows how you can supply your own HTTP headers:

(async() => {
    const headers = {
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
        "accept-language": "fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7",
        "referrer": "https://www.google.com/",
        "sec-ch-ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"98\", \"Google Chrome\";v=\"98\"",
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": "\"macOS\"",
        "sec-fetch-dest": "document",
        "sec-fetch-mode": "navigate",
        "sec-fetch-site": "same-origin",
        "sec-fetch-user": "?1",
        "upgrade-insecure-requests": "1",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4858.102 Safari/537.36"
    }

    const res = await getURL('https://ipapi.co/json', headers);
    console.log(res.body);


})();

In my case, I obtained the following response where we see that the request is from Brazil, which proves that the request was properly routed through the HTTP proxy (I have a French IP otherwise):

{
    "ip": "177.85.130.114",
    "version": "IPv4",
    "city": "Arapiraca",
    "region": "Alagoas",
    "region_code": "AL",
    "country": "BR",
    "country_name": "Brazil",
    "country_code": "BR",
    "country_code_iso3": "BRA",
    "country_capital": "Brasilia",
    "country_tld": ".br",
    "continent_code": "SA",
    "in_eu": false,
    "postal": "57300",
    "latitude": -9.7438,
    "longitude": -36.5931,
    "timezone": "America/Maceio",
    "utc_offset": "-0300",
    "country_calling_code": "+55",
    "currency": "BRL",
    "currency_name": "Real",
    "languages": "pt-BR,es,en,fr",
    "country_area": 8511965.0,
    "country_population": 209469333,
    "asn": "AS262608",
    "org": "Vapt Solucoes Tecnologicas Ltda"
}

Feel free to contact me by email/on Twitter if you have a better solution that you would like to share!

Yet another bot IPs blocklist: longtime bot IPs

Tue, 02 Nov 2021 00:00:00 +0000

I recently published 4 blocklists that essentially focus on data center IP ranges.

However, more and more attackers operate from residential IPs. That’s why I’ve decided to publish another blocking list that focuses on IP addresses that have been used by bots for a long period. Because of the way this list is built, it contains both data center IP ranges, as well as residential IPs.

The list is available at the following URL: https://antoinevastel.com/data/avastel-longtime-bot-ips.txt

How is the list built?

The long time infected IPs blocklist is built daily using the last 10 weeks of data collected by the malicious bot IPs API. For each IP address, we leverage the following information:

Number of events where an IP address was used by a bot;
Timespan between the first and last bot detection events.

IPs are included in the list if they meet the following criteria:

Timespan (duration between first and last bot detection event) is > 15 days;
Ratio between number of events/time span is > 0.2. This second criterion helps to ensure that we have enough events to take a significant decision. It avoids overestimating the lifetime of an IP address.

False-positive disclaimer

Even though residential IPs present in this list have been used by bots for > 15 days, they may still be shared with legitimate humans. To lower the false-positive risk, you should adjust the blocking decision based on other criteria than the IP address.

Bot IPs blocklists

Sun, 31 Oct 2021 00:00:00 +0000

It’s been ~2 months since I started to gather data related to malicious bot IPs. With the current version of the bot IPs API, it’s possible to obtain information about recent bot IPs or to test a particular IP address. However, it may not be the best format if you want to quickly and safely block malicious traffic in production.

To make it easier for people looking for a simple solution to block malicious traffic, I’ve decided to publish 4 IP blocklists linked to bad bots. Because of the way they’re built (more details after) and the fact these lists aim to block traffic solely based on their IP address, these lists are only constituted of data center IPs. Indeed, it’s not advised to block residential IPs solely based on the fact they’ve been used by bots/as a proxy recently since they are often shared with legitimate human users.

For conciseness, I publish IPs to block as /24 IP ranges (256 IPs). For the moment, the lists don’t contain individual IPs to block. Nevertheless, you can build your own list using data from the bot IPs API.

4 (+1) bot IP addresses blocklists:

I’ve built 5 blocking lists derived from the same data:

Avastel block ultra safe: contains /24 data center IP ranges that I consider ultra safe to block in production.
Avastel block safe: contains /24 data center IP ranges that I consider safe to block in production.
Avastel block moderate: contains /24 data center IP ranges that I consider moderately safe to block in production.
Avastel block aggressive: contains /24 data center IP ranges that I consider a bit risky to block in production.
New: Avastel all infected IPs last 7 days : contains all IP addresses that have been used by bots/proxies in the last 7 days.

How are these lists built?

The description applies to all list except “Avastel all infected IPs last 7 days” since this list is constituted of all bots/proxies IPs of the last 7 days.

The other lists are built daily using the last month of data collected by the malicious bot IPs API. For the moment, the strategy is simple:

Group all IPv4 addresses by their first 3 bytes.
Count the number of IPv4 addresses in each group, i.e. number of IPs per /24 range.
For each /24 IP range, if a significant fraction of the 256 IPs is flagged as malicious by the bot IPs API, then flag the whole /24 range as malicious.

The minimum fraction of bad bot IPs in a /24 IP range (step 3) influences the generated list. The lower the fraction of bad bot IPs, the greater the number of IP ranges to block. However, decreasing this number also increases the chance of wrongly flagging a whole /24 IP range.

Current bad bot fraction for the different lists:

Avastel block ultra safe: > 83% of known bad bot IPs.
Avastel block safe: > 73% of known bad bot IPs.
Avastel block moderate: > 63% of known bad bot IPs.
Avastel block aggressive: > 50% of known bad bot IPs.

Even for the most aggressive list, I consider these values to be safe since they’re computed on data center IPs only, but I’ll let you make yourself an opinion on real-world traffic.

Bots IPs API: documentation

Sun, 26 Sep 2021 00:00:00 +0000

API overview

I provide a free API to get information about IP addresses used by bots. Feel free to use it in any of your projects.

False positives: I try to keep a low false-positive rate by using heuristics for which I have great certainty the IP has been used by a bot. Except for Tor exit nodes, I DON’T crawl any malicious IP list. All the IPs flagged as bots are obtained through several techniques that enables me to be sure about the fact they were/are used by bots (proxies, honeypots). However, keep in mind that some of these IPs may be also shared by legitimate human users.

API routes

For the moment, the API has 2 routes:

Get 5k recent bots IPs: GET https://antoinevastel.com/bots/ips;
Check an IP address: GET https://antoinevastel.com/bots/ip/ip-you-want-to-check (replace ip-you-want-to-check with the IP address you want to check).

GET /bots/ips

An API call returns JSON content representing an array of objects linked to malicious IPs. It returns at most 5000 IPs. If there are more malicious IPs, it returns the latest malicious IPs detected. Each object of the JSON array has the following properties:

ip: malicious IP address
time: date representing when the IP was flagged as malicious
autonomousSystemOrganization: name of the IP autonomous system (enriched using Maxmind)
autonomousSystemNumber: autonomous system number linked to the IP (enriched using Maxmind)
country: country of the IP address (enriched using Maxmind)

There may be duplicates if the same IP was flagged multiple times.

GET /bots/ip/ip-you-want-to-check

For a given IP, an API call returns JSON content representing information about the IP requested.

{
  "matched": true,
  "ip": "91.202.133.37",
  "autonomousSystemOrganization": "SpaceNet LLC",
  "autonomousSystemNumber": 44686,
  "country": "UA",
  "events": [
    1629553470430
  ]
}

In case the IP address is present in the database, the matched field is set to true. Otherwise, it’s set to false. In all cases (even when the IP is not present in our database), we always return information about the IP:

ip: value of the requested IP,
autonomousSystemOrganization: name of the IP autonomous system (enriched using Maxmind)
autonomousSystemNumber: autonomous system number linked to the IP (enriched using Maxmind)
country: country of the IP address (enriched using Maxmind)
events: an array that contains a list of dates where the IP was flagged as malicious. In case the IP is not present in our database, it is an empty array

POST /bots/ips/

Endpoint to obtain information about multiple IP addresses at once. Output is similar to GET /bots/ip/ip-you-want-to-check but instead of returning information about a single IP address, it returns an array corresponding to all the IP addresses passed in the POST request payload.

Body of the POST request must be valid JSON with an ips property corresponding to the array of IP addresses to test, for example:

{
    "ips": ["185.82.126.222", "202.74.73.51", "202.74.73.53"]
}

For a given list of IPs, an API call returns JSON content representing information about the IPs requested. For example, for the 3 IPs above, we obtain:

[
    {
        "matched": false,
        "ip": "202.74.73.51",
        "events": [],
        "autonomousSystemOrganization": "PT Global Port Binekatara",
        "autonomousSystemNumber": 24522,
        "country": "ID"
    },
    {
        "matched": false,
        "ip": "202.74.73.53",
        "events": [],
        "autonomousSystemOrganization": "PT Global Port Binekatara",
        "autonomousSystemNumber": 24522,
        "country": "ID"
    },
    {
        "matched": true,
        "ip": "185.82.126.222",
        "autonomousSystemOrganization": "Sia Nano IT",
        "autonomousSystemNumber": 52173,
        "country": "LV",
        "events": [
            "2021-10-02T00:00:20.008Z",
            "2021-10-02T12:00:13.432Z",
            ...
            "2021-10-16T00:00:13.588Z",
            "2021-10-16T12:00:11.173Z"
        ]
    }
]

Warning 1: To manage load on my server, for the moment it’s not possible to provide more than 100 IPs per POST request. Warning 2: This API can be costly in terme of computation, please rate limit yourself responsibly if you need to parallelize/make many requests. Feel free to contact me if you have special requests regarding the use of the API.

Statistics IP addresses API

You can find statistics about the type of IPs present in the API database:

Reason the IP is flag: (residential) proxy, vulnerability scanning, Tor exit nodes;
Top autonomous systems;
Top countries.

New feature to malicious IPs API: check an IP

Tue, 24 Aug 2021 00:00:00 +0000

As promised in the previous blog post, I added new data sources to find more malicious IPs. I also added a new endpoint to the malicious IPs API: https://antoinevastel.com/bots/ip/ip-you-want-to-check

To make an API call on this endpoint, simply call /bots/ips/ip-you-want-to-check where you should replace ip-you-want-to-check with the IP address you want to check.

You can call it using Curl or any HTTP request library you want:

curl https://antoinevastel.com/bots/ip/157.100.36.194

Data returned by the API

For a given IP, an API call returns JSON content representing information about the IP requested.

{
  "matched": true,
  "ip": "91.202.133.37",
  "autonomousSystemOrganization": "SpaceNet LLC",
  "autonomousSystemNumber": 44686,
  "country": "UA",
  "events": [
    1629553470430
  ]
}

ip: value of the requested IP,
autonomousSystemOrganization: name of the IP autonomous system (enriched using Maxmind)
autonomousSystemNumber: autonomous system number linked to the IP (enriched using Maxmind)
country: country of the IP address (enriched using Maxmind)
events: an array that contains a list of timestamps where the IP was flagged as malicious. In case the IP is not present in our database, it is an empty array

DISCLAIMER: Please try to be responsible when making API calls, rate limit your requests.