Headless Hashcat Server

While cracking passwords as a hobby for over a decade I’ve been continually upgrading a rig dedicated to the task. In its current configuration Cpt-Crunch (no affiliation to the cereal) is a 4-GPU “headless” server which I access via SSH to run various cracking activities. This is my step-by-step guide for installing the OS, configuring the GPU’s, installing and testing Hashcat and John the Ripper.

Hardware

Motherboard: Supermicro SYS-7048GR-TR
Memory 64GB
Storage 110GB SSD (OS)
500GB SSD (/data)
GPUs: (2) GTX 1080 FE
(2) GTX 1070 FE

Install the OS

UPDATED 2019-10-31: This guide now uses Ubuntu 18.04.3-server-amd64 as the base OS.

NOTE: I ran into an issue booting the ubuntu-18.04.3-live-server-amd64.iso image as a vitual CD from my Zalman VE300. My workaround was to create a bootable USB:
sudo dd bs=4M if=ubuntu-18.04.3-live-server-amd64.iso of=/dev/sdd status=progress oflag=sync

Perform a standard installation and choose to install the OpenSSH server when prompted. No additional packages are needed at this time. Once the install has completed SSH into the server and update the OS to the latest packages:

$ sudo apt update
$ sudo apt upgrade
$ sudo apt dist-upgrade

Next, install the required packages:

$ sudo apt-get install xserver-xorg-core xdm xserver-xorg-dev libgtk-3-0 libxnvctrl-dev p7zip-full pkg-config gcc --no-install-recommends

Disable the Nouveau drivers:

$ printf 'blacklist nouveau\noptions nouveau modeset=0\n' | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
$ sudo update-initramfs -u

Perform a reboot to pick-up the current set of changes:

$ sudo reboot

Install NVidia Drivers

Stop the X display manager:

$ sudo systemctl stop xdm

Check for the latest driver at http://www.nvidia.com/Download/Find.aspx, the link below was valid as of Oct 31, 2019.

$ wget http://us.download.nvidia.com/XFree86/Linux-x86_64/435.21/NVIDIA-Linux-x86_64-435.21.run

Make the installer executable and launch:

$ chmod +x NVIDIA-Linux-x86_64-435.21.run && sudo ./NVIDIA-Linux-x86_64-435.21.run
  • Select OK to 32-bit library warning.
  • Select Install and overwrite existing files when notified of, an incomplete installation of libglvnd was found
  • Select NO when asked to run nvidia-xconfig
  • Select OK to complete

Generate an Xorg config that enables GPU fan control:

$ sudo nvidia-xconfig -s -a --force-generate --allow-empty-initial-configuration \
--cool-bits=4 --registry-dwords="PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefaultAC=0x1" --no-sli \
--connected-monitor="DFP-0" -o /etc/X11/xorg.conf

Add Xorg configuration options to disable the NVidia driver watchdog.

$ sudo sh -c "sed -i '/^[[:blank:]]*Driver[[:blank:]]*\"nvidia/a\ \ \ \ Option         \"Interactive\" \"False\"' /etc/X11/xorg.conf"

Add settings required to control the GPU fans from terminal sessions (e.g SSH)

$ sudo sh -c "cat >> /etc/X11/xdm/Xsetup << EOF
export PATH=/bin:/usr/bin:/sbin
export HOME=/root
export DISPLAY=:0
xset -dpms
xset s off
xhost +
EOF"

Set the DISPLAY environment variable during login:

$ echo "export DISPLAY=:0" >> ~/.bashrc

Now one last reboot to apply all the changes:

$ sudo reboot

When the reboot completes SSH back in and verify everything works:

$ nvidia-smi --query-gpu=gpu_name --format=csv,noheader

GeForce GTX 1080
GeForce GTX 1070
GeForce GTX 1080
GeForce GTX 1070

$ nvidia-settings -tq GPUCurrentFanSpeed

27
27
28
27

NOTE: The gpu-temp-retain option was removed from hashcat in v4.1.0 so the NVIDIA tools are used to manually set fan speeds.

To verify GPU fan control via NVIDIA drivers:

$ nvidia-settings -a GPUFanControlState=1

  Attribute 'GPUFanControlState' (cpt-crunch:0[gpu:0]) assigned value 1.
  Attribute 'GPUFanControlState' (cpt-crunch:0[gpu:1]) assigned value 1.
  Attribute 'GPUFanControlState' (cpt-crunch:0[gpu:2]) assigned value 1.
  Attribute 'GPUFanControlState' (cpt-crunch:0[gpu:3]) assigned value 1.
  
$ nvidia-settings -a GPUTargetFanSpeed=75

  Attribute 'GPUTargetFanSpeed' (cpt-crunch:0[fan:0]) assigned value 75.
  Attribute 'GPUTargetFanSpeed' (cpt-crunch:0[fan:1]) assigned value 75.
  Attribute 'GPUTargetFanSpeed' (cpt-crunch:0[fan:2]) assigned value 75.
  Attribute 'GPUTargetFanSpeed' (cpt-crunch:0[fan:3]) assigned value 75.
  
$ nvidia-settings -tq GPUCurrentFanSpeed

75
75
75
75

Install Hashcat

Download and unpack:

$ wget https://hashcat.net/files/hashcat-5.1.0.7z
$ 7z x hashcat-5.1.0.7z

Verify that Hashcat recognizes the video cards:

$ ./hashcat-5.1.0/hashcat64.bin -I

hashcat (v4.2.1) starting...

OpenCL Info:

Platform ID #1
  Vendor  : NVIDIA Corporation
  Name    : NVIDIA CUDA
  Version : OpenCL 1.2 CUDA 10.1.0

  Device ID #1
    Type           : GPU
    Vendor ID      : 32
    Vendor         : NVIDIA Corporation
    Name           : GeForce GTX 1080
    Version        : OpenCL 1.2 CUDA
    Processor(s)   : 20
    Clock          : 1733
    Memory         : 2029/8119 MB allocatable
    OpenCL Version : OpenCL C 1.2
    Driver Version : 435.21
    <SNIP>

Run a simple benchmark to verify Hashcat is operational:

$ ./hashcat-5.1.0/hashcat64.bin -m1000 -b
OpenCL Platform #1: NVIDIA Corporation
======================================
* Device #1: GeForce GTX 1080, 2029/8119 MB allocatable, 20MCU
* Device #2: GeForce GTX 1080, 2029/8119 MB allocatable, 20MCU
* Device #3: GeForce GTX 1070, 2029/8119 MB allocatable, 15MCU
* Device #4: GeForce GTX 1070, 2029/8119 MB allocatable, 15MCU

Benchmark relevant options:
===========================
* --optimized-kernel-enable

Hashmode: 1000 - NTLM

Speed.#1.........: 40089.5 MH/s (33.09ms) @ Accel:256 Loops:1024 Thr:256 Vec:2
Speed.#2.........: 40124.1 MH/s (33.06ms) @ Accel:256 Loops:1024 Thr:256 Vec:2
Speed.#3.........: 30201.4 MH/s (33.14ms) @ Accel:256 Loops:1024 Thr:256 Vec:2
Speed.#4.........: 30185.4 MH/s (33.17ms) @ Accel:256 Loops:1024 Thr:256 Vec:2
Speed.#*.........:   140.6 GH/s

Note to Self: The benchmarks show a drop in performance with NVIDIA Driver v4xx.00 when compared to v3xx.00. I need to perform some benchmarks under a real-world load to verifiy.

Speed.Dev.#1.....: 42997.9 MH/s (60.58ms) @ Accel:128 Loops:1024 Thr:1024 Vec:2
Speed.Dev.#2.....: 43085.5 MH/s (60.47ms) @ Accel:128 Loops:1024 Thr:1024 Vec:2
Speed.Dev.#3.....: 32656.8 MH/s (60.92ms) @ Accel:256 Loops:512 Thr:1024 Vec:2
Speed.Dev.#4.....: 32793.6 MH/s (60.69ms) @ Accel:256 Loops:512 Thr:1024 Vec:2
Speed.Dev.#*.....:   151.5 GH/s

(Optional) Install OpenCL runtime for CPU cracking

$ sudo apt-get install lsb-core --no-install-recommends

Download the latest OpenCL Runtime from Intel’s Developer Zone (v18.1.0.015 as of Oct 31, 2019). The download link (requires registration with Intel) is located about halfway down the page under the Intel® CPU Runtime for OpenCL™ Applications 18.1 for Linux* OS (64bit only) heading.

Unpack and run the installer:

$ tar zxf l_opencl_p_18.1.0.015.tgz
$ cd l_opencl_p_18.1.0.015/
$ sed 's/=decline/=accept/g' -i silent.cfg
$ echo 'INTEL_SW_IMPROVEMENT_PROGRAM_CONSENT=no' >> silent.cfg
$ sudo ./install.sh -s silent.cfg
$ ./hashcat-5.1.0/hashcat64.bin -I
hashcat (v5.1.0) starting...

OpenCL Info:

Platform ID #1
  Vendor  : Intel(R) Corporation
  Name    : Intel(R) CPU Runtime for OpenCL(TM) Applications
  Version : OpenCL 2.1 LINUX

  Device ID #1
    Type           : CPU
    Vendor ID      : 8
    Vendor         : Intel(R) Corporation
    Name           : Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
    Version        : OpenCL 2.1 (Build 0)
    Processor(s)   : 24
    Clock          : 2400
    Memory         : 16075/64301 MB allocatable
    OpenCL Version : OpenCL C 2.0
    Driver Version : 18.1.0.0920

(Optional) Install John the Ripper (with OpenCL support)

$ sudo apt install libssl-dev ocl-icd-opencl-dev opencl-headers pocl-opencl-icd
$ git clone -b bleeding-jumbo --single-branch https://github.com/magnumripper/JohnTheRipper.git
$ cd JohnTheRipper/src
$ ./configure
$ make -s clean && make -sj4
$ sudo cp -r ../run /opt/john

List the devices that John has detected:

$ /opt/john/john --list=opencl-devices | grep ") name:"
    Device #0 (1) name:     pthread-Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
    Device #0 (2) name:     Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
    Device #0 (3) name:     GeForce GTX 1080
    Device #1 (4) name:     GeForce GTX 1080
    Device #2 (5) name:     GeForce GTX 1070
    Device #3 (6) name:     GeForce GTX 1070

Updated:

Leave a comment