Dave Crawford

Dave Crawford

Headless Hashcat Server

While cracking passwords as a hobby for over a decade I've been continually upgrading a rig dedicated to the task. In its current configuration Cpt-Crunch (no affiliation to the cereal) is a 4-GPU "headless" server which I access via SSH to run various cracking activities. This is my step-by-step guide for installing the OS, configuring the GPU's, installing and testing Hashcat and John the Ripper.

Hardware

Motherboard: Supermicro SYS-7048GR-TR
Memory 64GB
Storage 110GB SSD (OS)
4TB disk(/data)
GPUs: (2) GTX 1080 FE
(2) GTX 1070 FE

Install the OS

UPDATED 2020-08-29: This guide now uses Ubuntu 20.04.1-server-amd64 as the base OS.

NOTE: I ran into an issue booting the ubuntu-18.04.3-live-server-amd64.iso image as a vitual CD from my Zalman VE300. My workaround was to create a bootable USB:
sudo dd bs=4M if=ubuntu-18.04.3-live-server-amd64.iso of=/dev/sdd status=progress oflag=sync

Perform a standard installation and choose to install the OpenSSH server when prompted. No additional packages are needed at this time. Once the install has completed SSH into the server and update the OS to the latest packages:

NOTE: The boot sequence may freeze at: [ OK ] Reached target Cloud-init target but the SSH service is available

Disable Cloud-Init to address the booting issue

$ sudo touch /etc/cloud/cloud-init.disabled

Remove the Unattended Upgrade tools (I learned the hard way when a kernel update hosed the NVIDIA drivers)

$ sudo apt remove unattended-upgrades

$ sudo apt update
$ sudo apt upgrade
$ sudo apt dist-upgrade

Next, install the required packages:

$ sudo apt install xserver-xorg-core xdm xserver-xorg-dev libgtk-3-0 libxnvctrl-dev p7zip-full pkg-config make gcc libglvnd-dev at-spi2-core nvidia-cuda-toolkit --no-install-recommends

Disable the Nouveau drivers:

$ printf 'blacklist nouveau\noptions nouveau modeset=0\n' | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
$ sudo update-initramfs -u

Perform a reboot to pick-up the current set of changes:

$ sudo reboot

Install NVidia GPU Drivers

Stop the X display manager:

$ sudo systemctl stop xdm

Check for the latest driver at http://www.nvidia.com/Download/Find.aspx, the link below was valid as of Aug 19, 2020.

$ wget http://us.download.nvidia.com/XFree86/Linux-x86_64/450.66/NVIDIA-Linux-x86_64-450.66.run
$ sudo sh NVIDIA-Linux-x86_64-450.66.run
  • Select NO when asked to install 32-bit compatibility libraries
  • Select NO when asked to run nvidia-xconfig
  • Select OK to complete

Generate an Xorg config that enables GPU fan control:

$ sudo nvidia-xconfig -s -a --force-generate --allow-empty-initial-configuration \
--cool-bits=4 --registry-dwords="PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefaultAC=0x1" --no-sli \
--connected-monitor="DFP-0" -o /etc/X11/xorg.conf

Add Xorg configuration options to disable the NVidia driver watchdog.

$ sudo sh -c "sed -i '/^[[:blank:]]*Driver[[:blank:]]*\"nvidia/a\ \ \ \ Option         \"Interactive\" \"False\"' /etc/X11/xorg.conf"

Add settings required to control the GPU fans from terminal sessions (e.g SSH)

$ sudo sh -c "cat >> /etc/X11/xdm/Xsetup << EOF
export PATH=/bin:/usr/bin:/sbin
export HOME=/root
export DISPLAY=:0
xset -dpms
xset s off
xhost +
EOF"

Set the DISPLAY environment variable during login:

$ echo "export DISPLAY=:0" >> ~/.bashrc

Now one last reboot to apply all the changes:

$ sudo reboot

When the reboot completes SSH back in and verify everything works:

$ nvidia-smi --query-gpu=gpu_name --format=csv,noheader

GeForce GTX 1080
GeForce GTX 1070
GeForce GTX 1080
GeForce GTX 1070

$ nvidia-settings -tq GPUCurrentFanSpeed

27
27
28
27

NOTE: The gpu-temp-retain option was removed from hashcat in v4.1.0 so the NVIDIA tools are used to manually set fan speeds.

To verify GPU fan control via NVIDIA drivers:

$ nvidia-settings -a GPUFanControlState=1

  Attribute 'GPUFanControlState' (cpt-crunch:0[gpu:0]) assigned value 1.
  Attribute 'GPUFanControlState' (cpt-crunch:0[gpu:1]) assigned value 1.
  Attribute 'GPUFanControlState' (cpt-crunch:0[gpu:2]) assigned value 1.
  Attribute 'GPUFanControlState' (cpt-crunch:0[gpu:3]) assigned value 1.
  
$ nvidia-settings -a GPUTargetFanSpeed=75

  Attribute 'GPUTargetFanSpeed' (cpt-crunch:0[fan:0]) assigned value 75.
  Attribute 'GPUTargetFanSpeed' (cpt-crunch:0[fan:1]) assigned value 75.
  Attribute 'GPUTargetFanSpeed' (cpt-crunch:0[fan:2]) assigned value 75.
  Attribute 'GPUTargetFanSpeed' (cpt-crunch:0[fan:3]) assigned value 75.
  
$ nvidia-settings -tq GPUCurrentFanSpeed

75
75
75
75

Install Hashcat

Check for the latest version at https://hashcat.net/hashcat/, the link below was valid as of Aug 19, 2020.

Download and unpack:

$ wget https://hashcat.net/files/hashcat-6.1.1.7z
$ 7z x hashcat-6.1.1.7z

Verify that Hashcat recognizes the video cards:

$ ./hashcat-6.1.1/hashcat.bin -I --quiet

CUDA Info:
==========

CUDA.Version.: 11.0

Backend Device ID #1 (Alias: #5)
  Name...........: GeForce GTX 1080
  Processor(s)...: 20
  Clock..........: 1733
  Memory.Total...: 8119 MB
  Memory.Free....: 8001 MB

<SNIP>

OpenCL Info:
============

OpenCL Platform ID #1
  Vendor..: NVIDIA Corporation
  Name....: NVIDIA CUDA
  Version.: OpenCL 1.2 CUDA 11.0.228

  Backend Device ID #5 (Alias: #1)
    Type...........: GPU
    Vendor.ID......: 32
    Vendor.........: NVIDIA Corporation
    Name...........: GeForce GTX 1080
    Version........: OpenCL 1.2 CUDA
    Processor(s)...: 20
    Clock..........: 1733
    Memory.Total...: 8119 MB (limited to 2029 MB allocatable in one block)
    Memory.Free....: 7936 MB
    OpenCL.Version.: OpenCL C 1.2
    Driver.Version.: 450.66

Run a simple benchmark to verify Hashcat is operational:

$ ./hashcat-6.1.1/hashcat.bin -m1000 -b --quiet

Hashmode: 1000 - NTLM

Speed.#1.........: 46449.6 MH/s (28.78ms) @ Accel:64 Loops:1024 Thr:1024 Vec:8
Speed.#2.........: 46580.0 MH/s (28.72ms) @ Accel:64 Loops:1024 Thr:1024 Vec:8
Speed.#3.........: 34670.1 MH/s (28.95ms) @ Accel:64 Loops:1024 Thr:1024 Vec:8
Speed.#4.........: 34691.1 MH/s (28.93ms) @ Accel:64 Loops:1024 Thr:1024 Vec:8
Speed.#*.........:   162.4 GH/s

Started: Wed Aug 19 21:23:54 2020
Stopped: Wed Aug 19 21:24:02 2020

(Optional) Install John the Ripper (with OpenCL support)

$ sudo apt install libssl-dev ocl-icd-opencl-dev opencl-headers pocl-opencl-icd
$ git clone -b bleeding-jumbo --single-branch https://github.com/magnumripper/JohnTheRipper.git
$ cd JohnTheRipper/src
$ ./configure
$ make -s clean && make -sj4
$ sudo cp -r ../run /opt/john

List the devices that John has detected:

$ ../run/john --list=opencl-devices | grep ") name:"

    Device #0 (1) name:     GeForce GTX 1080
    Device #1 (2) name:     GeForce GTX 1080
    Device #2 (3) name:     GeForce GTX 1070
    Device #3 (4) name:     GeForce GTX 1070

Updated:

Leave a comment