Reusing Hardware Performance Counters to Detect and Identify Kernel Control-Flow Modifying Rootkits

Xueyang Wang, Ramesh Karri

Research output: Contribution to journalArticle

Abstract

Kernel rootkits are formidable threats to computer systems. They are stealthy and can have unrestricted access to system resources. This paper presents NumChecker, a new virtual machine (VM) monitor based framework to detect and identify control-flow modifying kernel rootkits in a guest VM. NumChecker detects and identifies malicious modifications to a system call in the guest VM by measuring the number of certain hardware events that occur during the system call's execution. To automatically count these events, NumChecker leverages the hardware performance counters (HPCs), which exist in modern processors. By using HPCs, the checking cost is significantly reduced and the tamper-resistance is enhanced. We implement a prototype of NumChecker on Linux with the kernel-based VM. An HPC-based two-phase kernel rootkit detection and identification technique is presented and evaluated on a number of real-world kernel rootkits. The results demonstrate its practicality and effectiveness.

Original languageEnglish (US)
Article number7229276
Pages (from-to)485-498
Number of pages14
JournalIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
Volume35
Issue number3
DOIs
StatePublished - Mar 1 2016

Fingerprint

Flow control
Hardware
Computer hardware
Computer systems
Virtual machine
Malware
Costs

Keywords

  • Controlflow Modifying Kernel Rootkits
  • Hardware Performance Counters
  • Rootkit Detection and Identification
  • Virtualization

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Computer Graphics and Computer-Aided Design
  • Software

Cite this

@article{77625aec98024c8683d0165a3a0d33ca,
title = "Reusing Hardware Performance Counters to Detect and Identify Kernel Control-Flow Modifying Rootkits",
abstract = "Kernel rootkits are formidable threats to computer systems. They are stealthy and can have unrestricted access to system resources. This paper presents NumChecker, a new virtual machine (VM) monitor based framework to detect and identify control-flow modifying kernel rootkits in a guest VM. NumChecker detects and identifies malicious modifications to a system call in the guest VM by measuring the number of certain hardware events that occur during the system call's execution. To automatically count these events, NumChecker leverages the hardware performance counters (HPCs), which exist in modern processors. By using HPCs, the checking cost is significantly reduced and the tamper-resistance is enhanced. We implement a prototype of NumChecker on Linux with the kernel-based VM. An HPC-based two-phase kernel rootkit detection and identification technique is presented and evaluated on a number of real-world kernel rootkits. The results demonstrate its practicality and effectiveness.",
keywords = "Controlflow Modifying Kernel Rootkits, Hardware Performance Counters, Rootkit Detection and Identification, Virtualization",
author = "Xueyang Wang and Ramesh Karri",
year = "2016",
month = "3",
day = "1",
doi = "10.1109/TCAD.2015.2474374",
language = "English (US)",
volume = "35",
pages = "485--498",
journal = "IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems",
issn = "0278-0070",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "3",

}

TY - JOUR

T1 - Reusing Hardware Performance Counters to Detect and Identify Kernel Control-Flow Modifying Rootkits

AU - Wang, Xueyang

AU - Karri, Ramesh

PY - 2016/3/1

Y1 - 2016/3/1

N2 - Kernel rootkits are formidable threats to computer systems. They are stealthy and can have unrestricted access to system resources. This paper presents NumChecker, a new virtual machine (VM) monitor based framework to detect and identify control-flow modifying kernel rootkits in a guest VM. NumChecker detects and identifies malicious modifications to a system call in the guest VM by measuring the number of certain hardware events that occur during the system call's execution. To automatically count these events, NumChecker leverages the hardware performance counters (HPCs), which exist in modern processors. By using HPCs, the checking cost is significantly reduced and the tamper-resistance is enhanced. We implement a prototype of NumChecker on Linux with the kernel-based VM. An HPC-based two-phase kernel rootkit detection and identification technique is presented and evaluated on a number of real-world kernel rootkits. The results demonstrate its practicality and effectiveness.

AB - Kernel rootkits are formidable threats to computer systems. They are stealthy and can have unrestricted access to system resources. This paper presents NumChecker, a new virtual machine (VM) monitor based framework to detect and identify control-flow modifying kernel rootkits in a guest VM. NumChecker detects and identifies malicious modifications to a system call in the guest VM by measuring the number of certain hardware events that occur during the system call's execution. To automatically count these events, NumChecker leverages the hardware performance counters (HPCs), which exist in modern processors. By using HPCs, the checking cost is significantly reduced and the tamper-resistance is enhanced. We implement a prototype of NumChecker on Linux with the kernel-based VM. An HPC-based two-phase kernel rootkit detection and identification technique is presented and evaluated on a number of real-world kernel rootkits. The results demonstrate its practicality and effectiveness.

KW - Controlflow Modifying Kernel Rootkits

KW - Hardware Performance Counters

KW - Rootkit Detection and Identification

KW - Virtualization

UR - http://www.scopus.com/inward/record.url?scp=84963820176&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84963820176&partnerID=8YFLogxK

U2 - 10.1109/TCAD.2015.2474374

DO - 10.1109/TCAD.2015.2474374

M3 - Article

AN - SCOPUS:84963820176

VL - 35

SP - 485

EP - 498

JO - IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems

JF - IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems

SN - 0278-0070

IS - 3

M1 - 7229276

ER -