Detection Methodologies

• Signature-based
• Anomaly-based
• Stateful protocol analysis

IDPS technologies use many methodologies to detect incidents. Most IDPS technologies use multiple detection methodologies, either separately or integrated, to provide more broad and accurate detection.

Signature-based

A signature is a pattern that corresponds to a known threat. Signature-based detection is the process of comparing signatures against observed events to identify possible incidents.

• A telnet attempt with a username of “root”, which is a violation of an organization’s security policy
• An e-mail with a subject of “Free pictures!” and an attachment filename of “freepics.exe”, which are characteristics of a known form of malware
• An operating system log entry with a status code value of 645, which indicates that the host’s auditing has been disabled.

Anomaly-based

Anomaly-based detection is the process of comparing definitions of what activity is considered normal against observed events to identify significant deviations.

An IDPS using anomaly-based detection has profiles that represent the normal behavior of such things as users, hosts, network connections, or applications. The profiles are developed by monitoring the characteristics of typical activity over a period of time. An initial profile is generated over a period of time (typically days, sometimes weeks) sometimes called a training period.

Profiles for anomaly-based detection can either be static or dynamic.

• A static profile will eventually become inaccurate, so it needs to be regenerated periodically.
• A dynamic profile is adjusted constantly as additional events are observed.

Stateful protocol analysis

Some vendors use the term “deep packet inspection” to refer to performing some type of stateful protocol analysis, often combined with a firewall capability that can block communications determined to be malicious.

• This publication uses the term “stateful protocol analysis” because it is appropriate for analyzing both network-based and host-based activity, whereas “deep packet inspection” is an appropriate term for network-based activity only.
• Also, historically there has not been consensus in the security community as to the meaning of “deep packet inspection”.

Stateful protocol analysis is the process of comparing predetermined profiles of generally accepted definitions of benign protocol activity for each protocol state against observed events to identify deviations.

Unlike anomaly-based detection, which uses host or network-specific profiles, stateful protocol analysis relies on vendor-developed universal profiles that specify how particular protocols should and should not be used.

Pros

The “stateful” in stateful protocol analysis means that the IDPS is capable of understanding and tracking the state of network, transport, and application protocols that have a notion of state.

• For example, when a user starts a File Transfer Protocol (FTP) session, the session is initially in the unauthenticated state. Unauthenticated users should only perform a few commands in this state, such as viewing help information or providing usernames and passwords.
• An important part of understanding state is pairing requests with responses, so when an FTP authentication attempt occurs, the IDPS can determine if it was successful by finding the status code in the corresponding response. Once the user has authenticated successfully, the session is in the authenticated state, and users are expected to perform any of several dozen commands. Performing most of these commands while in the unauthenticated state would be considered suspicious, but in the authenticated state performing most of them is considered benign.

Stateful protocol analysis can identify unexpected sequences of commands, such as issuing the same command repeatedly or issuing a command without first issuing a command upon which it is dependent.

Another state tracking feature of stateful protocol analysis is that for protocols that perform authentication, the IDPS can keep track of the authenticator used for each session, and record the authenticator used for suspicious activity. This is helpful when investigating an incident. Some IDPSs can also use the authenticator information to define acceptable activity differently for multiple classes of users or specific users.

The “protocol analysis” performed by stateful protocol analysis methods usually includes reasonableness checks for individual commands, such as minimum and maximum lengths for arguments. If a command typically has a username argument, and usernames have a maximum length of 20 characters, then an argument with a length of 1000 characters is suspicious. If the large argument contains binary data, then it is even more suspicious.

Cons

Stateful protocol analysis methods use protocol models, which are typically based primarily on protocol standards from software vendors and standards bodies (e.g., Internet Engineering Task Force [IETF] Request for Comments [RFC]). The protocol models also typically take into account variances in each protocol’s implementation.

• Many standards are not exhaustively complete in explaining the details of the protocol, which causes variations among implementations.
• Also, many vendors either violate standards or add proprietary features, some of which may replace features from the standards.
• For proprietary protocols, complete details about the protocols are often not available, making it difficult for IDPS technologies to perform comprehensive, accurate analysis.
• As protocols are revised and vendors alter their protocol implementations, IDPS protocol models need to be updated to reflect those changes.
• The primary drawback to stateful protocol analysis methods is that they are very resource-intensive because of the complexity of the analysis and the overhead involved in performing state tracking for many simultaneous sessions.

Another serious problem is that stateful protocol analysis methods cannot detect attacks that do not violate the characteristics of generally acceptable protocol behavior, such as performing many benign actions in a short period of time to cause a denial of service.

Yet another problem is that the protocol model used by an IDPS might conflict with the way the protocol is implemented in particular versions of specific applications and operating systems, or how different client and server implementations of the protocol interact.

Source: NIST SP 800-94