{"componentChunkName":"component---src-templates-post-template-js","path":"/clamav-scan-bm-en","result":{"data":{"markdownRemark":{"id":"d83a180c-b727-52d5-9e91-7f79c744913b","html":"
\n

This page has been machine-translated from the original page.

\n
\n

In the previous article, Search Algorithms Powering AntiVirus 1, I looked at an implementation of the Aho–Corasick algorithm and how it is used in ClamAV.

\n

This time, again using ClamAV’s code as a base, I will summarize Boyer–Moore (BM), Wu-Manber (WM), and related techniques that are applied to signature pattern matching.

\n\n

Table of Contents

\n\n

(Background) Pattern Matching in ClamAV (Recap)

\n

As mentioned in the previous article, ClamAV’s core file-scanning function, cli_scan_fmap, is, simply put, a function that scans a memory map abstracted as the cl_fmap structure.

\n

Inside it, processing such as hash-based checks and pattern matching against the loaded signature database is performed.

\n

After performing scan initialization, the cli_scan_fmap function reads up to SCANBUFF (0x20000 bytes) from the file map as one chunk and then scans it with the matcher_run function.

\n

\n \n \n \n \n \n \n

\n

This matcher_run function is called with the following parameters and scans the received buffer.

\n
static inline cl_error_t matcher_run(\n    const struct cli_matcher *root,\n    const unsigned char *buffer, uint32_t length,\n    const char **virname, struct cli_ac_data *mdata,\n    uint32_t offset,\n    const struct cli_target_info *tinfo,\n    cli_file_t ftype,\n    struct cli_matched_type **ftoffset,\n    unsigned int acmode,\n    unsigned int pcremode,\n    struct cli_ac_result **acres,\n    fmap_t *map,\n    struct cli_bm_off *offdata,\n    struct cli_pcre_off *poffdata,\n    cli_ctx *ctx\n)
\n

Here, the cli_bm_scanbuff function first performs scanning with a Boyer-Moore-based algorithm, after which the cli_ac_scanbuff function performs pattern matching with the Aho-Corasick algorithm.

\n

I covered the cli_ac_scanbuff function in the previous article, so this time I will focus on scanning with the Boyer-Moore method through the cli_bm_scanbuff function.

\n

What Is the Boyer–Moore (BM) Algorithm

\n

The BM algorithm is a fast algorithm for searching whether a specified keyword (signature) is contained in a particular text.

\n

The basic idea of the BM algorithm is to preprocess the keyword pattern and achieve fast keyword search by comparing characters from the end of the keyword toward the beginning and by avoiding unnecessary checks as much as possible when testing whether the target string matches the keyword.

\n

The standard BM algorithm generally refers to the version that applies the two rules described below: the Bad Character Rule and the Good Suffix Rule.

\n

This standard BM algorithm has a worst-case time complexity of O(N*M) (text length N × keyword length M), but it seems that by adding the rule known as the Galil rule, the worst-case complexity can be reduced to linear time.

\n

There is also a derivative of the BM algorithm called the Boyer–Moore–Horspool (BMH) algorithm, which uses only the Bad Character Rule and further simplifies BM.

\n

In signature pattern matching for AntiVirus software such as ClamAV, this Boyer–Moore–Horspool (BMH) method is apparently used in some cases for several reasons, so in this article I will mainly focus on BMH.

\n

Bad Character Rule

\n

In the BM algorithm, the Bad Character Rule is a rule for skipping unnecessary checks based on where a character that mismatched the keyword appears inside the keyword.

\n

The diagram in the following article was especially easy to understand for the Bad Character Rule.

\n

\n \n \n \n \n \n \n \n \n

\n

Reference: 文字列検索アルゴリズム③ ー Boyer-Moore法 #競技プログラミング - Qiita

\n

When applying the BM algorithm, after aligning the beginning of keyword T with the current search position in the target string S, you compare the target string and the keyword starting from the word at the end of the keyword.

\n

When a mismatch occurs between the target character and the keyword character, the Bad Character Rule makes string search efficient by shifting the next search position as far as possible.

\n

With the Bad Character Rule, a shift table is created in advance by preprocessing the search keyword (that is, how many characters the search position can be shifted when a mismatch occurs), and the search position is skipped as much as possible depending on the character that mismatched at the end.

\n

For example, in the example explained on Wikipedia, the keyword NNAAMAN and the text are compared from the end, and the rule is evaluated at the position of the mismatched N.

\n

If this N also appears to the left of that position, the search position is skipped to that occurrence. If the character does not appear in the keyword at all, then the keyword can no longer match there, so the search position is skipped to the next character after it.

\n

\n \n {text_char}' found at pattern index {last_occurrence}.\"\n if last_occurrence != -1 and last_occurrence < j\n else f\"Bad char '{text_char}' logic (not in pattern or to right).\"\n )\n\n yield SearchStep(\n shift=shift,\n mismatch_index=j,\n bad_char=text_char,\n shift_amount=shift_amount,\n description=reason\n )\n \n shift += shift_amount\n

When you run a script based on the above, you can see character search from the end using the Bad Character Rule and how the search position shifts when a mismatch occurs, as shown below.

\n

\n \n \n \n \n \n \n \n \n

\n

On the other hand, when using the BMH algorithm, regardless of where the mismatch occurs, the algorithm focuses on the last character and slides the search position so that the last character aligns with the rightmost occurrence of that character within the keyword.

\n

(As in the example below, if the last character d is not included in the keyword, the check is performed by aligning the beginning of the keyword with the next character.)

\n

\n \n \n \n \n \n \n \n \n

\n

Because of this, when using the BMH algorithm there is no need to consider where the check failed, and only the last character needs to be examined. This makes the implementation simpler than the Bad Character Rule and also allows the search position to be shifted farther than with the Bad Character Rule.

\n

The result of implementing the table-building process used by BMH in Python is as follows.

\n
class BMHSearcher:\n\n    def __init__(self, pattern: str):\n        self.pattern = pattern\n        self.m = len(pattern)\n        self._shift_table: Dict[str, int] = self._build_shift_table()\n\n    def _build_shift_table(self) -> Dict[str, int]:\n\n        table = {}\n        # Register all characters except the last character in the pattern (from 0 to m-2)\n        for i in range(self.m - 1):\n            char = self.pattern[i]\n            table[char] = self.m - 1 - i\n        return table
\n

This is almost the same as the Bad Character Rule implementation, but the difference is that the information stored in the table is not the index where a character in the keyword last appears in the keyword, but rather the distance from the final character.

\n

Wu-Manber (WM)

\n

The Wu-Manber (WM) method is a search algorithm designed to search multiple patterns at the same time based on the idea of the BMH algorithm.

\n

The BMH idea of trying to shift as far as possible by using a mismatch at the last character becomes more efficient as the probability rises that the last character does not appear in the keyword.

\n

However, if the number of patterns to search increases, the probability that some character in the pattern set matches the last character also increases, and eventually the algorithm converges to a state where a large shift width can no longer be obtained.

\n

To solve this multi-pattern search problem, the WM method evaluates and skips search positions based not on a single character but usually on blocks of 2 or 3 characters.

\n

In the WM method, in the preprocessing stage, the minimum pattern length in the pattern set is taken as m, the block size—usually 2 or 3—is taken as B, and three tables, SHIFT, HASH, and PREFIX, are created using the first m characters of each pattern.

\n

The SHIFT table is an extension of the BM shift table and is used during text scanning to determine how many characters in the text can be skipped.

\n

The HASH table contains pointers to lists of patterns that share the same suffix (the last B characters of the string obtained by taking the first m characters of the pattern), and the PREFIX table contains the hash value of each pattern’s prefix (the first B characters). These are used during pattern matching.

\n

I implemented this preprocessing in Python as follows.

\n
class WuManber:\n\n    def __init__(self, patterns: List[str], block_size: int = 2) -> None:\n        if not patterns:\n            raise ValueError(\"Invalid patterns.\")\n        \n        self.patterns = patterns\n        self.block_size = block_size\n        self.min_len = min(len(p) for p in patterns)\n\n        if self.min_len < self.block_size:\n            raise ValueError(f\"Block size {self.block_size} is larger than the minimum pattern length {self.min_len}.\")\n\n        # SHIFT table\n        self.shift_table: Dict[str, int] = {}\n\n        # HASH table\n        self.hash_table: Dict[str, List[int]] = defaultdict(list)\n\n        # PREFIX table / Python's built-in hash() returns an integer, so use it as is\n        self.prefix_table: List[int] = []\n\n        self._build_tables()\n\n    def _get_hash(self, text_block: str) -> int:\n        return hash(text_block)\n\n    def _build_tables(self) -> None:\n\n        for idx, pattern in enumerate(self.patterns):\n\n            prefix_part = pattern[:self.block_size]\n            self.prefix_table.append(self._get_hash(prefix_part))\n\n            limit_pattern = pattern[:self.min_len]\n            \n            # shift_table: {'ap': 3, 'pp': 2, 'pl': 1, 'le': 0, 'am': 3}\n            for i in range(self.min_len - self.block_size + 1):\n                block = limit_pattern[i : i + self.block_size]\n                shift = self.min_len - 1 - (i + self.block_size - 1)\n                \n                if block not in self.shift_table:\n                    self.shift_table[block] = shift\n                else:\n                    self.shift_table[block] = min(self.shift_table[block], shift)\n                \n                if shift == 0:\n                    self.hash_table[block].append(idx)
\n

First, in the initialization process below, various tables are created using patterns, the list of search keywords, and the default block size of 2.

\n
def __init__(self, patterns: List[str], block_size: int = 2) -> None:\n    if not patterns:\n        raise ValueError(\"Invalid patterns.\")\n\n    self.patterns = patterns\n    self.block_size = block_size\n    self.min_len = min(len(p) for p in patterns)\n\n    if self.min_len < self.block_size:\n        raise ValueError(f\"Block size {self.block_size} is larger than the minimum pattern length {self.min_len}.\")\n\n    # SHIFT table\n    self.shift_table: Dict[str, int] = {}\n\n    # HASH table\n    self.hash_table: Dict[str, List[int]] = defaultdict(list)\n\n    # PREFIX table / Python's built-in hash() returns an integer, so use it as is\n    self.prefix_table: List[int] = []\n\n    self._build_tables()
\n

At this point, the length of the shortest pattern among all patterns is stored as min_len. (If this min_len is smaller than the block size, search using the standard WM method cannot be used.)

\n

The _build_tables method that actually creates the tables is implemented as follows.

\n
def _build_tables(self) -> None:\n\n    for idx, pattern in enumerate(self.patterns):\n\n        prefix_part = pattern[:self.block_size]\n        self.prefix_table.append(self._get_hash(prefix_part))\n\n        limit_pattern = pattern[:self.min_len]\n\n        for i in range(self.min_len - self.block_size + 1):\n            block = limit_pattern[i : i + self.block_size]\n            shift = self.min_len - 1 - (i + self.block_size - 1)\n\n            if block not in self.shift_table:\n                self.shift_table[block] = shift\n            else:\n                self.shift_table[block] = min(self.shift_table[block], shift)\n\n            if shift == 0:\n                self.hash_table[block].append(idx)
\n

Here, the code loops over the list of patterns. First, it appends to the PREFIX table the integer value produced by hashing the first B characters of the pattern (pattern[:self.block_size]) with Python’s built-in hash() function.

\n

Note: This makes it possible to retrieve the hash of the leading block from the PREFIX table using the pattern index.

\n

Next, after storing the first m characters of the pattern as limit_pattern = pattern[:self.min_len], the code runs a loop for m - B + 1 iterations and creates the SHIFT table there.

\n

Here, blocks of length B are taken out one by one from the first m characters of each pattern, and a safely shiftable amount is added to the SHIFT table for each block.

\n

If a block is the last block of the pattern of length m (shift == 0), the pattern index is added to the list keyed by that block, creating the HASH table.

\n
block = limit_pattern[i : i + self.block_size]\nshift = self.min_len - 1 - (i + self.block_size - 1)\n\nif block not in self.shift_table:\n    self.shift_table[block] = shift\nelse:\n    self.shift_table[block] = min(self.shift_table[block], shift)\n\nif shift == 0:\n    self.hash_table[block].append(idx)
\n

With this, all the tables needed to perform search with the WM method have been created.

\n

Next, let’s look at the text search process (search function) that uses these tables.

\n
def search(self, text: str) -> Iterator[Tuple[int, str]]:\n    n = len(text)\n    m = self.min_len\n    B = self.block_size\n\n    if n < m:\n        return\n\n    idx = m - 1\n    default_shift = m - B + 1\n\n    while idx < n:\n        # Obtain the suffix block and determine the shift amount\n        suffix_block = text[idx - B + 1 : idx + 1]\n        shift = self.shift_table.get(suffix_block, default_shift)\n\n        # If the shift amount is 0, compare with candidate patterns (otherwise shift)\n        if shift == 0:\n\n            candidate_indices = self.hash_table.get(suffix_block, [])\n            text_start_pos = idx - m + 1\n            text_prefix_block = text[text_start_pos : text_start_pos + B]\n            text_prefix_hash = self._get_hash(text_prefix_block)\n\n            for p_idx in candidate_indices:\n                if self.prefix_table[p_idx] != text_prefix_hash:\n                    continue\n\n                pattern = self.patterns[p_idx]\n                p_len = len(pattern)\n\n                if text_start_pos + p_len <= n:\n                    if text[text_start_pos : text_start_pos + p_len] == pattern:\n                        yield (text_start_pos, pattern)\n\n            idx += 1\n\n        else:\n            idx += shift
\n

When the search function is called, it first takes m characters from the text, extracts the last B characters as suffix_block, and compares them with the SHIFT table.

\n

If the last B characters do not exist in the SHIFT table, the search position is moved by m - B + 1. If they do exist in the SHIFT table, the search position is moved by the shift amount stored in the table.

\n

If the shift amount is 0, there is a possibility that a pattern is contained at the current search position, so the list of pattern indices whose suffix includes that block is extracted from the HASH table as candidate_indices.

\n

Next, after retrieving from the PREFIX table the hash of the leading block of the patterns whose suffix contains that block, the code compares it with the hash of the leading block in the text at the current search position. If they match, it performs a full-text comparison against the pattern.

\n

The reason for doing prefix matching with the PREFIX table here seems to be that when searching for words in English text, suffix blocks shared by many words, such as tion or ee, may match frequently.

\n

Therefore, by checking not only the suffix block but also the leading block, and then performing a full-text search only for the candidates that pass both checks, the process can be made more efficient.

\n

Reference: AFASTALGORITHMFORMULTI-PATTERNSEARCHING

\n

Reference: Variations of Wu-Manber String Matching Algorithm

\n

Reference: wumanber/wumanber.hpp at master · bubiche/wu_manber

\n

Reading ClamAV’s Implementation

\n

Up to this point, I have summarized the BM algorithm, the BMH algorithm, and the WM method, which supports multi-pattern matching based on the same idea.

\n

From here, I will read the actual implementation of ClamAV’s cli_bm_scanbuff function and examine how pattern matching is implemented in AntiVirus software.

\n

Function Calls

\n

The bm in the function name cli_bm_scanbuff stands for the BM algorithm. It is implemented as “Extended Boyer-Moore,” and its actual behavior is close to the WM method.

\n

The cli_bm_scanbuff function is called with the following parameters, and when scanning files with clamscan, the data from the scanned file is passed in as buffer.

\n

\n \n