{"componentChunkName":"component---src-templates-post-template-js","path":"/clamav-scan-fmap-en","result":{"data":{"markdownRemark":{"id":"cdfe7bf1-bae8-5a6d-9d62-7d4d45df7361","html":"
\n

This page has been machine-translated from the original page.

\n
\n

In the previous article, I briefly followed the processing flow up to the point where the cli_scan_fmap function is called when scanning an Eicar file with clamscan.

\n
perf_start(ctx, PERFT_RAW);\nret = cli_scan_fmap(ctx,\n                    type == CL_TYPE_TEXT_ASCII ? CL_TYPE_ANY : type,\n                    false,\n                    &ftoffset,\n                    acmode,\n                    NULL);\nperf_stop(ctx, PERFT_RAW);
\n

This time, starting from the implementation of cli_scan_fmap, the core function for file scanning in ClamAV, I will summarize the outline of the Aho–Corasick algorithm used for signature matching in AntiVirus software.

\n\n

Table of Contents

\n\n

Background: Pattern Matching in ClamAV

\n

The cli_scan_fmap function, the core function for file scanning in ClamAV, roughly speaking scans a memory map abstracted as the cl_fmap structure introduced in the previous article.

\n

Inside it, processing such as hash-based checks and pattern matching against the loaded signature database is performed.

\n

After performing initialization for scanning, cli_scan_fmap appears to read up to SCANBUFF (0x20000 bytes) from the file map as a single chunk and then scan it with the matcher_run function.

\n

\n \n \n \n \n \n \n \n \n

\n

This matcher_run function is called with the following parameters.

\n
static inline cl_error_t matcher_run(\n    const struct cli_matcher *root,\n    const unsigned char *buffer, uint32_t length,\n    const char **virname, struct cli_ac_data *mdata,\n    uint32_t offset,\n    const struct cli_target_info *tinfo,\n    cli_file_t ftype,\n    struct cli_matched_type **ftoffset,\n    unsigned int acmode,\n    unsigned int pcremode,\n    struct cli_ac_result **acres,\n    fmap_t *map,\n    struct cli_bm_off *offdata,\n    struct cli_pcre_off *poffdata,\n    cli_ctx *ctx\n)
\n

The matcher_run function scans the received buffer. It first performs a Boyer-Moore-based scan with cli_bm_scanbuff (its actual behavior is closer to the Wu method), and then performs pattern matching with the Aho-Corasick algorithm in cli_ac_scanbuff.

\n

What Is the Aho-Corasick Algorithm?

\n

The Aho-Corasick algorithm is a string-search algorithm originally devised to efficiently search for specific keywords in documents. It is one way to perform what is sometimes called a “Common Prefix Search”: extracting every keyword that partially matches from any position in a given text.

\n

Reference: Aho–Corasick algorithm - Wikipedia

\n

Reference: Aho Corasick 法 - naoyaのはてなダイアリー

\n

Reference: 高速な文字列探索：Daachorseの技術解説 - LegalOn Technologies Engineering Blog

\n

In the Aho-Corasick algorithm, an automaton (a model of computer states and transitions) is built in advance from the keyword patterns you want to search for, and pattern matching is then performed against a given buffer.

\n

For AntiVirus pattern matching such as in ClamAV, the automaton can be created beforehand from the preloaded signature database, so the computational cost required for pattern matching is linear with respect to the input buffer (and the matched entries).

\n

In this way, the Aho-Corasick algorithm can efficiently check whether many signatures defined in a signature database are contained in a specific data stream, making it well suited to implementing pattern-matching processing in AntiVirus software such as ClamAV.

\n

\n \n \n \n \n \n \n \n \n

\n

Reference: The Aho-Corasick Paradigm in Modern Antivirus Engines: A Cornerstone of Signature-Based Malware Detection

\n

Implementing the Aho-Corasick Algorithm

\n

Building a Trie

\n

To implement the Aho-Corasick algorithm, we first create an automaton based on a Trie built from a fixed set of keywords.

\n

A Trie is a tree structure that represents a set of strings.

\n

As shown below, a Trie uses the empty string as the Root and manages multiple strings together by grouping those that share the same prefix.

\n

\n \n \n \n \n \n \n \n \n

\n

In the example above, the set of keywords \"A\", \"to\", \"tea\", \"ted\", \"ten\", \"i\", \"in\", \"inn\" is represented as a Trie, and the first node contains the unique prefixes \"t\", \"A\", \"i\".

\n

After that, the Trie is completed by assigning nodes one character at a time for items in the keyword set that share common prefixes.

\n

Reference: すごいTrie #アルゴリズム - Qiita

\n

Reference: トライ (データ構造) - Wikipedia

\n

Implementing a Trie

\n

Next, we will implement a Trie in Python.

\n

First, we define the TrieNode class as a node in the Trie.

\n

Each node will hold the following two pieces of information.

\n
    \n
  1. A list indicating the next nodes (TrieNode) (defined as a dictionary in this example)
    2. \n
  2. A value indicating whether this node is terminal
    3. \n
\n
@dataclass\nclass TrieNode:\n    children: Dict[str, \"TrieNode\"] = field(default_factory=dict)\n    is_end: bool = False
\n

Here, field(default_factory=dict) instructs the class defined with @dataclass to use a newly created object produced by dict each time an instance is generated. (See the references below for details.)

\n

Reference: dataclass で万物に型を付けよう｜しゅんけー

\n

Reference: 【Python】dataclassで初心者が必ずハマる default_factory の罠を徹底解説 #Python - Qiita

\n

Now that we have defined Trie nodes, next we implement the Trie class, which adds nodes and searches them.

\n

As described above, the Root of the Trie is empty, so when instantiating the class we assign an empty TrieNode to self.root.

\n
class Trie:\n    def __init__(self) -> None:\n        self.root = TrieNode()
\n

In the insert method for registering keywords in the Trie, the string keyword is taken one character at a time and the Trie is traversed from the Root in order.

\n

At this time, if the target character does not yet exist in the child nodes, a newly created TrieNode instance is registered in the dictionary as a child node.

\n

Next, we implement the is_contains method, which searches whether a specific keyword is registered in the Trie, as follows.

\n
def is_contains(self, keyword: str) -> bool:\n    node = self._find(keyword)\n    return node.is_end if node else False\n\ndef _find(self, keyword: str) -> Optional[TrieNode]: # TrieNode | None\n    node = self.root\n    for ch in keyword:\n        node = node.children.get(ch)\n        if node is None:\n            return None\n    return node
\n

The final code is shown below.

\n
from dataclasses import dataclass, field\nfrom typing import Dict, Optional\n\n\n@dataclass\nclass TrieNode:\n    children: Dict[str, \"TrieNode\"] = field(default_factory=dict)\n    is_end: bool = False\n\n\nclass Trie:\n    def __init__(self) -> None:\n        self.root = TrieNode()\n\n    def insert(self, keyword: str) -> None:\n        node = self.root\n        for ch in keyword:\n            if ch not in node.children:\n                node.children[ch] = TrieNode()\n            node = node.children[ch]\n        node.is_end = True\n\n    def is_contains(self, keyword: str) -> bool:\n        node = self._find(keyword)\n        return node.is_end if node else False\n\n    def _find(self, keyword: str) -> Optional[TrieNode]: # TrieNode | None\n        node = self.root\n        for ch in keyword:\n            node = node.children.get(ch)\n            if node is None:\n                return None\n        return node\n\n\nif __name__ == \"__main__\":\n    trie = Trie()\n    for w in [\"he\", \"she\", \"his\", \"hers\", \"I\", \"my\", \"ME\", \"mine\"]:\n        trie.insert(w)\n\n    assert(trie.is_contains(\"her\") == False)\n    assert(trie.is_contains(\"hero\") == False)\n    assert(trie.is_contains(\"she\") == True)\n    assert(trie.is_contains(\"I\") == True)\n    assert(trie.is_contains(\"i\") == False)\n    assert(trie.is_contains(\"ME\") == True)\n    assert(trie.is_contains(\"Me\") == False)
\n

About the Aho-Corasick Automaton

\n

With the Trie implementation above, we can now determine whether “a specific keyword exists in a predefined set,” but a simple Trie cannot be used to detect the “partial matches within a byte stream” that AntiVirus pattern matching requires.

\n

So, based on the Trie structure, we create a finite automaton for searching with the Aho-Corasick method.

\n

Specifically, we add Failure Links and Output to the Trie implemented as a tree structure.

\n

Failure Links are links that let the search transition to the next state that may match, instead of restarting the search, when matching a keyword character fails partway through.

\n

As a result, for example, when a pattern that has matched ...ABCD mismatches on the next character, instead of starting over it can use the fact that the suffix ...BCD may still match, transition to the B -> C -> D nodes in the Trie structure, and continue searching efficiently.

\n

Reference: AHO CORASICK ALGORITHM. The Aho-Corasick algorithm is a… | by Viraj Tammana | Medium

\n

Reference: Aho-Corasick Algorithm: Key Concepts, Code & Real Examples

\n

This mechanism prevents the search position from ever rewinding within the input byte stream, so pattern matching can be performed in linear time with respect to the input.

\n

The Output values, meanwhile, contain the keywords that match when that state is reached.

\n

\n \n \n \n \n \n \n \n \n

\n

Also, nodes in the AC method are defined as the cli_ac_node structure.

\n
struct cli_ac_patt {\n    uint16_t *pattern, *prefix, length[3], prefix_length[3];\n    uint32_t mindist, maxdist;\n    uint32_t sigid;\n    uint32_t lsigid[3];\n    uint16_t ch[2];\n    char *virname;\n    void *customdata;\n    uint16_t ch_mindist[2];\n    uint16_t ch_maxdist[2];\n    uint16_t parts, partno, special, special_pattern;\n    struct cli_ac_special **special_table;\n    uint16_t rtype, type;\n    uint32_t offdata[4], offset_min, offset_max;\n    uint32_t boundary;\n    uint8_t depth;\n    uint8_t sigopts;\n};\n\nstruct cli_ac_list {\n    struct cli_ac_patt *me;\n    union {\n        struct cli_ac_node *node;\n        struct cli_ac_list *next;\n    };\n    struct cli_ac_list *next_same;\n};\n\nstruct cli_ac_node {\n    struct cli_ac_list *list;\n    struct cli_ac_node **trans, *fail;\n};
\n

Within this implementation, it starts from the Root node obtained by root->ac_root, then follows the transition destinations corresponding to buffer[i] one by one from trans (the next node), and when the node is terminal it begins the pattern-matching process.

\n
current = root->ac_root;\n\nfor (i = 0; i < length; i++) {\n    current = current->trans[buffer[i]];\n\n    if (UNLIKELY(IS_FINAL(current))) {\n        struct cli_ac_list *faillist = current->fail->list;\n        pattN                        = current->list;\n        /* omitted */\n    }\n}
\n

Various checks are performed here, but the point I particularly want to focus on is the ac_forward_match_branch function, which is called further from the ac_findmatch function.

\n
/* state should reset on call, recursion depth = number of alternate specials */\n/* each loop iteration starts on the NEXT sequence to validate */\nstatic int ac_forward_match_branch(const unsigned char *buffer, uint32_t bp, uint32_t offset, uint32_t fileoffset, uint32_t length,\n                                   const struct cli_ac_patt *pattern, uint32_t pp, uint16_t specialcnt, uint32_t *start, uint32_t *end)\n{\n    int match;\n    uint16_t wc, i;\n\n    match = 1;\n\n    /* forward (pattern) validation; determines end */\n    for (i = pp; i < pattern->length[0] && bp < length; i++) {\n        AC_MATCH_CHAR(pattern->pattern[i], buffer[bp], 0);\n        if (!match)\n            return 0;\n\n        bp++;\n    }\n    *end = bp;
\n

In the code above, this is where the scanned buffer and the corresponding signature pattern are finally compared.

\n

If you actually debug an Eicar scan, you can confirm that at this point the signature pattern in pattern->pattern and the scan data in buffer are being compared one byte at a time.

\n

\n \n