Portal de Periódicos da CAPES

Operating system security by integrity checking and recovery using write‐protected storage

2013; Institution of Engineering and Technology; Volume: 8; Issue: 2 Linguagem: Inglês

10.1049/iet-ifs.2012.0346

1751-8717

Jerzy Kaczmarek, Michał R. Wróbel,

Security and Verification in Computing

IET Information SecurityVolume 8, Issue 2 p. 122-131 ArticleFree Access Operating system security by integrity checking and recovery using write-protected storage Jerzy Kaczmarek, Jerzy Kaczmarek Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology, Gdansk, PolandSearch for more papers by this authorMichal R. Wrobel, Corresponding Author Michal R. Wrobel wrobel@eti.pg.gda.pl Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology, Gdansk, PolandSearch for more papers by this author Jerzy Kaczmarek, Jerzy Kaczmarek Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology, Gdansk, PolandSearch for more papers by this authorMichal R. Wrobel, Corresponding Author Michal R. Wrobel wrobel@eti.pg.gda.pl Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology, Gdansk, PolandSearch for more papers by this author First published: 01 March 2014 https://doi.org/10.1049/iet-ifs.2012.0346Citations: 8AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract An integrity checking and recovery (ICAR) system is presented here, which protects file system integrity and automatically restores modified files. The system enables files cryptographic hashes generation and verification, as well as configuration of security constraints. All of the crucial data, including ICAR system binaries, file backups and hashes database are stored in a physically write-protected storage to eliminate the threat of unauthorised modification. A buffering mechanism was designed and implemented in the system to increase operation performance. Additionally, the system supplies user tools for cryptographic hash generation and security database management. The system is implemented as a kernel extension, compliant with the Linux security model. Experimental evaluation of the system was performed and showed an approximate 10% performance degradation in secured file access compared to regular access. 1 Introduction Computer systems are commonly used to store important information regarding technical, financial and personal data, which require effective protection from unauthorised access or data fraud. Numerous threats for computer system security may be mentioned, including viruses, Trojan programs, rootkits and others. Most computer system attacks are performed using network access and the likelihood of a security breach is increased if the computer system is not adequately protected. Methods for system protection have been designed and applied, including firewall systems, access control security policies and intrusion detection systems. However, there are still serious threats of successful attacks and we can assume that there will always be a risk of successful intrusions on computer systems. Therefore, it is necessary to apply additional methods that detect intrusions to a computer system apart from methods that attempt to prevent such attacks. Methods for file modification monitoring are one of the most effective methods of intrusion detection. File integrity checking systems [1] utilise the mechanism of a unique digital fingerprint that is calculated from the contents of a file. The fingerprint, also known as a checksum, is stored in a secured database and retrieved to check if an unauthorised modification to a file has been made. Algorithms used to calculate files fingerprints are critical for the effectiveness of the method, as they must ensure that any file modification will result in a different checksum value. Additionally, it must be impossible to reconstruct file contents from its checksum and simultaneously checksum calculation should be computationally effective. Algorithms that generate fixed length cryptographic hashes are most commonly used for file checksum generation. There are cryptographic functions available such as Message Digest Algorithm, Secure Hash Algorithm and Tiger Message Digest Algorithm. The algorithms must be resistant to attacks such as first pre-image attacks, second pre-image attacks and collision generation [2]. Although the existing integrity checking systems improve system security [3], they also have weaknesses that may be used by an intruder to eliminate the protection in certain situations. Typically, the systems insufficiently protect the security database and are vulnerable to attacks that modify its binaries. The weakness concerns both systems that are run as user applications and systems that are integrated in the kernel. Additionally, the existing systems do not provide file backups, which results in the need of additional administrative work to restore infected files each time an intrusion occurs. We propose a novel system that enables file integrity verification – integrity checking and recovery (ICAR) – implemented as an in-kernel Linux security modules (LSM)-based module [4]. Our system overcomes the shortcomings of existing solutions taking the following main assumptions: Write-protected media is used to store crucial files: system kernel with ICAR module, backups of the protected files and cryptographic hashes database. The system supplies a backup of protected files that enable automated recovery if the files have been modified in an unauthorised way. ICAR is integrated with operating system kernel using the LSM mechanism and runs in the kernel space. Experiments showed that our system is an efficient mechanism of operating system protection against unauthorised file modifications. The rest of the paper is organised as follows. The next section presents related work and background for our research. Section 3 presents the ICAR protection model, architecture and applied algorithms. System implementation, installation and configuration are discussed in Section 4. Section 5 presents experimental evaluation. 2 Background and related work Most file system integrity checking tools leverage a common pattern for operation steps as shown in Fig. 1 [5]. First, the integrity checking process reads configuration data to determine which files should be protected. Then the process calculates cryptographic hashes of the files and stores the hashes in a database. Fig. 1Open in figure viewerPowerPoint Main components for file system integrity checking tools During regular system operation, the process checks integrity by recalculating cryptographic hashes of files and comparing them with the values stored in the database. If a security violation is detected, appropriate security actions are taken. The actions may cover generation of a report about potential intrusions, notification to the system administrator, file access denial or system shutdown. The Linux operating system distinguishes between user-level code and kernel-level code, which is an important feature for integrity checking tools. Two kinds of tools are distinguished in this context: those that run in user space with root privileges and those that run in kernel space. User-level code covers regular applications, text and graphical user interfaces and operating system commands invoked by a user. Kernel-level code covers, among other things, applications for memory management, process management and file system operations. A typical threat to computer system security that occurs at user-level code is a situation in which an error, such as buffer overflow, in an application is used to overtake control of the computer system. This kind of threat is virtually impossible to eliminate because of a large number of applications and their high complexity. User-level integrity checking tools are run as regular applications. Usually, the integrity checking procedure is executed periodically, so integrity violations are detected during explicit executions rather than automatically on occurrence. The Tripwire [5] tool developed in the Purdue University in 1992 is one of the first commonly used intrusion detection system. Tripwire is a seminal integrity checking system for other user-level tools such as AIDE, Veracity and Integrit [6]. The tool enables definition of protected file lists, signature generation, file verification and reporting of violations. It should be noted that user-level integrity checking tools are not aimed at intrusion prevention, but rather effective detection of an already occurred security violation. The procedure of file integrity checking is triggered either periodically or manually by the administrator. Kernel-level integrity checking tools supply higher security than user-level ones. Operating system kernel verifies security constraints during operations, for example, access privileges when a process requests a file access. Regular kernel operations may be extended with integrity checking if a security mechanism is a part of the operating system, in particular a procedure for verifying file cryptographic hash may be supplied. If hash verification indicates that the file has not been modified, file access is granted. However, if hash verification fails, which might indicate an unauthorised modification, file access may be denied. Security systems can be integrated with the kernel in several different ways. Stackable file system [7] consists of a number of independent functional layers and may be used as an efficient solution that enables implementation of file system security in the kernel space. File system functionality may be extended by adding an additional layer responsible for security. The layer is typically located on the top of the stack during file system mounting and intercepts and controls file access by processes. The solution enables separation of file system integrity checking code and kernel code. System calls is another approach that may be used to implement kernel-level security, as the mechanism enables programs to request kernel services [8]. In order to implement a security extension, standard system calls are replaced with custom functions implementing secure file access. The mechanism, however, has many drawbacks [9]: it is not race-free, may require code duplication and may be limited in expressing the security context. Finally, the LSM project [9] has been developed to supply an access control framework for the Linux kernel that enables different control models implemented as loadable kernel modules. The basic abstraction of the LSM interface is to mediate access to internal kernel objects. The mechanism uses hooks that are placed in the kernel code just ahead of the access. The hooks are used to call LSM functions that supply information about granting or denying a requested access. The LSM framework is used, among others in AppArmor and SELinux [10]. The secure on-the-fly file integrity checker (SOFFIC) [11] and in-kernel integrity checker and intrusion detection file system (I3 FS) [12] systems are important implementations of file system integrity checking that leverage kernel-level access. The SOFFIC system modifies system calls responsible for file access. It stores the list of protected files and their initial hashes in regular files, which is the primary drawback of the system because attackers can easily remove or change such important data. I3 FS system is an in-kernel, on-access integrity checking system that is designed on a stackable file system. It is capable of blocking access to affected files and notifying the administrator. The design assumes that the file system is the most appropriate location for security modules because most intrusions cause file modification. The system implements four databases located in system kernel space that concern: (i) policy options associated with files, (ii) checksums of file data, (iii) metadata checksums and (iv) access counters. Both SOFFIC and I3 FS, similar to Tripwire, are vulnerable to attacks that modify the database of file hashes or the operating system kernel. Attacks of this kind may effectively disable the security mechanism. Our solution differs in that we leverage the LSM mechanism, which effectively protects our tool from unauthorised modification at the kernel level. ICAR supplies a backup mechanism that enables continuous work despite an intrusion. Additionally, we use write-protected storage to store cryptographic hashes, security module and system kernel ensuring that no modification can be made to them. Different approach to file integrity checking are used in some network intrusion detection system [13] and hardware systems such as intrusion detection on disk [14]. But despite the good performance, such solutions are more complicated and expensive than those designed by us. 3 ICAR protection model The design of a security mechanism requires analysis concerning scope, time and manner of protection. A running operating system may be viewed as both files and running processes that may become a potential target of an attack and should be protected. The Linux operating system is a convenient platform for research in the field of security as it is freely available in both binary and source code. It can be modified and extended using the open source approach and existing licenses. Therefore, the system was chosen as the testbed operating system for design and implementation work. Although numerous methods and tools have been supplied for system protection, it cannot be guaranteed that an intruder will not violate system security and grant unprivileged access to system resources. The existing Linux security mechanisms supply a relatively more effective protection for running processes than for the file system. Therefore, a mechanism should be designed that protects file contents, especially considering important configuration and system files. The proposed security system consists of three layers as shown in Fig. 2. System layers are as follows: Kernel-level layer – is responsible for the verification of file system integrity. Data layer – contains critical data for the security mechanism: database that stores initial cryptographic hashes and backup copies of protected files. Utility layer – equips the end user with tools that simplify the administrators work in configuration and management of the system. Fig. 2Open in figure viewerPowerPoint ICAR system layers The module of kernel-level layer is loaded into RAM as a part of operating system kernel during system startup. It is responsible for calculating file cryptographic hashes that are further stored by the data layer. Given the known attacks on the elderly functions [15], it is recommended to use a SHA 256 algorithm or a new one. The layer also contains a cache that is used by the protection algorithm, described further in this section, to store the results of file verifications that have already been performed. The use of cache increases the speed of the security system as it limits the necessity of signature calculating for files that have already been verified. File hashes stored by the second layer are used by the kernel-level layer to detect if an unauthorised modification has been made to the system. Additionally, file copies are used to restore the original file contents if unauthorised file modifications have been detected (as described further in Section 3.1). Both cryptographic hashes and file copies are stored in write-protected data storage. The mount point of the storage is additionally protected by ICAR to prevent from mounting a fake security database. The kernel-level layer intercepts mount operations, analogous to intercepting file read operations, and blocks mount attempts of the directory with the security database. Developed helper tools cover applications that enable selection of protected files, generate initial cryptographic hashes and files backup, which are stored in the data-layer. Tools also allow to monitor file system integrity. If necessary, a system administrator may receive alarm notifications from the monitoring system. Information about compromised files and file recovery may simplify administrators work in detecting intrusions. 3.1 File protection algorithm Fig. 3 shows the main algorithm for integrity checking used by ICAR during reading or executing attempt. The algorithm for integrity checking is activated each time a process requests a read access to a file. First, the algorithm reads the security database to check if the accessed file is in the group of protected files. If there is no information about the file in the database, it is assumed that the file is not protected for integrity and access is granted for the requesting process. Fig. 3Open in figure viewerPowerPoint File system protection algorithm If there is information about the file, which means that the file is protected for integrity, ICAR verifies if file contents has been modified since system startup. The check requires calculating the cryptographic hash from the present contents and then comparing that against the value of the initial hash stored in security database. However, despite the fact that the system uses a relatively effective kernel built-in cryptographic function, the calculation is time-consuming. Considering performance issues, ICAR implements a cache that stores information about positively verified files. If a file has been checked on a previous access, there is no need to recalculate the hash of a file as long as the file has not been modified. Therefore, when a process requests access to a file that is already in the cache, permission is granted immediately. Otherwise when the cache does not contain information about a file, which is protected for integrity, it is necessary to calculate the cryptographic hash from the present file contents and compare it against the initial value. If the verification is successful, the security mechanism stores the information about the file in the cache for further use and grants access to the file for the requesting process. If the calculated hash differs from the initial hash, a security procedure is activated covering an attempt to restore the original file content from backup and notifying the system administrator. In the first step, the mechanism attempts to restore the original file by copying it from the write-protected storage backup to the hard disk of the computer system. If file restoration is successful, the tampered file is replaced with the original one. Then, the algorithm follows steps for positive file verification granting access to the file for the requesting process. If, however, file restoration fails for any reason, file access is denied. The file is not accessible for users and processes running in the system for reading and executing. The security system may take other security actions in response to a detected unauthorised modification of a file depending on configuration. The system may log the information in system logs, alert administrator, block IP addresses or even shutdown the operating system. System shutdown may be necessary if file modifications are detected frequently, which may indicate a considerable attack on the system. Fig. 4 shows two helper algorithms. The first is responsible for cleaning cache during a write operation. Independently from integrity checking, the system monitors file modifications and clears cache information about a file when its modification is detected. The second algorithm describes how the system protects the mount point with the ICAR data against unmounting or remounting operations. Fig. 4Open in figure viewerPowerPoint File system protection algorithm 3.2 Cache construction The structure of the cache is an important issue related to the performance of the ICAR system. The solution was designed with the assumption of low memory usage and fast file verification. It is sufficient to store a unique identification number of verified files in the cache. Therefore, even when hundreds of thousands of files are protected, the size of the buffer will be in the order of megabytes. This approach also eliminates the risk of an attack, in which the contents of verified files is changed using hard links (information about these files is stored in the cache). To ensure high search performance, it is necessary to use a hash table to store the information on previously verified files. A suitable hash function must be selected to minimise generation of collisions, as well as an algorithm for dealing with collisions occurrence. Additionally, periodic cleaning of the cache is assumed, which allows reduction of the size of the cache and refresh its contents. This mechanism may be activated periodically or when exceeding a predefined limit. 3.3 Security issues of ICAR integration with the operating system An effective implementation of the security system requires resolution of two design issues: a method of data storage and a method for integration with operating system kernel. We use a write-protected storage for system kernel, the integrity checking module, initial cryptographic hashes and file backups. We ensure write protection using hardware mechanisms, which eliminates the threat of modification and guarantees the effectiveness of the solution. In practice, a LiveCD Linux distribution or write-protected USB flash memory are most popular mechanisms that can be used in the developed system. The system assumes that the kernel and the ICAR system are the only modules stored in write-protected storage and other system files may be stored in traditional disks. The bootup process loads system kernel from the write-protected storage and then continues with traditional booting using data from hard disks. If ICAR detects a file modification during integrity checking, it restores the original file contents using the backup copy from the secure storage. ICAR design assumes that the module is integrated with system kernel, which minimises the threat of its elimination by an unauthorised user. In practice, different solutions are available for such integration in Linux, including system stackable file systems, system calls and LSM. Table 1 shows methods of kernel–module integration and their selected properties. Table 1. Methods of kernel-module integration Stackable filesystem System call interception Linux security model initialisation method mount the file system load kernel module load kernel module required changes implementation of file system layer with modified read and write functions read and write functions implementation and substitution read and write functions implementation programming language abstract high level language (FIST) C C properties easy to maintain, noticeable performance overhead inefficient and prone to time-of-check-to-time-of-use race standard part of the Linux kernel We decided to use LSM because it is well suited for security actions such as verification and filtering of user operations. LSM supplies a well-defined API, allowing integration of security modules with system kernel. It is also resistant to TOCTTOU attacks [16, 17]. ICAR is implemented as an LSM-hooked module that resides in Linux system kernel. The use of LSM ensures that our module will not be removed from the operating system kernel by an intruder. An unauthorised modification of cache is a serious threat that must be prevented. Three groups of attacks on cache has been identified: malware modules loaded into the operating system kernel, writing directly to kernel memory and writing directly through partition device file. All of these possible attacks can be blocked by the application of the grsecurity kernel patch. 4 System implementation The implementation of the ICAR system is an extension for the Linux operating system and aims at verifying the effectiveness of the designed mechanism. The cdlinux.pl Linux distribution was chosen as the runtime environment, although ICAR may be applied in any Linux distribution. The cdlinux.pl is a Debian-based distribution developed by authors since 2001, available from www.cdlinux.pl. ICAR is implemented in the C language and consists of two main parts: a kernel module and user-level tools. The LSM-based kernel module covers over 1.000 lines of code of kernel module responsible for file protection. The cache mechanism utilises the Linux inode hash function and stores its data in slab allocator. The implementation supplies user-level tools that simplify setting-up, initialisation and configuration of ICAR. Additionally, a script was prepared that automates the creation of the operating system containing the designed security mechanism. The generated system may be used to create a LiveCD image or USB memory contents. ICAR sources are available at http://www.cdlinux.pl/icar. The system is available under the GPL license and may be used by computer system users or other researches for further studies. 4.1 System installation and configuration ICAR installation and configuration requires administration steps similar to installation of other integrity checking software. Fig. 5 shows the following main steps of ICAR installation: ICAR installation in the operating system, which covers: ○ Patching system kernel with the ICAR module. ○ Installation of ICAR user-level tools. Selection of protected files. Generation of initial cryptographic hashes. Preparation of write-protected media containing: initial cryptographic hashes, protected file copies and operating system kernel with the ICAR module. System restart with the new kernel from the write-protected media. Fig. 5Open in figure viewerPowerPoint Main steps of security database update process ICAR installation and configuration requires special security means as the operating system may be prone to attacks during reconfiguration of the module. It is recommended that the target machine is clean and isolated (run in single-user mode and disconnected from any network). After taking the security means, the ICAR system may be installed and configured. If the system has already been installed, it should be disabled and reconfigured concerning new file versions or protected file selection. A configured system may be used as long as the contents and selection of protected files do not change. Updated system data is generated and written in a write-protected storage. The operation requires physical access to the computer (i.e. it cannot be performed remotely through a network connection). The restrictions ensure high security of the configuration process. 4.2 Classification of protected files It is virtually impossible to protect all of the files in a system because of their size and number. Protection of all files would inevitably lead to significant performance degradation. Therefore, it is necessary to select which files should be protected. The selection results from a security policy that in turn depends on the value of stored information and computer system type, such as a desktop machine or a corporate server. Significance for security and modification frequency are the main issues that decide whether a file should be integrity protected or not. Files stored on a computer system are classified as operating system files, application files, system configuration files, user data files and others. Protection of operating system and application files is critical for the security of the whole computer system. If an intruder modifies one of those files, he may easily overtake operating system control or install malicious software. The advised security policy assumes that these files should be verified for integrity and cannot be modified without ICAR reconfiguration. Configuration files are the second security group. These files are less crucial as their unauthorised modification does not immediately result in a security threat. Additionally, relatively frequent modifications of these files may be necessary during regular administrative work. Therefore, the advised security policy assumes that only selected configuration files are protected for integrity. Actually, system administrators decide which configuration files should be protected from modifications depending on system purpose and installed software. Files from other groups, including user data files and temporary files, are modified frequently and it is not recommended to select them for integrity protection. Although the files are exposed to attacks, their privileges are limited, which poses a reduced threat to the operating system. These files should be backed-up regularly and appropriate policies should be designed to restore the files if an intrusion has been detected or the files have been corrupted. 5 Evaluation and performance We conducted tests on ICAR to evaluate the performance overhead of the implementation. Two kinds of tests were performed. The first test involved the single calculation of MD5 checksum for files of various sizes, and the second test involved typical file operations (i.e. size modification of graphical files, application compilation and conversion of audio and video files). All tests were performed on a machine with an Intel Core2 Quad Q6600 2.40 GHz processor, 3 GB RAM memory and a Samsung HD322IJ 320 GB 7200 rpm hard disk. We used the Linux 2.6.28 kernel in the experiments. Data-layer of the ICAR system, with the initial cryptographic hashes database and backup of protected files, has been stored on the write-protected USB flash memory. No additional appl