WO2002084939A1 - System and method for securely executing a executable to preserve the integrity of files from unauthorized access for network security - Google Patents

System and method for securely executing a executable to preserve the integrity of files from unauthorized access for network security Download PDF

Info

Publication number
WO2002084939A1
WO2002084939A1 PCT/US2001/011552 US0111552W WO02084939A1 WO 2002084939 A1 WO2002084939 A1 WO 2002084939A1 US 0111552 W US0111552 W US 0111552W WO 02084939 A1 WO02084939 A1 WO 02084939A1
Authority
WO
WIPO (PCT)
Prior art keywords
executable
checksum
inode
library file
execution
Prior art date
Application number
PCT/US2001/011552
Other languages
French (fr)
Inventor
Mark Zielinski
Gabriel Logan
Brian Matthews
Original Assignee
Mark Zielinski
Gabriel Logan
Brian Matthews
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mark Zielinski, Gabriel Logan, Brian Matthews filed Critical Mark Zielinski
Priority to PCT/US2001/011552 priority Critical patent/WO2002084939A1/en
Publication of WO2002084939A1 publication Critical patent/WO2002084939A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
    • G06F21/52—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow

Definitions

  • the following invention relates to a method and system for securing a computer system and, in particular, to a method and system for preserving the integrity of files contained in the computer system thereby securing the computer system from unauthorized access.
  • Computer systems are used to connect people and business worldwide.
  • the Internet which consists of a web of networked computer systems, has become a ubiquitous platform for global communications and commerce. Because of the central role networked computer systems have in today's economy, it is vital that the security of networked computer
  • a common prior art security device is a firewall that consists of hardware and software designed to keep unauthorized network traffic from accessing information within a networked computer system.
  • Firewalls typically examine the source address/source port number, destination address/destination port number, and protocol of packets contained in the traffic and, based on user-defined rules, determine whether the packets should be allowed to pass its boundary in either direction. If a packet contains a source address that is not included in the firewall's authorized list, then that packet will be denied access to the networked computer system.
  • firewall security A significant limitation of firewall security is that because firewalls do not inspect the contents of individual packets for potentially damaging exploit code, firewalls are unable to prevent attacks from more sophisticated security threats such as worms, viruses and Trojan Horse attacks. Furthermore, because all packets entering a networked computer system must be checked by the firewall before passing through, firewalls represent both a single point of failure and a bottleneck, especially as network traffic increases.
  • IDS intrusion detection system
  • intrusion detection systems actively screen the contents of network traffic for malicious activity based on attack signatures
  • an intrusion detection system can only identify attacks having signatures stored in its database but cannot identify any other attacks.
  • intrusion detection systems merely identify that attack has occurred based on signature analysis but cannot prevent attacks from occurring.
  • those skilled in the art continuously devise attacks for defeating intrusion detection systems regardless of whether such attacks are included in the system's database. Thus, intrusion detection systems do not provide networked computer systems with comprehensive protection against malicious attacks.
  • ACL access control list
  • An entry in an ACL may include a particular file stored on the host computer and an indication that user Jeff has permission to read, write and execute the file, user Brian has the right to read the file only while user Mo has no permission to access the file at all.
  • Such unauthorized access to certain system calls allows the user to gain root or administrator control of the computer system with which the user can copy, delete and modify any system files and executablees thereby compromising the system.
  • a host-based access control list is not sufficient to prevent malicious attacks.
  • Tate may be used to identify changes in a file based on a CRC check
  • the Tate mechanism is inadequate for preventing attacks on the computer system.
  • nothing in Tate prevents an attacker from accessing the system and modifying a file, computing the CRC of the modified file and then appending a new file seal containing the CRC of the modified file.
  • the Tate mechanism subsequently compares a recalculated CRC to the CRC now contained in the file seal, the CRCs will match and the modifications made by the attacker to the file will go unnoticed.
  • Such a executable would allow an attacker to modify files for the purpose of compromising the system in a manner that bypasses the security mechanism of Tate.
  • Tate mechanism a comparison between the CRC in the file seal and a recalculated CRC for the file is only performed on startup and periodically during system operation. In between CRC checks, however, the Tate mechanism is unable to prevent an attacker from modifying system files. Thus, the mechanism disclosed in Tate is unable to prevent malicious attacks on a computer system.
  • Tripwire Yet another prior art host-based security mechanism is provided by Tripwire (www.tripwire. com) in which an administrator executes an application on a computer host that creates a baseline database of files and directories contained in the computer host.
  • the baseline includes attributes for each file including the file's size, create-times and checksum.
  • the application executes an application on a computer host that creates a baseline database of files and directories contained in the computer host.
  • the baseline includes attributes for each file including the file's size, create-times and checksum.
  • the application is again executed by the administrator, it compares the files in their current state against this baseline database, and identifies any changes, additions, or deletions made to the file. In this way, the administrator can detect any unauthorized changes made to files contained in the baseline and this information can be used to identify the occurrence and scope of an attack on the computer host.
  • the Tripwire mechanism may be able to detect changes in files that would indicate that an attack has occurred, the Tripwire mechanism is inadequate in preventing an attack on a computer host because the Tripwire mechanism only detects changes made to files and directories after they occur at which time an attacker may have already compromised the computer host. In addition, because file changes, and therefore possible intrusions, are only detected when the administrator periodically executes the application, the Tripwire mechanism does not provides intrusion detection at other times.
  • the prior art security architecture that includes firewalls, intrusion detection systems as well as host-based security mechanisms such as access control lists and file monitoring, cannot prevent malicious attacks against networked computer systems.
  • the present invention is directed to overcoming the drawbacks of the prior art.
  • a method for securely executing a program on a computer system and includes the step of calculating a baseline checksum for the executable. Next, a baseline checksum for all libraries the executable depends upon is calculated. The baseline checksum for the executable and the baseline checksum(s) for the respective libraries the executable depends upon are stored in the kernel. A request to execute the executable is then received. A checksum for the executable and checksum(s) for all libraries the executable depends upon are calculated.
  • the executable is executed. If a single baseline checksum for the executable does not equal the checksum for the executable or a single baseline checksum for any one of libraries does not equal the checksum stored in kernel memory, then the executable is not executed.
  • the executable has an inode and the at least one library file has an inode and the inode of the executable is stored into a binary cache if the at least one baseline checksum for the executable equals the at least one execution checksum for the executable.
  • the inode of the library (libraries) is (are) stored into a libary cache if the at least one baseline checksum for the library (libraries) equal(s) the at least one execution checksum for the library (libraries).
  • the inode of the executable is removed from the binary cache if the executable has been modified in any way.
  • the inode of a library is removed from the library cache if the at least one library file has been modified.
  • the another instance of the executable is executed if the inode of the another instance of the executable is equal to the inode of the executable in the first cache and the inode of the another at least one library file equals the inode of the at least one library file in the second cache.
  • the at least one execution checksum for the another instance of the executable is recalculated if the inode of the another instance of the executable is not equal to the inode of the executable in the first cache and the executable is executed if the at least one baseline checksum for the executable equals the at least one execution checksum for the another instance of the executable and the inode of the another at least one library file equals the inode of the at least one library file in the second cache.
  • the at least one execution checksum for the another at least one library file is recalculated if the inode of the another at least one library file is not equal to the inode of the at least one library file in the second cache and the executable is executed if the at least one execution checksum for the another at least one library file equals the at least one baseline checksum for the at least one library file and the inode of the another instance of the executable equals the inode of the executable in the first cache.
  • the at least one execution checksum for another instance of the executable is recalculated if the inode of the another instance of the executable is not equal to the inode of the executable in the first cache
  • the at least one execution checksum for the another at least one library file is recalculated if the inode of the another at least one library file is not equal to the inode of the at least one library file in the second cache and the executable is executed if the at least one execution checksum for the another instance of the executable equals the at least one baseline checksum for the executable and the at least one execution checksum for the another at least one library file equals the at least one baseline checksum for the at least one library file.
  • an operating system kernel executes on the computer system and the secure location is in the kernel.
  • the first cache and the second cache are in the kernel.
  • the at least one baseline checksum for the executable, the at least one execution checksum for the executable, the at least one baseline checksum for the at least one library file and the at least one execution checksum for the at least one library file are selected from the group of MD5, SHA1 and RTPEMD160.
  • the at least one baseline checksum for the executable, the at least one execution checksum for the executable, the at least one baseline checksum for the at least one library file and the at least one execution checksum for the at least one library file includes MD5, SHA1 and RIPEMD160.
  • a system for securely executing an executable on a computer system
  • the computer system includes an operating system kernel for providing services for the executable, and wherein the executable invokes a system call request for receiving services from the kernel.
  • the system includes a checksum list stored in the kernel, the checksum list including at least one baseline checksum for the executable.
  • an enhanced system call receiving the system call request, calculating at least one execution checksum for the executable and comparing the at least one baseline checksum for the executable and the at least one execution checksum for the executable.
  • the enhanced system call allows the system call request and if the at least one baseline checksum for the executable does not equal the at least one execution checksum for the executable, the enhanced system call denies the system call request.
  • the executable includes at least one dynamically linked library file
  • the checksum list includes at least one baseline checksum for the at least one library file
  • the enhanced system call calculates at least one execution checksum for the at least one library file and compares the at least one baseline checksum for the at least one library file and the at least one execution checksum for the at least one library file. If the at least one baseline checksum for the executable equals the at least one execution checksum for the executable and the at least one baseline checksum for the at least one library file equals the at least one execution checksum for the at least one library file, the enhanced system call allows said system call request.
  • the enhanced system call denies the system call request.
  • a method and system for preserving the integrity of files executed on a networked computer system thereby preventing attacks against the networked computer system along with backdoors.
  • FIG. 1 is a functional block diagram illustrating a system in which a executable executing on a computer system gains access to system services and resources provided by the computer system and an operating system kernel running on the computer system;
  • FIG. 2 is an intrusion prevention system 2 of the present invention
  • FIG. 3 is a flowchart illustrating the operation of the intrusion prevention system of FIG. 2.
  • executable 3 may be invoked by a user accessing the computer system from an access device 4 such as, by way of non-limiting example, a personal computer communicating with the computer system over the Internet.
  • an access device 4 such as, by way of non-limiting example, a personal computer communicating with the computer system over the Internet.
  • an access device 4 such as, by way of non-limiting example, a personal computer communicating with the computer system over the Internet.
  • any number of system resources such as, by way of non-limiting example, file access and communications may be required for executable 3 to accomplish its task.
  • executable 3 may be an Apache web server program running on the computer system that must be able to create, access and modify files stores on the computer system's hard disk.
  • executable 3 invokes one of a plurality of system calls 7 that is designed to request from kernel 5 the desired system service.
  • the system call gains access to kernel 5 via a kernel trap 9 that uses an interrupt mechanism to get the attention of kernel 5.
  • Kernel 5 then receives the system call request and executes the request accordingly. For example, if executable 3 requires access to a file named test.html, then an open(test.html) system
  • kernel 5 uses device drivers 13 to retrieve the file test, doc from a hardware device 11, such as a hard disk, and provides executable 3 with access to the file. Similarly, executable 3 may gain access to any other system resources it requires.
  • executable 3 may be modified by an attacker to cause executable 3 to request from kernel 5 any other system call that is not otherwise required by executable 3. The attacker may then gain access to system calls thereby enabling the attacker to compromise the computer system.
  • FIG. 2 there is shown the intrusion prevention system 2 of the present invention.
  • the intrusion prevention system 2 of the present invention. It is known that in most operating systems, there is a single system call that is called by every executable/program in order for the executable/program to receive services from the operating system kernel. For example, in most operating systems, an executable must call an exec system call to gain access to system services. Thus, a program/executable cannot perform required functions, such as open files or writes files, without invoking the exec system call.
  • the exec system call is modified to form an enhanced system call 17 that performs various features and functions of the present invention as well as the functions of the exec system call.
  • System 2 includes an administrator interface module 19 that receives from an administrator operating an administrator access device 6 a list of executables that the administrator desires to protect for unauthorized modifications. Upon receipt of a list of executables to protect, administrator interface module 19 sets a flag indicating to enhanced
  • enhanced system call 17 When intrusion prevention system 2 is initially activated, enhanced system call 17 first accesses administrator interface module 19 to determine which executables the administrator desires to protect. For each executable contained in administrator interface module, enhanced system call 17 calculates a checksum for the executable and stores the checksum, along with a file identifier such as, by way of non-limiting example, the inode of the file, in a checksum list 23 contained in kernel 15. In addition, for all executables that also require library files to execute, enhanced system call 17 also calculates the checksums for those library files and stores the library file checksums, along with a library inode, into checksum list 23. Thus, checksum list 23 contain baseline checksums for all of the executables, and related library files, that the administrator desires to protect.
  • enhanced system call 17 When a user operating access device 4 invokes a particular executable such as, for example, an Apache web server program, in order to access system services the executable calls the exec system call which, in system 2, has been modified to be enhanced system call 17, as described above. Before the executable is allowed to execute and is provided with access to system services, enhanced system call 17 first calculates a checksum on the executable and then compares this checksum to the baseline checksum previously stored in checksum list 23 in kernel 15 for the particular executable. If the baseline checksum in checksum list 23 matches the checksum of the calling executable presently calculated by enhanced system call 17 at execution time, then it is determined that the executable binary file has not been modified and is identical to the executable binary file in existence at the time intrusion prevention mechanism was activated.
  • a particular executable such as, for example, an Apache web server program
  • enhanced system call 17 determines whether the calling executable is a static binary or dynamically linked which would then require dynamically-linked library files for execution. If it is a dynamically linked binary, then enhanced system call 17 retrieves each required library and calculates checksum for each such library. Enhanced system call 17 then compares the calculated checksum for each library file with the baseline checksum for the corresponding library file that is stored in checksum 23. If the calculated execution checksum and the baseline checksum for each library file match, then it is determined that the library file(s) required by the calling executable have also not been modified. Thus, because the executable binary file and the library files required by the calling executable have not been modified, enhanced system call 17 allows the calling executable to execute and gain access to system services provided by kernel 15.
  • enhanced system call 17 stores the inode of the executable in a binary cache 25 and the inodes of any associated library files in library cache 21. For example, if it is determined by enhanced system call 17 that an Apache web server program has not been modified, then enhanced system call 17 stores in binary cache 25 the inode of the Apache web server program, for example inode no. 722 (which includes, among other things, the location on the disk that contain the Apache web server program, as is well-known in the art). If another instance of the Apache web server program having an inode no.
  • enhanced system call 17 first checks to see if inode no. 722 is stored in binary cache 25. If it is, this indicates that the integrity of the Apache web server program was previously verified. The inodes of dynamically-linked library files are similarly checked against inodes stored in library cache 21 and, if a match exists, enhanced system call 17 allows the new instance of the Apache web server program to execute.
  • inode number for a calling executable does not match the inode number for the same executable stored in binary cache 25.
  • an inode number of 815 corresponding to an Apache web server program is stored in binary cache 25. If a user creates a new program and names it Apache, this new program may be assigned an inode number of 765.
  • enhanced system call 17 will determine that the inode number (765) of the calling Apache program is different than the inode number (815) stored in binary cache 25 associated with the Apache web server program. Therefore, the calling Apache program will not be allowed to execute based on its inode. As a result, enhanced system call 17 must then calculate the checksum of the calling Apache program to determine whether it matches the baseline checksum for the Apache web server program stored in checksum list 23.
  • the checksum that enhanced system call 17 applies to program binary files and library files is the SHA1 function that calculates a 160-bit cryptographic checksum according to known techniques.
  • the checksum used may be an MD5 128-bit message digest or a RIPEMD160 160-bit cryptographic checksum.
  • enhanced system call 17 may be configured to apply more than one checksum to a executable binary file and/or library file.
  • more than one baseline checksum is stored in checksum list 23 for a given executable binary file and/or library file. In order for execution of the executable to be allowed, enhanced system call 17 must calculate the different checksums for the calling executable binary file and library files and determine that each of the calculated checksums match a corresponding baseline checksum.
  • system 2 includes a log file 27 that is in communications with enhanced system call 17.
  • System 2 may be configured so that any desired event may be logged to log file 27 including, but not limited to, successful executions of executables (and related library files) and/or illegal attempts to execute an executable (and related library files). Furthermore, when an illegal execution is identified, system 2 may be configured to take any
  • the system could likewise be configured to contact a firewall or router device to write a firewall wall rule or acl in the router to terminate the ability of the attacker to continue contact with that or any other server on the network.
  • Step 1 a request for kernel services issued by executable 3 is received by enhanced system call 17.
  • Step 2 it is determined whether the inode of the calling executable equals an inode of the executable contained in binary cache 25. If no matching inode is found then, in Step 9, a checksum of the calling executable is generated and, in Step 10, it is determined whether the calculated checksum equals the checksum for the executable contained in checksum list 23. If the checksums do not match, then execution of the executable is denied in Step 12.
  • Step 11 the inode for the calling executable is placed in binary cache 25 and, in Step 3, it is determined whether the calling executable requires dynamically-linked libraries. If the calling executable does not use any dynamically-linked libraries then execution of the calling executable is allowed in Step 8. If the calling executable does use dynamically-linked library(s) then, in Step 4, it is determined whether the inode(s) of the library file(s) equals the respective inode(s) for the library file(s) contained in library cache 21. If a matching inode is found, then execution of the executable is allowed in Step 8. If no matching inode is found then, in Step 5, a checksum of the library files are generated and, in Step 6, it is determined whether the calculated checksum equals the checksum for the library files stored in checksum list 23. If the checksums do not
  • Step 7 the inodes for the library files are placed in library cache 21 and execution of the executable is allowed in Step 8.
  • a method and system for preventing an intrusion into a networked computer system by preserving the integrity of executables executing on the networked computer.
  • the method and system preserves the integrity of such executables by only allowing an executable to execute if the checksums of the executable binary and any required dynamically-linked library files calculated at the time of execution of the executable equals baseline checksums for the executable and library files previously computed and stored in checksum list 23.
  • the method and system of the present invention prevents an attacker from compromising a networked computer system through the unauthorized modification of executable and library files.

Abstract

A method for securely executing a executable (3) on a computer system including the step of calculating a baseline checksum for the executable (3). Next, the baseline checksum for the executable (3) is stored in a secure location - the kernel (5). Next, a request to execute the executable (3) is received. Next, an execution checksum for the executable (3) is calculated. The executable (3) is executed if the baseline checksum for the executable (3) equals the execution checksum for the executable (3). The executable (3) is not executed if the baseline checksum for the executable (3) does not equal the execution checksum for the executable (3).

Description

SYSTEM AND METHOD FOR SECURELY EXECUTING A EXECUTABLE TO PRESERVE THE INTEGRITY OF FILES FROM UNAUTHORIZED ACCESS FOR NETWORK SECURITY
CROSS REFERENCE TO RELATED APPLICATIONS
This application claims the benefit of provisional application entitled "SYSTEM AND
METHOD FOR ENABLING ENHANCEMENT OF SECURITY ON A COMPUTER
OPERATING SYSTEM," filed April 10, 2000, bearing the Serial No. 60/ and provisional application entitled "SYSTEM AND METHOD FOR NETWORK SECURITY," filed on November 17, 2000, bearing the Serial No. 60/ , the contents of which are relied upon and incorporated herein by reference.
BACKGROUND
The following invention relates to a method and system for securing a computer system and, in particular, to a method and system for preserving the integrity of files contained in the computer system thereby securing the computer system from unauthorized access.
Computer systems are used to connect people and business worldwide. For example, the Internet, which consists of a web of networked computer systems, has become a ubiquitous platform for global communications and commerce. Because of the central role networked computer systems have in today's economy, it is vital that the security of networked computer
systems be maintained.
Everyday, new and different vulnerabilities are discovered in networked computer systems. These vulnerabilities may reside in any component of the computer system including its operating system or any application programs running on the computer system, such as a browser. Hackers may compromise a networked computer system using readily available tools that are designed to exploit the system's vulnerabilities. Because many of these tools require little technical knowledge to operate, a relative novice can initiate a successful attack.
A variety of prior art devices and techniques are used to provide security for networked computer systems. For example, a common prior art security device is a firewall that consists of hardware and software designed to keep unauthorized network traffic from accessing information within a networked computer system. Firewalls typically examine the source address/source port number, destination address/destination port number, and protocol of packets contained in the traffic and, based on user-defined rules, determine whether the packets should be allowed to pass its boundary in either direction. If a packet contains a source address that is not included in the firewall's authorized list, then that packet will be denied access to the networked computer system.
A significant limitation of firewall security is that because firewalls do not inspect the contents of individual packets for potentially damaging exploit code, firewalls are unable to prevent attacks from more sophisticated security threats such as worms, viruses and Trojan Horse attacks. Furthermore, because all packets entering a networked computer system must be checked by the firewall before passing through, firewalls represent both a single point of failure and a bottleneck, especially as network traffic increases.
Another common prior art security device is an intrusion detection system (IDS) that monitors network activity for attacks by identifying patterns of network traffic and comparing the patterns to known attack signatures stored in the IDS. Thus, if an attack is initiated against a networked computer system that deploys an IDS on its network and the signature of the particular attack is stored in database of the IDS, then the IDS will detect the attack and may take further actions such as logging the occurrence of the attack and notifying security personnel that an attack is in progress (or has occurred).
Although intrusion detection systems actively screen the contents of network traffic for malicious activity based on attack signatures, an intrusion detection system can only identify attacks having signatures stored in its database but cannot identify any other attacks. In addition, intrusion detection systems merely identify that attack has occurred based on signature analysis but cannot prevent attacks from occurring. Furthermore, those skilled in the art continuously devise attacks for defeating intrusion detection systems regardless of whether such attacks are included in the system's database. Thus, intrusion detection systems do not provide networked computer systems with comprehensive protection against malicious attacks.
Even a layered security approach that deploys both firewall technology as well as IDS technology cannot prevent attacks against networked computer systems. For example, if there exists a web server on a network protected by a firewall and an IDS, a new attack (i.e., unknown to the IDS) that is initiated to port 80 (which the firewall will allow) will gain access to the web server regardless of the presence of the firewall and the IDS. Thus, these prior art devices are inadequate to fully protect a networked computer system from attack.
In addition to perimeter security techniques such as firewalls and intrusion detection systems, there exist security mechanisms that reside on the host computer system itself. One such security mechanism is called an access control list ("ACL") that is a database maintained by the host computer operating system that describes the type of access each user of the host computer has to any program, service or file on the host computer. For example, an entry in an ACL may include a particular file stored on the host computer and an indication that user Jeff has permission to read, write and execute the file, user Brian has the right to read the file only while user Mo has no permission to access the file at all. Thus, by using an ACL to limit access to particular files to only those that require such access, an additional layer of security is provided for protecting sensitive information stored on the host computer.
While the prior art ACL security mechanism may restrict access to selected files and programs, the use of an ACL provides no security against misuse of files to which a user is granted or gains access. For example, if a user is granted access rights to an Apache web server program residing on a particular networked computer system, the user may compromise the system by causing a buffer overflow to occur in the Apache web server program using known techniques. Once such an overflow condition occurs, the user can then insert additional code into the Apache program allowing the user access to additional system calls that are not normally called by the Apache program. Such unauthorized access to certain system calls (for e.g., setuid in the Unix operating system) allows the user to gain root or administrator control of the computer system with which the user can copy, delete and modify any system files and executablees thereby compromising the system. Thus, even the use of a host-based access control list is not sufficient to prevent malicious attacks.
Another prior art host-based security mechanism is disclosed in U.S. Patent No. 5,991,774 issued to Tate et al. in which a sealfile program computes a Cyclic Redundancy Check (CRC) for selected files contained in a computer system and then incorporates each CRC in a file seal that is appended to the end of each respective file. Upon system startup and periodically during runtime of the system, the CRC of each file is recalculated and compared to the CRC contained in the file seal. Any discrepancies between the calculated CRC and the CRC contained in the file seal may indicate that the file has been corrupted or tampered with. Although the mechanism in Tate may be used to identify changes in a file based on a CRC check, the Tate mechanism is inadequate for preventing attacks on the computer system. First, nothing in Tate prevents an attacker from accessing the system and modifying a file, computing the CRC of the modified file and then appending a new file seal containing the CRC of the modified file. When the Tate mechanism subsequently compares a recalculated CRC to the CRC now contained in the file seal, the CRCs will match and the modifications made by the attacker to the file will go unnoticed. Such a executable would allow an attacker to modify files for the purpose of compromising the system in a manner that bypasses the security mechanism of Tate. Furthermore, in the Tate mechanism, a comparison between the CRC in the file seal and a recalculated CRC for the file is only performed on startup and periodically during system operation. In between CRC checks, however, the Tate mechanism is unable to prevent an attacker from modifying system files. Thus, the mechanism disclosed in Tate is unable to prevent malicious attacks on a computer system.
Yet another prior art host-based security mechanism is provided by Tripwire (www.tripwire. com) in which an administrator executes an application on a computer host that creates a baseline database of files and directories contained in the computer host. The baseline includes attributes for each file including the file's size, create-times and checksum. When the application is again executed by the administrator, it compares the files in their current state against this baseline database, and identifies any changes, additions, or deletions made to the file. In this way, the administrator can detect any unauthorized changes made to files contained in the baseline and this information can be used to identify the occurrence and scope of an attack on the computer host. Although the Tripwire mechanism may be able to detect changes in files that would indicate that an attack has occurred, the Tripwire mechanism is inadequate in preventing an attack on a computer host because the Tripwire mechanism only detects changes made to files and directories after they occur at which time an attacker may have already compromised the computer host. In addition, because file changes, and therefore possible intrusions, are only detected when the administrator periodically executes the application, the Tripwire mechanism does not provides intrusion detection at other times.
In summary, the prior art security architecture that includes firewalls, intrusion detection systems as well as host-based security mechanisms such as access control lists and file monitoring, cannot prevent malicious attacks against networked computer systems.
Accordingly, it is desirable to provide a system and method for preserving the integrity of files executed on a networked computer system thereby preventing attacks against the networked computer system.
SUMMARY OF THE INVENTION
The present invention is directed to overcoming the drawbacks of the prior art. Under the present invention a method is provided for securely executing a program on a computer system and includes the step of calculating a baseline checksum for the executable. Next, a baseline checksum for all libraries the executable depends upon is calculated. The baseline checksum for the executable and the baseline checksum(s) for the respective libraries the executable depends upon are stored in the kernel. A request to execute the executable is then received. A checksum for the executable and checksum(s) for all libraries the executable depends upon are calculated. If the baseline checksum for the executable stored in kernel memory is the same as the checksum generated for the executable and the baseline checksum(s) for the libraries the executable depends upon (stored in kernel memory) equals the execution checksum for the libraries, then the executable is executed. If a single baseline checksum for the executable does not equal the checksum for the executable or a single baseline checksum for any one of libraries does not equal the checksum stored in kernel memory, then the executable is not executed.
In an exemplary embodiment, the executable has an inode and the at least one library file has an inode and the inode of the executable is stored into a binary cache if the at least one baseline checksum for the executable equals the at least one execution checksum for the executable. The inode of the library (libraries) is (are) stored into a libary cache if the at least one baseline checksum for the library (libraries) equal(s) the at least one execution checksum for the library (libraries).
In another exemplary embodiment, the inode of the executable is removed from the binary cache if the executable has been modified in any way. Similarly, the inode of a library is removed from the library cache if the at least one library file has been modified.
In yet another exemplary embodiment, if a request to execute another instance of the executable is received wherein the another instance of the executable requires another at least one library file having an inode, the another instance of the executable is executed if the inode of the another instance of the executable is equal to the inode of the executable in the first cache and the inode of the another at least one library file equals the inode of the at least one library file in the second cache.
In still yet another exemplary embodiment, if a request to execute another instance of the executable is received, the at least one execution checksum for the another instance of the executable is recalculated if the inode of the another instance of the executable is not equal to the inode of the executable in the first cache and the executable is executed if the at least one baseline checksum for the executable equals the at least one execution checksum for the another instance of the executable and the inode of the another at least one library file equals the inode of the at least one library file in the second cache.
In an exemplary embodiment, if a request to execute another instance of the executable is received, the at least one execution checksum for the another at least one library file is recalculated if the inode of the another at least one library file is not equal to the inode of the at least one library file in the second cache and the executable is executed if the at least one execution checksum for the another at least one library file equals the at least one baseline checksum for the at least one library file and the inode of the another instance of the executable equals the inode of the executable in the first cache.
In another exemplary embodiment, if a request to execute another instance of the executable is received, the at least one execution checksum for another instance of the executable is recalculated if the inode of the another instance of the executable is not equal to the inode of the executable in the first cache, the at least one execution checksum for the another at least one library file is recalculated if the inode of the another at least one library file is not equal to the inode of the at least one library file in the second cache and the executable is executed if the at least one execution checksum for the another instance of the executable equals the at least one baseline checksum for the executable and the at least one execution checksum for the another at least one library file equals the at least one baseline checksum for the at least one library file.
In yet another exemplary embodiment, an operating system kernel executes on the computer system and the secure location is in the kernel. In still yet another exemplary embodiment, the first cache and the second cache are in the kernel.
In an exemplary embodiment, the at least one baseline checksum for the executable, the at least one execution checksum for the executable, the at least one baseline checksum for the at least one library file and the at least one execution checksum for the at least one library file are selected from the group of MD5, SHA1 and RTPEMD160.
In another exemplary embodiment, the at least one baseline checksum for the executable, the at least one execution checksum for the executable, the at least one baseline checksum for the at least one library file and the at least one execution checksum for the at least one library file includes MD5, SHA1 and RIPEMD160.
Under the present invention, a system is provided for securely executing an executable on a computer system wherein the computer system includes an operating system kernel for providing services for the executable, and wherein the executable invokes a system call request for receiving services from the kernel. The system includes a checksum list stored in the kernel, the checksum list including at least one baseline checksum for the executable. Also included is an enhanced system call receiving the system call request, calculating at least one execution checksum for the executable and comparing the at least one baseline checksum for the executable and the at least one execution checksum for the executable. If the at least one baseline checksum for the executable equals the at least one execution checksum for the executable, the enhanced system call allows the system call request and if the at least one baseline checksum for the executable does not equal the at least one execution checksum for the executable, the enhanced system call denies the system call request.
10 In an exemplary embodiment, the executable includes at least one dynamically linked library file, the checksum list includes at least one baseline checksum for the at least one library file and the enhanced system call calculates at least one execution checksum for the at least one library file and compares the at least one baseline checksum for the at least one library file and the at least one execution checksum for the at least one library file. If the at least one baseline checksum for the executable equals the at least one execution checksum for the executable and the at least one baseline checksum for the at least one library file equals the at least one execution checksum for the at least one library file, the enhanced system call allows said system call request. If the at least one baseline checksum for the executable does not equal the at least one execution checksum for the executable or the at least one baseline checksum for the at least one library file does not equal the at least one execution checksum for the at least one library file, the enhanced system call denies the system call request.
Accordingly, a method and system is provided for preserving the integrity of files executed on a networked computer system thereby preventing attacks against the networked computer system along with backdoors.
The invention accordingly comprises the features of construction, combination of elements and arrangement of parts that will be exemplified in the following detailed disclosure, and the scope of the invention will be indicated in the claims. Other features and advantages of the invention will be apparent from the description, the drawings and the claims.
DESCRIPTION OF THE DRAWINGS
For a fuller understanding of the invention, reference is made to the following description taken in conjunction with the accompanying drawings, in which:
11 FIG. 1 is a functional block diagram illustrating a system in which a executable executing on a computer system gains access to system services and resources provided by the computer system and an operating system kernel running on the computer system;
FIG. 2 is an intrusion prevention system 2 of the present invention; and FIG. 3 is a flowchart illustrating the operation of the intrusion prevention system of FIG. 2.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
Referring now to FIG. 1, there is shown a functional block diagram illustrating a system 1 in which an executable 3 executing on a computer system gains access to system services and resources provided by the computer system and an operating system kernel 5 running on the computer system. Executable 3 may be invoked by a user accessing the computer system from an access device 4 such as, by way of non-limiting example, a personal computer communicating with the computer system over the Internet. Depending on the purpose of executable 3, any number of system resources such as, by way of non-limiting example, file access and communications may be required for executable 3 to accomplish its task. For example, executable 3 may be an Apache web server program running on the computer system that must be able to create, access and modify files stores on the computer system's hard disk. To gain access to these system services, executable 3 invokes one of a plurality of system calls 7 that is designed to request from kernel 5 the desired system service. The system call gains access to kernel 5 via a kernel trap 9 that uses an interrupt mechanism to get the attention of kernel 5. Kernel 5 then receives the system call request and executes the request accordingly. For example, if executable 3 requires access to a file named test.html, then an open(test.html) system
12 call is issued by executable 3 to kernel 5. In response, kernel 5 uses device drivers 13 to retrieve the file test, doc from a hardware device 11, such as a hard disk, and provides executable 3 with access to the file. Similarly, executable 3 may gain access to any other system resources it requires.
As described above, executable 3 may be modified by an attacker to cause executable 3 to request from kernel 5 any other system call that is not otherwise required by executable 3. The attacker may then gain access to system calls thereby enabling the attacker to compromise the computer system.
Referring now to FIG. 2, there is shown the intrusion prevention system 2 of the present invention. It is known that in most operating systems, there is a single system call that is called by every executable/program in order for the executable/program to receive services from the operating system kernel. For example, in most operating systems, an executable must call an exec system call to gain access to system services. Thus, a program/executable cannot perform required functions, such as open files or writes files, without invoking the exec system call. In an exemplary embodiment in which system 2 is implemented for those operating systems that include the exec system call (or an equivalent), the exec system call is modified to form an enhanced system call 17 that performs various features and functions of the present invention as well as the functions of the exec system call.
System 2 includes an administrator interface module 19 that receives from an administrator operating an administrator access device 6 a list of executables that the administrator desires to protect for unauthorized modifications. Upon receipt of a list of executables to protect, administrator interface module 19 sets a flag indicating to enhanced
13 system call 17 that the administrator desires the intrusion prevention mechanism of the present invention to be activated.
When intrusion prevention system 2 is initially activated, enhanced system call 17 first accesses administrator interface module 19 to determine which executables the administrator desires to protect. For each executable contained in administrator interface module, enhanced system call 17 calculates a checksum for the executable and stores the checksum, along with a file identifier such as, by way of non-limiting example, the inode of the file, in a checksum list 23 contained in kernel 15. In addition, for all executables that also require library files to execute, enhanced system call 17 also calculates the checksums for those library files and stores the library file checksums, along with a library inode, into checksum list 23. Thus, checksum list 23 contain baseline checksums for all of the executables, and related library files, that the administrator desires to protect. When a user operating access device 4 invokes a particular executable such as, for example, an Apache web server program, in order to access system services the executable calls the exec system call which, in system 2, has been modified to be enhanced system call 17, as described above. Before the executable is allowed to execute and is provided with access to system services, enhanced system call 17 first calculates a checksum on the executable and then compares this checksum to the baseline checksum previously stored in checksum list 23 in kernel 15 for the particular executable. If the baseline checksum in checksum list 23 matches the checksum of the calling executable presently calculated by enhanced system call 17 at execution time, then it is determined that the executable binary file has not been modified and is identical to the executable binary file in existence at the time intrusion prevention mechanism was activated.
14 Once it has been determined that the execution checksum of the executable binary file equals the baseline checksum of the executable binary file stored in checksum list 23, enhanced system call 17 then determines whether the calling executable is a static binary or dynamically linked which would then require dynamically-linked library files for execution. If it is a dynamically linked binary, then enhanced system call 17 retrieves each required library and calculates checksum for each such library. Enhanced system call 17 then compares the calculated checksum for each library file with the baseline checksum for the corresponding library file that is stored in checksum 23. If the calculated execution checksum and the baseline checksum for each library file match, then it is determined that the library file(s) required by the calling executable have also not been modified. Thus, because the executable binary file and the library files required by the calling executable have not been modified, enhanced system call 17 allows the calling executable to execute and gain access to system services provided by kernel 15.
If, however, the checksum of the executable binary file or the checksums of any of the library files required by the executable do not match the corresponding baseline checksums stored in checksum 23, then this indicates that the executable binary file and/or the library files have been modified. Because modifications to either the executable binary file or the library files required by the executable may indicate a malicious attack on the computer system, execution of the executable will be denied by enhanced system call 17. Thus, an attacker will not be able to modify a executable binary file, such as an Apache web server program or any of the libraries Apache may depend upon, to gain access to restricted system calls because enhanced system call 17 will deny execution of any modified executable or library file.
15 In an exemplary embodiment, in the event enhanced system call 17 allows execution of a particular executable because there was a checksum match, enhanced system call 17 stores the inode of the executable in a binary cache 25 and the inodes of any associated library files in library cache 21. For example, if it is determined by enhanced system call 17 that an Apache web server program has not been modified, then enhanced system call 17 stores in binary cache 25 the inode of the Apache web server program, for example inode no. 722 (which includes, among other things, the location on the disk that contain the Apache web server program, as is well-known in the art). If another instance of the Apache web server program having an inode no. of 722 is invoked by a user, then before the checksum of the Apache web server program is calculated again, enhanced system call 17 first checks to see if inode no. 722 is stored in binary cache 25. If it is, this indicates that the integrity of the Apache web server program was previously verified. The inodes of dynamically-linked library files are similarly checked against inodes stored in library cache 21 and, if a match exists, enhanced system call 17 allows the new instance of the Apache web server program to execute.
If an inode number that matches the inode of a calling executable (or library file) is contained in binary cache 25 (or library cache 21), then this indicates that the calling executable (or library file) has not been modified because any modification to the executable binary file (or library file) results in the inode for the particular executable binary (or library file) being flushed from binary cache 25 (or library cache 21). In order for any modifications to a executable binary file to be effective, the modified executable file must be written to disk. Because writing a executable binary file to disk requires access to system services provided by the kernel, an attempt to modify the executable binary file will be received by enhanced system call 17, as described above. In response to the request to write to disk the modified file, enhanced system
16 call 17 forwards the request to the appropriate system calls and also flushes the inode number of the modified executable from binary cache 25. Because the inode number of the modified executable is no longer in binary cache 25, the next time the particular executable is invoked, enhanced system call 17 will not be able to use the presence of the executable' s inode number in binary cache 25 as an indication of the integrity of executable binary file. The integrity of the executable must then be checked by having enhanced system call 17 calculate the checksum of the executable and compare the calculated checksum to the baseline checksum stored in checksum list 23. A similar procedure applies for library files. Accordingly, whenever an inode in binary cache 25 (or library cache 21) matches the inode of a calling executable (or associated library file), this indicates that enhanced system call 17 has verified the integrity of the calling executable (and associated library files).
There are certain situations where the inode number for a calling executable does not match the inode number for the same executable stored in binary cache 25. For example, assume an inode number of 815 corresponding to an Apache web server program is stored in binary cache 25. If a user creates a new program and names it Apache, this new program may be assigned an inode number of 765. When this new program is invoked, enhanced system call 17 will determine that the inode number (765) of the calling Apache program is different than the inode number (815) stored in binary cache 25 associated with the Apache web server program. Therefore, the calling Apache program will not be allowed to execute based on its inode. As a result, enhanced system call 17 must then calculate the checksum of the calling Apache program to determine whether it matches the baseline checksum for the Apache web server program stored in checksum list 23.
17 A similar situation arises where a executable binary file is moved to a different location on disk in which case the executable binary file, if on a new mount point, is assigned a new inode number. If a user then invokes the executable after the new inode number has been assigned to the executable binary file, then the new inode number of the executable will not match the inode number for the executable previously stored in binary cache 25. As in the previous case, here too enhanced system call 17 must then calculate the checksum of the invoked executable to determine whether it matches the baseline checksum for the executable stored in checksum list 23.
In an exemplary embodiment, the checksum that enhanced system call 17 applies to program binary files and library files is the SHA1 function that calculates a 160-bit cryptographic checksum according to known techniques. Alternatively, the checksum used may be an MD5 128-bit message digest or a RIPEMD160 160-bit cryptographic checksum. In an exemplary embodiment, enhanced system call 17 may be configured to apply more than one checksum to a executable binary file and/or library file. In this embodiment, more than one baseline checksum is stored in checksum list 23 for a given executable binary file and/or library file. In order for execution of the executable to be allowed, enhanced system call 17 must calculate the different checksums for the calling executable binary file and library files and determine that each of the calculated checksums match a corresponding baseline checksum.
In an exemplary embodiment, system 2 includes a log file 27 that is in communications with enhanced system call 17. System 2 may be configured so that any desired event may be logged to log file 27 including, but not limited to, successful executions of executables (and related library files) and/or illegal attempts to execute an executable (and related library files). Furthermore, when an illegal execution is identified, system 2 may be configured to take any
18 number of actions such as, by way of non-limiting example, locking the account of the transgressor, terminating any sessions the transgressing account has open on the computer system and/or notifying an administrator of the illegal execution attempt. The system could likewise be configured to contact a firewall or router device to write a firewall wall rule or acl in the router to terminate the ability of the attacker to continue contact with that or any other server on the network.
Referring now to FIG. 3, there is shown a flowchart illustrating the operation of intrusion prevention system 2 of the present invention. Initially, in Step 1, a request for kernel services issued by executable 3 is received by enhanced system call 17. Next, in Step 2, it is determined whether the inode of the calling executable equals an inode of the executable contained in binary cache 25. If no matching inode is found then, in Step 9, a checksum of the calling executable is generated and, in Step 10, it is determined whether the calculated checksum equals the checksum for the executable contained in checksum list 23. If the checksums do not match, then execution of the executable is denied in Step 12. If the checksums do match then, in Step 11, the inode for the calling executable is placed in binary cache 25 and, in Step 3, it is determined whether the calling executable requires dynamically-linked libraries. If the calling executable does not use any dynamically-linked libraries then execution of the calling executable is allowed in Step 8. If the calling executable does use dynamically-linked library(s) then, in Step 4, it is determined whether the inode(s) of the library file(s) equals the respective inode(s) for the library file(s) contained in library cache 21. If a matching inode is found, then execution of the executable is allowed in Step 8. If no matching inode is found then, in Step 5, a checksum of the library files are generated and, in Step 6, it is determined whether the calculated checksum equals the checksum for the library files stored in checksum list 23. If the checksums do not
19 match, then execution of the executable is denied in Step 12. If the checksums do match then, in Step 7, the inodes for the library files are placed in library cache 21 and execution of the executable is allowed in Step 8.
Accordingly, a method and system is provided for preventing an intrusion into a networked computer system by preserving the integrity of executables executing on the networked computer. The method and system preserves the integrity of such executables by only allowing an executable to execute if the checksums of the executable binary and any required dynamically-linked library files calculated at the time of execution of the executable equals baseline checksums for the executable and library files previously computed and stored in checksum list 23. By ensuring that executable binaries and library files cannot execute if modified, the method and system of the present invention prevents an attacker from compromising a networked computer system through the unauthorized modification of executable and library files.
Based on the above description, it will be obvious to one of ordinary skill to implement the system and methods of the present invention in one or more computer programs that are executable on a programmable system including at least one programmable executable or coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. Each computer program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language if desired; and in any case, the language may be a compiled or interpreted language. Suitable executables include, by way of example, both general and special purpose microexecutables. Furthermore, alternate embodiments of the invention that implement the system in hardware, firmware or a combination of both hardware and software, as well as
20 distributing modules and/or data in a different fashion will be apparent to those skilled in the art and are also within the scope of the invention.
It will thus be seen that the objects set forth above, among those made apparent from the preceding description, are efficiently attained and, since certain changes may be made in carrying out the above executable, in a described product, and in the construction set forth without departing from the spirit and scope of the invention, it is intended that all matter contained in the above description shown in the accompanying drawing shall be interpreted as illustrative and not in a limiting sense.
It is also to be understood that the following claims are intended to cover all of the generic and specific features of the invention herein described, and all statements of the scope of the invention, which, as a matter of language, might be said to fall therebetween.
21

Claims

1. A method for securely executing a executable on a computer system, comprising the steps of: calculating a baseline checksum for said executable; storing said baseline checksum for said executable in a secure location; receiving a request to execute said executable; calculating an execution checksum for said executable; executing said executable if said baseline checksum for said executable equals said execution checksum for said executable; and not executing said executable if said baseline checksum for said executable does not equal said execution checksum for said executable.
2. The method of claim 1, wherein said executable includes at least one dynamically linked library file, wherein the method further comprises the steps of: calculating a baseline checksum for said at least one library file; storing said baseline checksum for said at least one library file in said secure location; wherein the step of calculating an execution checksum includes the step of: calculating an execution checksum for said at least one library file; wherein the step of executing said executable includes the step of: executing said executable if said baseline checksum for said executable equals said execution checksum for said executable and said baseline checksum for said at least one library file equals said execution checksum for said at least one library file; and wherein the step of not executing said executable includes the step of:
22 not executing said executable if said baseline checksum for said executable does not equal said execution checksum for said executable or said baseline checksum for said at least one library file does not equal said execution checksum for said at least one library file.
3. The method of claim 2, wherein said baseline checksum for said executable, said execution checksum for said executable, said baseline checksum for said at least one library file and said execution checksums for said at least one library file is selected from the group of MD5, SHAl and RIPEMD160.
4. The method of claim 2, wherein said executable has an inode and said at least one library file has an inode and wherein the method further comprising the steps of: storing said inode of said executable into a binary cache if said baseline checksum for said executable equals said execution checksum for said executable; and storing said inode of said at least one library file into a library cache if said baseline checksum for said at least one library file equals said execution checksum for said at least one library file.
5. The method of claim 4, further comprising the steps of: removing said inode of said executable from said first cache if said executable has been modified; and removing said inode of said at least one library file from said second cache if said at least one library file has been modified.
6. The method of claim 5, further comprising the steps of: receiving a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance of said executable requiring another at least one library file having an inode;
23 executing said another instance of said executable if said inode of said another instance of said executable equals said inode of said executable in said first cache and said inode of said another at least one library file equals said inode of said at least one library file in said second cache; and not executing said another instance of said executable if said inode of said another instance of said executable does not equal said inode of said executable in said first cache or said inode of said another at least one library file does not equal said inode of said at least one library file in said second cache.
7. The method of claim 5, further comprising the steps of: receiving a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance if said executable requiring another at least one library file having an inode; recalculating said execution checksum for said another instance of said executable if said inode of said another instance of said executable is not equal to said inode of said executable in said first cache; executing said another instance of said executable if said baseline checksum for said executable equals said execution checksum for said another instance of said executable and said inode of said another at least one library file equals said inode of said at least one library file in said second cache; and not executing said another instance of said executable if said baseline checksum for said executable does not equal said execution checksum for said another instance of said executable or said inode of said another at least one library file does not equal said inode of said at least one library file in said second cache.
24
8. The method of claim 5, further comprising the steps of: receiving a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance if said executable requiring another at least one library file having an inode; recalculating said execution checksum for said another at least one library file if said inode of said another at least one library file is not equal to said inode of said at least one library file in said second cache; executing said another instance of said executable if said execution checksum for said another at least one library file equals said baseline checksum for said at least one library file and said inode of said another instance of said executable equals said inode of said executable in said first cache; and not executing said another instance of said executable if said execution checksum for said another at least one library file does not equal said baseline checksum for said at least one library file or said inode of said another instance of said executable does not equal said inode of said executable in said first cache
9. The method of claim 5, further comprising the steps of: receiving a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance of said executable requiring another at least one library file having an inode; recalculating said execution checksum for said another instance of said executable if said inode of said another instance of said executable is not equal to said inode of said executable in said first cache;
25 recalculating said execution checksum for said another at least one library file if said inode of said another at least one library file is not equal to said inode of said at least one library file in said second cache; and executing said another instance of said executable if said execution checksum for said another instance of said executable equals said baseline checksum for said executable and said execution checksum for said another at least one library file equals said baseline checksum for said at least one library file.
10. The method of claim 1, wherein an operating system kernel executes on said computer system and said secure location is in said kernel.
11. The method of claim 4, wherein an operating system kernel executes on said computer system and wherein said first cache and said second cache are in said kernel.
12. The method of claim 1, wherein the step of calculating a baseline checksum includes the step of: calculating a plurality of baseline checksums for said executable; wherein the step of calculating an execution checksum includes the step of: calculating a plurality of execution checksums for said executable; wherein the step of executing said executable includes the step of: executing said executable if each of said plurality of baseline checksums for said executable equals a respective one of said plurality of execution checksums for said executable; and wherein the step of not executing said executable includes the step of: not executing said executable if one of said plurality of baseline checksums for said executable does not equal a respective one of said plurality of execution checksums for said executable.
26
13. The method of claim 12, wherein said plurality of baseline checksums for said executable and said plurality of said execution checksums for said executable includes MD5, SHAl and RIPEMD160.
14. The method of claim 2, wherein the step of calculating a baseline checksum for said at least one library file includes the step of: calculating a plurality of baseline checksums for said at least one library file; wherein the step of calculating an execution checksum includes the step of: calculating a plurality execution checksums for said at least one library file; wherein the step of executing said executable includes the step of: executing said executable if each of said plurality of baseline checksums for said executable equals a respective one of said plurality of execution checksums for said executable and each of said plurality of baseline checksums for said at least one library file equals a respective one of said execution checksums for said at least one library file; and wherein the step of not executing said executable includes the step of: not executing said executable if one of said plurality of baseline checksums for said executable does not equal a respective one of said plurality of execution checksums for said executable or one of said plurality of baseline checksums for said at least one library file does not equal a respective one of said execution checksums for said at least one library file.
15. The method of claim 14, wherein said plurality of baseline checksums for said at least one library file and said plurality of said execution checksums for said at least one library file includes MD5, SHAl and IPEMD160.
16. The method of claim 1, further comprising the step of: logging an event to a log file.
27
17. The method of claim 16, wherein said event is a successful execution of said executable.
18. The method of claim 16, wherein said event is a unsuccessful execution of said executable.
19. The method of claim 1, further comprising the step of: receiving a list of executables for securely executing on said computer system.
20. A method for securely executing a executable on a computer system, said executable including at least one dynamically linked library file comprising the steps of: calculating at least one baseline checksum for said executable; calculating at least one baseline checksum for said at least one library file; storing said at least one baseline checksum for said executable and said at least one baseline checksum for said at least one library file in a secure location; receiving a request to execute said executable; calculating at least one execution checksum for said executable; calculating at least one execution checksum for said at least one library file; executing said executable if said at least baseline checksum for said executable equals said at least one execution checksum for said executable and said at least one baseline checksum for said at least one library file equals said at least one execution checksum for said at least one library file; and not executing said executable if said at least baseline checksum for said executable does not equal said execution checksum for said executable or said at least one baseline checksum for said at least one library file does not equal said at least one execution checksum for said at least one library file.
28
21. The method of claim 20, wherein said executable has an inode and said at least one library file has an inode and wherein the method further comprising the steps of: storing said inode of said executable into a first cache if said at least one baseline checksum for said executable equals said at least one execution checksum for said executable; and storing said inode of said at least one library file into a second cache if said at least one baseline checksum for said at least one library file equals said at least one execution checksum for said at least one library file.
22. The method of claim 21, further comprising the steps of: removing said inode of said executable from said first cache if said executable has been modified; and removing said inode of said at least one library file from said second cache if said at least one library file has been modified.
23. The method of claim 22, further comprising the steps of: receiving a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance of said executable requiring another at least one library file having an inode; executing said another instance of said executable if said inode of said another instance of said executable equals said inode of said executable in said first cache and said inode of said another at least one library file equals said inode of said at least one library file in said second cache; and not executing said another instance of said executable if said inode of said another instance of said executable does not equal said inode of said executable in said first cache or said
29 inode of said another at least one library file does not equal said inode of said at least one library file in said second cache
24. The method of claim 22, further comprising the steps of: receiving a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance if said executable requiring another at least one library file having an inode; recalculating said at least one execution checksum for said another instance of said executable if said inode of said another instance of said executable is not equal to said inode of said executable in said first cache; executing said another instance of said executable if said at least one baseline checksum for said executable equals said at least one execution checksum for said another instance of said executable and said inode of said another at least one library file equals said inode of said at least one library file in said second cache; and not executing said another instance of said executable if said at least one baseline checksum for said executable does not equal said at least one execution checksum for said another instance of said executable or said inode of said another at least one library file does not equal said inode of said at least one library file in said second cache.
25. The method of claim 22, further comprising the steps of: receiving a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance if said executable requiring another at least one library file having an inode;
30 recalculating said at least one execution checksum for said another at least one library file if said inode of said another at least one library file is not equal to said inode of said at least one library file in said second cache; executing said another instance of said executable if said at least one execution checksum for said another at least one library file equals said at least one baseline checksum for said at least one library file and said inode of said another instance of said executable equals said inode of said executable in said first cache; and not executing said another instance of said executable if said at least one execution checksum for said another at least one library file does not equal said at least one baseline checksum for said at least one library file or said inode of said another instance of said executable does not equal said inode of said executable in said first cache
26. The method of claim 22, further comprising the steps of: receiving a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance if said executable requiring another at least one library file having an inode; recalculating said at least one execution checksum for said another instance of said executable if said inode of said another instance of said executable is not equal to said inode of said executable in said first cache; recalculating said at least one execution checksum for said another at least one library file if said inode of said another at least one library file is not equal to said inode of said at least one library file in said second cache; and executing said another instance of said executable if said at least one execution checksum for said another instance of said executable equals said at least one baseline checksum for said
31 executable and said at least one execution checksum for said another at least one library file equals said at least one baseline checksum for said at least one library file.
27. The method of claim 20, wherein an operating system kernel executes on said computer system and said secure location is in said kernel.
28. The method of claim 22, wherein an operating system kernel executes on said computer system and wherein said first cache and said second cache are in said kernel.
29. The method of claim 20, wherein said at least one baseline checksum for said executable, said at least one execution checksum for said executable, said at least one baseline checksum for said at least one library file and said at least one execution checksums for said at least one library file is selected from the group of MD5, SHAl and RTPEMD160.
30. The method of claim 20, wherein said at least one baseline checksum for said executable, said at least one execution checksum for said executable, said at least one baseline checksum for said at least one library file and said at least one execution checksums for said at least one library file includes MD5, SHAl and RIPEMD160.
31. The method of claim 20, further comprising the step of: logging an event to a log file.
32. The method of claim 31, wherein said event is a successful execution of said executable.
33. The method of claim 31, wherein said event is a unsuccessful execution of said executable.
34. The method of claim 20, further comprising the step of: receiving a list of executables for securely executing on said computer system.
35. A system for securely executing a executable on a computer system, wherein said computer system includes an operating system kernel for providing services for said executable,
32 and wherein said executable invokes a system call request for receiving services from said kernel, the system comprising: a checksum list stored in said kernel, said checksum list including at least one baseline checksum for said executable; and an enhanced system call receiving said system call request, said enhanced system call calculating at least one execution checksum for said executable and comparing said at least one baseline checksum for said executable and said at least one execution checksum for said executable; wherein if said at least one baseline checksum for said executable equals said at least one execution checksum for said executable, said enhanced system call allows said system call request and if said at least one baseline checksum for said executable does not equal said at least one execution checksum for said executable, said enhanced system call denies said system call request.
36. The system of claim 35, wherein said executable includes at least one dynamically linked library file, said checksum list includes at least one baseline checksum for said at least one library file and said enhanced system call calculates at least one execution checksum for said at least one library file and compares said at least one baseline checksum for said at least one library file and said at least one execution checksum for said at least one library file, wherein if said at least one baseline checksum for said executable equals said at least one execution checksum for said executable and said at least one baseline checksum for said at least one library file equals said at least one execution checksum for said at least one library file, said enhanced system call allows said system call request, and if said at least one baseline checksum for said executable does not equal said at least one execution checksum for said executable or said at
33 least one baseline checksum for said at least one library file does not equal said at least one execution checksum for said at least one library file, said enhanced system call denies said system call request.
37. The system of claim 36, wherein said executable has an inode and said at least one library file has an inode and further comprising a first cache and a second cache, wherein said enhanced system call stores said inode of said executable into said first cache if said at least one baseline checksum for said executable equals said at least one execution checksum for said executable and said enhanced system call stores said inode of said at least one library file into said second cache if said at least one baseline checksum for said at least one library file equals said at least one execution checksum for said at least one library file.
38. The system of claim 37, wherein said enhanced system call removes said inode for said executable from said first cache if said enhanced system call receives a request to modify said executable and said enhanced system call removes said inode of said at least one library file from said second cache if said enhanced system call receives a request to modify said at least one library file.
39. The system of claim 38, wherein said enhanced system call receives a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance of said executable requiring another at least one library file having an inode, and wherein said enhanced system call allows said another instance of said executable to execute if said inode of said another instance of said executable equals said inode of said executable in said first cache and said inode of said another at least one library file equals said inode of said library file in said second cache and said enhanced system call does not allow said another instance of said executable to execute if said inode of said another instance of said
34 executable does not equal said inode of said executable in said first cache or said inode of said another at least one library file does not equal said inode of said library file in said second cache
40. The system of claim 38, wherein said enhanced system call receives a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance if said executable requiring another at least one library file having an inode, wherein said enhanced system call recalculates said at least one execution checksum for said another instance of said executable if said inode of said another instance of said executable is not equal to said inode of said executable in said first cache, wherein said enhanced system call allows said another instance of said executable to execute if said at least one baseline checksum for said executable equals said at least one execution checksum for said another instance of said executable and said inode of said another at least one library file equals said inode of said library file in said second cache and said enhanced system call does not allow said another instance of said executable to execute if said at least one baseline checksum for said executable does not equal said at least one execution checksum for said another instance of said executable or said inode of said another at least one library file does not equal said inode of said library file in said second cache.
41. The system of claim 38, wherein said enhanced system call receives a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance if said executable requiring another at least one library file having an inode, wherein said enhanced system call recalculates said at least one execution checksum for said another at least one library file if said inode of said another at least one library file is not equal to said inode of said library file in said second cache, wherein said enhanced system call allows said another instance of said executable to execute if said at least one execution checksum for
35 said another at least one library file equals said at least one baseline checksum for said at least one library file and said inode of said another instance of said executable equals said inode of said executable in said first cache and wherein said enhanced system call does not said another instance of said executable to execute if said at least one execution checksum for said another at least one library file does not equal said at least one baseline checksum for said at least one library file or said inode of said another instance of said executable does not equal said inode of said executable in said first cache.
42. The system of claim 38, wherein said enhanced system call receives a request to execute another instance of said executable, said another instance of said executable having an inode, said another instance if said executable requiring another at least one library file having an inode, wherein said enhanced system call recalculates said at least one execution checksum for said another instance of said executable if said inode of said another instance of said executable is not equal to said inode of said executable in said first cache, wherein said enhanced system call recalculates said at least one execution checksum for said another at least one library file if said inode of said another at least one library file is not equal to said inode of said library file in said second cache, and wherein said enhanced system call allows said another instance of said executable to execute if said at least one execution checksum for said another instance of said executable equals said at least one baseline checksum for said executable and said at least one execution checksum for said another at least one library file equals said at least one baseline checksum for said at least one library file.
43. The system of claim 37, wherein said first cache and said second cache are in said kernel.
44. The system of claim 35, wherein said at least one baseline checksum for said executable, said at least one execution checksum for said executable, said at least one baseline checksum for
36 said at least one library file and said at least one execution checksums for said at least one library file is selected from the group of MD5, SHAl and R--PEMD160.
45. The system of claim 35, wherein said at least one baseline checksum for said executable, said at least one execution checksum for said executable, said at least one baseline checksum for said at least one library file and said at least one execution checksums for said at least one library file includes MD5, SHAl and RIPEMD160.
46. The system of claim 35, further comprising a log file wherein said enhanced system call logs an event to said log file.
47. The system of claim 46, wherein said event is a successful execution of said executable.
48. The system of claim 46, wherein said event is a unsuccessful execution of said executable.
49. The system of claim 35, further comprising an administrator interface module for receiving a list of executables to be securely executed on said computer system.
37
PCT/US2001/011552 2001-04-10 2001-04-10 System and method for securely executing a executable to preserve the integrity of files from unauthorized access for network security WO2002084939A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/US2001/011552 WO2002084939A1 (en) 2001-04-10 2001-04-10 System and method for securely executing a executable to preserve the integrity of files from unauthorized access for network security

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2001/011552 WO2002084939A1 (en) 2001-04-10 2001-04-10 System and method for securely executing a executable to preserve the integrity of files from unauthorized access for network security

Publications (1)

Publication Number Publication Date
WO2002084939A1 true WO2002084939A1 (en) 2002-10-24

Family

ID=21742485

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2001/011552 WO2002084939A1 (en) 2001-04-10 2001-04-10 System and method for securely executing a executable to preserve the integrity of files from unauthorized access for network security

Country Status (1)

Country Link
WO (1) WO2002084939A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2007123492A1 (en) * 2006-04-25 2007-11-01 Khee Seng Chua Method of safeguarding against malicious software (malware)
US20200411047A1 (en) * 2019-06-25 2020-12-31 International Business Machines Corporation Detecting electronic system modification

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5440723A (en) * 1993-01-19 1995-08-08 International Business Machines Corporation Automatic immune system for computers and computer networks
US5666411A (en) * 1994-01-13 1997-09-09 Mccarty; Johnnie C. System for computer software protection
US5797015A (en) * 1995-04-18 1998-08-18 Pitney Bowes Inc. Method of customizing application software in inserter systems
US5950205A (en) * 1997-09-25 1999-09-07 Cisco Technology, Inc. Data transmission over the internet using a cache memory file system
US6026474A (en) * 1996-11-22 2000-02-15 Mangosoft Corporation Shared client-side web caching using globally addressable memory
US6058389A (en) * 1997-10-31 2000-05-02 Oracle Corporation Apparatus and method for message queuing in a database system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5440723A (en) * 1993-01-19 1995-08-08 International Business Machines Corporation Automatic immune system for computers and computer networks
US5666411A (en) * 1994-01-13 1997-09-09 Mccarty; Johnnie C. System for computer software protection
US5797015A (en) * 1995-04-18 1998-08-18 Pitney Bowes Inc. Method of customizing application software in inserter systems
US6026474A (en) * 1996-11-22 2000-02-15 Mangosoft Corporation Shared client-side web caching using globally addressable memory
US5950205A (en) * 1997-09-25 1999-09-07 Cisco Technology, Inc. Data transmission over the internet using a cache memory file system
US6058389A (en) * 1997-10-31 2000-05-02 Oracle Corporation Apparatus and method for message queuing in a database system

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2007123492A1 (en) * 2006-04-25 2007-11-01 Khee Seng Chua Method of safeguarding against malicious software (malware)
US20200411047A1 (en) * 2019-06-25 2020-12-31 International Business Machines Corporation Detecting electronic system modification

Similar Documents

Publication Publication Date Title
US11575704B2 (en) Real-time detection of and protection from malware and steganography in a kernel mode
US8195953B1 (en) Computer program with built-in malware protection
US8272059B2 (en) System and method for identification and blocking of malicious code for web browser script engines
US7673137B2 (en) System and method for the managed security control of processes on a computer system
US7665139B1 (en) Method and apparatus to detect and prevent malicious changes to tokens
USRE43987E1 (en) System and method for protecting a computer system from malicious software
US6684329B1 (en) System and method for increasing the resiliency of firewall systems
JP4629332B2 (en) Status reference monitor
US20160180081A1 (en) Systems And Methods For Security In Computer Systems
US7707620B2 (en) Method to control and secure setuid/gid executables and processes
Chien et al. Blended attacks exploits, vulnerabilities and buffer-overflow techniques in computer viruses
US7533413B2 (en) Method and system for processing events
US20030159070A1 (en) System and method for comprehensive general generic protection for computers against malicious programs that may steal information and/or cause damages
US20080077994A1 (en) Trusted enclave for a computer system
Schmid et al. Protecting data from malicious software
KR20060092832A (en) Containment of worms
Deng et al. Lexical analysis for the webshell attacks
Turaev et al. Prevention of ransomware execution in enterprise environment on windows os: Assessment of application whitelisting solutions
WO2022185031A1 (en) Methods and systems for detecting and blocking malicious actions in an operating system
JP4638494B2 (en) Computer data protection methods
WO2002084939A1 (en) System and method for securely executing a executable to preserve the integrity of files from unauthorized access for network security
Wu et al. Self-healing spyware: detection, and remediation
US20230418933A1 (en) Systems and methods for folder and file sequestration
Endsuleit et al. A security analysis on jade (-s) v. 3.2
Bishop et al. Results-oriented security

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CR CU CZ DE DK DM DZ EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
121 Ep: the epo has been informed by wipo that ep was designated in this application
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 69(1) EPC

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP