Problems of interaction of the malicious code and protection programs in architecture of modern operating systems

The paper considers the interaction between malware and security software environments of modern operating systems. In particular, a number of aspects which are related to the software module that provides an opportunity for sustainable and undetectable presence of the offender in computer systems is considered. A number of statements is made about relationships between the technologies used in security software and ensuring “invisibility” of the executable malicious code. The possibility of undetectable rootkit presence in modern security software is shown on practice. In addition, the mechanism of system calls and drivers subsystem of Windows NT is analyzed. Furthermore, necessary practical requirements for implementation of security software are developed. The model of random restriction of malicious software for security software is constructed.