Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.

/pdf/anomaly-detection-a-survey-1x33g9m0a3.pdf

Anomaly detection: A survey

This paper presents an overview of the field of software systems requirements engineering (RE). It describes the main areas of RE practice, and highlights some key open research issues for the future.

/pdf/requirements-engineering-a-roadmap-3lcu5ikya8.pdf

Requirements engineering: a roadmap

Intrusion detection systems rely on a wide variety of observable data to distinguish between legitimate and illegitimate activities. We study one such observable-sequences of system calls into the kernel of an operating system. Using system-call data sets generated by several different programs, we compare the ability of different data modeling methods to represent normal behavior accurately and to recognize intrusions. We compare the following methods: simple enumeration of observed sequences; comparison of relative frequencies of different sequences; a rule induction technique; and hidden Markov models (HMMs). We discuss the factors affecting the performance of each method and conclude that for this particular problem, weaker methods than HMMs are likely sufficient.

/pdf/detecting-intrusions-using-system-calls-alternative-data-26bjpapj0b.pdf

Detecting intrusions using system calls: alternative data models

Will Tracz, our esteemed editor and Used-Program salesman, has written an entertaining, non-technical book dealing with the practice (and lack of) of software reuse. Its a collection of essays, mostly rehashed (reused?) and updated from various columns and papers published over the years.. Its a short (a bit over 200 pages) easy reading and enjoyable book (I read most of it in one sitting). Some of the essays discuss what was printed in the past and a discussion of the current status of the points.

Safeware: System Safety and Computers

The paper focuses on intrusion detection and countermeasures with respect to widely-used operating systems and networks. The design and architecture of an intrusion detection system built from distributed agents is proposed to implement an intelligent system on which data mining can be performed to provide global, temporal views of an entire networked system. A starting point for agent intelligence in the system is the research into the use of machine learning over system call traces from the privileged sendmail program on UNIX. The authors use a rule learning algorithm to classify the system call traces for intrusion detection purposes and show the results.

/pdf/intelligent-agents-for-intrusion-detection-vf31xaphc0.pdf

Intelligent agents for intrusion detection

Lightweight agents for intrusion detection

The use of software fault trees for requirements identification and analysis in an Intrusion Detection System (IDS) is described.

A software fault tree approach to requirements analysis of an intrusion detection system

https://faculty.ist.psu.edu/vhonavar/Papers/helmer02.pdf

Automated discovery of concise predictive rules for intrusion detection

The integration of Software Fault Tree (SFT), which describes intrusions and Coloured Petri Nets (CPNs) that specifies design, is examined for an Intrusion Detection System (IDS). The IDS under development is a collection of mobile agents that detect, classify, and correlate the system and network activities. SFTs, augmented with nodes that describe trust, temporal and contextual relationships, are used to describe intrusions. CPNs for intrusion detection are built using CPN templates created from the augmented SFTs. Hierarchical CPNs are created to detect critical stages of intrusions. The agentbased implementation of the IDS is then constructed from the CPNs. Examples of intrusions and descriptions of the prototype implementation are used to demonstrate how the CPN approach has been used in the development of the IDS. The main contribution of this paper is an approach to systematic specification, design and implementation of an IDS; Innovations include (1) using stages of intrusions to structure the specification and design of the IDS; (2) augmentation of SFT with trust, temporal and contextual nodes to model intrusions; (3) algorithmic construction of CPNs from augmented SFT; and (4) generation of mobile agents from CPNs.

/pdf/software-fault-tree-and-coloured-petri-net-based-wa4m9r8d88.pdf

Guy Helmer

Papers

Intelligent agents for intrusion detection

Lightweight agents for intrusion detection

A software fault tree approach to requirements analysis of an intrusion detection system

Automated discovery of concise predictive rules for intrusion detection

Software fault tree and coloured Petri net based specification, design and implementation of agent-based intrusion detection systems