Artificial ignorance, in network behavior anomaly detection (NBAD), is the strategic practice of disregarding noisy data in log files. A network manger who decides to use artificial ignorance must first establish a baseline for normal log activity. Once parameters have been established, programming in software can alert the manager when there is activity outside the norm.Content Continues Below
Artificial ignorance addresses the first step of "Identify, Measure, Monitor and Report," a common risk management practice, by focusing attention on outlier data points and ignoring those data points that fall within an acceptable range. The concept of using "ignore this" filters is generally credited to Marcus Ranum, an information security (infosec) pioneer.
In addition to identifying anomalies in network behavior, the strategic practice of disregarding routine information can be useful in other situations that require the near- or real-time analysis of large data sets (big data). Practical applications include the identification of anomalies in email messages (spam filters), the identification of anomalies in online or offline financial transactions (identity theft and fraud prevention), monitoring the performance of smart devices (IoT analytics) and fine-tuning predictive models.
In machine learning (ML) and deep learning, the term artificial ignorance is increasingly being used to describe predictive models that fail. Causes of artificial ignorance in artificial intelligence (AI) programming include poor data quality, poor understanding of the business problem, poorly constructed validation sets, high bias (underfitting) or high variances (overfitting).