<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE article  PUBLIC "-//NLM//DTD Journal Publishing DTD v3.0 20080202//EN" "http://dtd.nlm.nih.gov/publishing/3.0/journalpublishing3.dtd"><article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" dtd-version="3.0" xml:lang="en" article-type="research article"><front><journal-meta><journal-id journal-id-type="publisher-id">JIS</journal-id><journal-title-group><journal-title>Journal of Information Security</journal-title></journal-title-group><issn pub-type="epub">2153-1234</issn><publisher><publisher-name>Scientific Research Publishing</publisher-name></publisher></journal-meta><article-meta><article-id pub-id-type="doi">10.4236/jis.2014.52003</article-id><article-id pub-id-type="publisher-id">JIS-43038</article-id><article-categories><subj-group subj-group-type="heading"><subject>Articles</subject></subj-group><subj-group subj-group-type="Discipline-v2"><subject>Computer Science&amp;Communications</subject></subj-group></article-categories><title-group><article-title>
 
 
  False Positive Responses Optimization for Intrusion Detection System
 
</article-title></title-group><contrib-group><contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>alal</surname><given-names>Baayer</given-names></name><xref ref-type="aff" rid="aff1"><sup>1</sup></xref></contrib><contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Boubker</surname><given-names>Regragui</given-names></name><xref ref-type="aff" rid="aff1"><sup>1</sup></xref></contrib><contrib contrib-type="author" xlink:type="simple"><name name-style="western"><surname>Aziz</surname><given-names>Baayer</given-names></name><xref ref-type="aff" rid="aff1"><sup>1</sup></xref></contrib></contrib-group><aff id="aff1"><addr-line>SIME Laboratory, ENSIAS, Mohammed V Suissi University, Rabat, Morocco</addr-line></aff><pub-date pub-type="epub"><day>20</day><month>02</month><year>2014</year></pub-date><volume>05</volume><issue>02</issue><fpage>19</fpage><lpage>36</lpage><history><date date-type="received"><day>11</day>	<month>November</month>	<year>2013</year></date><date date-type="rev-recd"><day>11</day>	<month>December</month>	<year>2013</year>	</date><date date-type="accepted"><day>18</day>	<month>December</month>	<year>2013</year></date></history><permissions><copyright-statement>&#169; Copyright  2014 by authors and Scientific Research Publishing Inc. </copyright-statement><copyright-year>2014</copyright-year><license><license-p>This work is licensed under the Creative Commons Attribution International License (CC BY). http://creativecommons.org/licenses/by/4.0/</license-p></license></permissions><abstract><p>
 
 
   In Intrusion Detection Systems (IDS), the operation costs represent one of the big challenges for researchers. They are apart from the IDS cost acquisition and they comprise the costs of maintenance, administration, response, running and errors reactions costs. In the present paper, we focus on the missed reactions which include False Positive (FP) and False Negative (FN) reactions. For that a new optimization cost model is proposed for IDS. This optimization proposes a minimal interval where the IDSs work optimally. In simulation, we found this interval as a trade-off between the damage costs and the FP.  
    
 
</p></abstract><kwd-group><kwd>Cost Model; Intrusion Detection System; False Positive; False Negative; Damage Costs and  Optimization</kwd></kwd-group></article-meta></front><body><sec id="s1"><title>1. Introduction</title><p>An Intrusion Detection System (IDS) [<xref ref-type="bibr" rid="scirp.43038-ref1">1</xref>] assures the process of detection and responding to malicious activity [<xref ref-type="bibr" rid="scirp.43038-ref2">2</xref>] that threats the computing and networking resources. It is generally based on four main components: the knowledge base [<xref ref-type="bibr" rid="scirp.43038-ref3">3</xref>] , the source of information (detector) [<xref ref-type="bibr" rid="scirp.43038-ref4">4</xref>] , the analysis module [<xref ref-type="bibr" rid="scirp.43038-ref5">5</xref>] and the response module, that makes responses based on appropriate analyzes.</p><p>This reply can be passive with standard alarm reports [<xref ref-type="bibr" rid="scirp.43038-ref6">6</xref>] or active based on additional module called IRS (Intrusion Response System) [<xref ref-type="bibr" rid="scirp.43038-ref7">7</xref>] . An IRS can be defined as a system, constantly supervise computer networks health based on IDS alerts, by launching efficiently suitable countermeasures against malevolent or illegal activities. These actions help to have deterioration prevention and keep the monitored system in its normal situation [<xref ref-type="bibr" rid="scirp.43038-ref8">8</xref>] .</p><p>An IRS can be static (predefined countermeasures attacks [<xref ref-type="bibr" rid="scirp.43038-ref9">9</xref>] ), dynamic (response based on the severity/confidence degree of the attack [<xref ref-type="bibr" rid="scirp.43038-ref10">10</xref>] ) and cost-sensitive (balance between the intrusion damage and the cost of response [<xref ref-type="bibr" rid="scirp.43038-ref11">11</xref>] ). The static IRS is easy to build and to maintain but it is predictable and vulnerable to intrusions, in particular, the denial of service (DoS) [<xref ref-type="bibr" rid="scirp.43038-ref12">12</xref>] . The Dynamic IRS is more sophisticated than static but do not introduce the cost as main element.</p><p>In other hand, the cost-sensitive IRS presents a good alternative to ensure responses that attempt to balance the intrusion damage and the cost of response [<xref ref-type="bibr" rid="scirp.43038-ref13">13</xref>] .</p><p>In our paper we focus on cost-sensitive IRS where the response cost is a financial value of a correct response launched against real attacks and the damage is the value of losses if the same response had not been launched against the same real attack [<xref ref-type="bibr" rid="scirp.43038-ref14">14</xref>] . Generally, actions are launched if the response cost is inferior to the damage value [<xref ref-type="bibr" rid="scirp.43038-ref15">15</xref>] . So, the success of a given response is strongly dependent on the good balance between the attack affectation damage and the system resources restoring costs.</p><p>When an IRS launches a wrong response against a real attack or in front of a normal activity, it generates a FP response. We talk about FN as the act not to detect any intrusion or launch any response when abnormal activity is observed. The FP and FN responses can harshly degrade the overall performances of IDS [<xref ref-type="bibr" rid="scirp.43038-ref16">16</xref>] and they are still subject of various research works.</p><p>Those likely false reactions in IDS cannot be totally weeded out in IDS. In that case, many models had been proposed to reduce their impacts [<xref ref-type="bibr" rid="scirp.43038-ref17">17</xref>] . These minimizations were done without having sufficient light on the trade-off between the FP and FN with cost aspect. This trade off is defined as a main performance indicator in IDS [<xref ref-type="bibr" rid="scirp.43038-ref18">18</xref>] based on the ROC curve [<xref ref-type="bibr" rid="scirp.43038-ref19">19</xref>] with cost notions. This ROC curve is suitably used to establish effectively the cost as a reliable metric.</p><p>In IDS, the fixed cost-sensitive model presents higher FP costs and extra FN costs [<xref ref-type="bibr" rid="scirp.43038-ref11">11</xref>] . So, to avoid this problem a new optimization cost model for IDS is required.</p><p>In the rest of the paper we focus our study on FP and we present a new optimization cost model which lets the IDS works with minimum costs by reducing the impact of FP responses. Our proposed model presents a minimal interval where the IDSs work optimally.</p><p>The rest of this paper is organized as follows. Section 2 presents the IRS module and performance indicators. Section 3 presents an overview on curve ROC in IDS. In Section 4 we review related work. Section 5 presents our improvement with simulations and results. The conclusion is given in Section 6.</p></sec><sec id="s2"><title>2. IRS Module &amp; Performance Indicators</title><p>Among the four principal modules of an IDS, we distinguish the response decision module that permit basing on analysis results to launch an alarm as passive response or communicate a decision to IRS to activate an active response. The IDS functional architecture is represented according to the  figure 1.</p><p>The Intrusion Response System (IRS) is a mechanism destined to ensure output as intrusion response following an IDS systems analysis. Various solutions of IRS had been applied by notification or by manual or automatic responses. To be able to initiate a response, it is necessary to determine what kind of attack we are faced.</p><p>According to our work described in [<xref ref-type="bibr" rid="scirp.43038-ref20">20</xref>] , we proposed an IRS organization according to their parameters assessment, which is shown following the figure 2.</p><p>We focus on cost sensitive IRS. For that many indicators are defined for performances.</p><p>We define various indicators and metrics used in IDS performance evaluation. Those metrics evaluate the ability of IDSs to detect effectively malicious activities. So in order to assimilate the performance characteristics of various IDSs, several indicators or measures are needed to quantitatively assess the competence of the intrusion detection. To date, many indicators have been proposed to evaluate IDS. But it is imperative to learn and study the behavior of the IDS before having the ability to evaluate its performance. We distinguish:</p><p>• True negative (TN): represents the number of normal activities seen by the IDS as normal.</p><p>• True positive (TP): represents the number of intrusions seen by the IDS as true intrusions.</p><p>• False negative (FN): represents the number of intrusions seen by the IDS as normal.</p><p>• False positive (FP): represents the number of normal activities seen by the IDS as intrusions.</p><p>• Accuracy: shows the percentage of real intrusions from the real number of intrusions reported by the IDS:</p><disp-formula id="scirp.43038-formula12877"><label>(1)</label><graphic position="anchor" xlink:href="1-7800184x\614c8f63-1a22-424e-bfc7-b62baebc8752.jpg"  xlink:type="simple"/></disp-formula><p>• Rate or probability of detection: shows the percentage of reported intrusions from the number of intrusions:</p></sec></body><back><ref-list><title>References</title><ref id="scirp.43038-ref1"><label>1</label><mixed-citation publication-type="other" xlink:type="simple">Denning, D. (1987) An Intrusion-Detection Model. IEEE Transactions on Software Engineering, SE-13, 222-232. http://dx.doi.org/10.1109/TSE.1987.232894</mixed-citation></ref><ref id="scirp.43038-ref2"><label>2</label><mixed-citation publication-type="other" xlink:type="simple">Endorf, C., Schultz, E. and Mellander, J. (2004) Intrusion Detection &amp; Prevention. McGraw-Hill/Osborne.</mixed-citation></ref><ref id="scirp.43038-ref3"><label>3</label><mixed-citation publication-type="other" xlink:type="simple">Zanero, S. and Savaresi, S.M. (2004) Unsupervised Learning Techniques for an Intrusion Detection System. Proceedings of the 2004 ACM Symposium on Applied Computing, Nicosia, 14-17 March 2004.</mixed-citation></ref><ref id="scirp.43038-ref4"><label>4</label><mixed-citation publication-type="other" xlink:type="simple">Ertoz, L., Eilertson, E., Lazarevic, A., Tan, P., Srivastava, J., Kumar, V. and Dokas, P. (2004) The MINDS—Minnesota Intrusion Detection System. Next Generation Data Mining, MIT Press.</mixed-citation></ref><ref id="scirp.43038-ref5"><label>5</label><mixed-citation publication-type="other" xlink:type="simple">Gul, I. and Hussain, M. (2011) Distributed Cloud Intrusion Detection Model. International Journal of Advanced Science and Technology, 34, 71.</mixed-citation></ref><ref id="scirp.43038-ref6"><label>6</label><mixed-citation publication-type="other" xlink:type="simple">Elshoush, H.T. and Osman, I.M. (2011) Alert Correlation in Collaborative Intelligent Intrusion Detection Systems—A Survey. Journal of Applied Soft Computing, 11, 4349-4365. http://dx.doi.org/10.1016/j.asoc.2010.12.004</mixed-citation></ref><ref id="scirp.43038-ref7"><label>7</label><mixed-citation publication-type="other" xlink:type="simple">Anuar, N.B., Papadaki, M., Furnell, S. and Clarke, N. (2010) An Investigation and Survey of Response Options for Intrusion Response Systems. Information Security for South Africa, Sandton, 2-4 August 2010, 1-8.</mixed-citation></ref><ref id="scirp.43038-ref8"><label>8</label><mixed-citation publication-type="journal" xlink:type="simple"><name name-style="western"><surname>Shameli-Sendi</surname><given-names> A.</given-names></name>,<name name-style="western"><surname> Ezzati-Jivan</surname><given-names> N.</given-names></name>,<name name-style="western"><surname> Jabbarifar</surname><given-names> M. and Dagenais</given-names></name>,<name name-style="western"><surname> M. </surname><given-names>  </given-names></name>,<etal>et al</etal>. (<year>2012</year>)<article-title>Intrusion Response Systems: Survey and Taxonomy</article-title><source> SIGMOD Record</source><volume> 12</volume>,<fpage> 1</fpage>-<lpage>14</lpage>.<pub-id pub-id-type="doi"></pub-id></mixed-citation></ref><ref id="scirp.43038-ref9"><label>9</label><mixed-citation publication-type="other" xlink:type="simple">Mu, C., Shuai, B. and Liu, H. (2010) Analysis of Response Factors in Intrusion Response Decision Making. 3rd International Joint Conference on Computational Science and Optimization, Huangshan, 28-31 May 2010, 395-399.</mixed-citation></ref><ref id="scirp.43038-ref10"><label>10</label><mixed-citation publication-type="other" xlink:type="simple">Zonouz, S.A., Khurana, H., Sanders, W.H. and Yardley, T.M. (2009) RRE: A Game-Theoretic Intrusion Response and Recovery Engine. Proceedings of the IEEE/IFIP International Conference on Dependable Systems and Networks, Lisbon, 29 June-2 July 2009, 439-448.</mixed-citation></ref><ref id="scirp.43038-ref11"><label>11</label><mixed-citation publication-type="other" xlink:type="simple">Zhou, M. and Yao, G. (2011) Improved Cost-Sensitive Model of Intrusion Response System Based on Clustering. International Conference in Electrics, Communication and Automatic Control Proceedings, 931-937.</mixed-citation></ref><ref id="scirp.43038-ref12"><label>12</label><mixed-citation publication-type="other" xlink:type="simple">Svecs, I., Sarkar, T., Basu, S. and Wong, J. (2010) XIDR: A Dynamic Framework Utilizing Cross-Layer Intrusion Detection for Effective Response Deployment. IEEE 34th Annual Computer Software and Applications Conference Workshops, Seoul, 19-23 July 2010, 287-292.</mixed-citation></ref><ref id="scirp.43038-ref13"><label>13</label><mixed-citation publication-type="other" xlink:type="simple">Stakhanova, N., Basu, S. and Wong, J. (2007) A Cost-Sensitive Model for Preemptive Intrusion Response Systems. Proceedings of the 21st International Conference on Advanced Networking and Applications, Niagara Falls, 21-23 May, 428-435.</mixed-citation></ref><ref id="scirp.43038-ref14"><label>14</label><mixed-citation publication-type="other" xlink:type="simple">Strasburg, C., Stakhanova, N., Basu, S. and Wong, J.S. (2009) A Framework for Cost Sensitive Assessment of Intrusion Response Selection. Proceedings of IEEE Computer Software and Applications Conference, Seattle, 20-24 July 2009, 355-360.</mixed-citation></ref><ref id="scirp.43038-ref15"><label>15</label><mixed-citation publication-type="other" xlink:type="simple">Stakhanova, N., Basu, S. and Wong, J. (2007) A Cost-Sensitive Model for Preemptive Intrusion Response Systems. Proceedings of the IEEE AINA, Niagara Falls, 21-23 May 2007, 428-435.</mixed-citation></ref><ref id="scirp.43038-ref16"><label>16</label><mixed-citation publication-type="other" xlink:type="simple">Timm, K. (2009) Strategies to Reduce False Positives and False Negatives in NIDS. Security Focus Article. http://www.securityfocus.com/infocus/1463</mixed-citation></ref><ref id="scirp.43038-ref17"><label>17</label><mixed-citation publication-type="journal" xlink:type="simple"><name name-style="western"><surname>Victor</surname><given-names> G.V.</given-names></name>,<name name-style="western"><surname> Sreenivasa</surname><given-names> R.M. and Venkaiah</given-names></name>,<name name-style="western"><surname> V.CH. </surname><given-names>  </given-names></name>,<etal>et al</etal>. (<year>2010</year>)<article-title>Intrusion Detection Systems—Analysis and Containment of False Positives Alert</article-title><source> International Journal of Computer Applications</source><volume> 5</volume>,<fpage> 27</fpage>-<lpage>33</lpage>.<pub-id pub-id-type="doi"></pub-id></mixed-citation></ref><ref id="scirp.43038-ref18"><label>18</label><mixed-citation publication-type="other" xlink:type="simple">Lippmann, R., Fried, D.J., Graf, I., Haines, J.W., Kendall, K.R., McClung, D., Weber, D., Webster, S.H., Wyograd, D., Cunningham, R.K. and Zissman, M.A. (2000) Evaluating Intrusion Detection Systems: The 1998 DARPA Off-Line Intrusion Detection Evaluation. Proceedings of DARPA Information Survivability Conference and Exposition, Hilton Head, 25-27 January 2000, 12-26.</mixed-citation></ref><ref id="scirp.43038-ref19"><label>19</label><mixed-citation publication-type="other" xlink:type="simple">Stolfo, S., Fan, W., Lee, W., Prodromidis, A. and Chan, P. (2000) Costbased Modeling for Fraud and Intrusion Detection: Results from the JAM Project. Proceedings of DARPA Information Survivability Conference and Exposition, Los Alamitos, 2, 130-144.</mixed-citation></ref><ref id="scirp.43038-ref20"><label>20</label><mixed-citation publication-type="other" xlink:type="simple">Baayer, J. and Regragui, B. (2009) WOTIC’09—“Architecture Fonctionnelle d’un IPS, Etat de l’Art et Classification de Ses Systèmes de Réponse d’Intrusion (IRS)”. Université Ibn Zohr, Agadir.</mixed-citation></ref><ref id="scirp.43038-ref21"><label>21</label><mixed-citation publication-type="other" xlink:type="simple">Swets, J.A. (1996) Signal Detection Theory and ROC Analysis in Psychology and Diagnostics: Collected Papers. Lawrence Erlbaum Associates, Mahwah.</mixed-citation></ref><ref id="scirp.43038-ref22"><label>22</label><mixed-citation publication-type="other" xlink:type="simple">Foo, B., Wu, Y.-S., Mao, Y.-C., Bagchi, S. and Spafford, E.H. (2005) ADEPTS: Adaptive Intrusion Response Using Attack Graphs in an E-Commerce Environment. Proceedings of DSN, 28 June-1 July, 508-517.</mixed-citation></ref><ref id="scirp.43038-ref23"><label>23</label><mixed-citation publication-type="other" xlink:type="simple">Toth, T. and Kregel, C. (2002) Evaluating the Impact of Automated Intrusion Response Mechanisms. Proceeding of the 18th Annual Computer Security Applications Conference, Los Alamitos, 301-310.</mixed-citation></ref><ref id="scirp.43038-ref24"><label>24</label><mixed-citation publication-type="journal" xlink:type="simple"><name name-style="western"><surname>Balepin</surname><given-names> I.</given-names></name>,<name name-style="western"><surname> Maltsev</surname><given-names> S.</given-names></name>,<name name-style="western"><surname> Rowe</surname><given-names> J. and Levitt</given-names></name>,<name name-style="western"><surname> K. </surname><given-names>  </given-names></name>,<etal>et al</etal>. (<year>2003</year>)<article-title>Using Specification-Based Intrusion Detection for Automated Response</article-title><source> Proceedings of RAID</source><volume> 2820</volume>,<fpage> 136</fpage>-<lpage>154</lpage>.<pub-id pub-id-type="doi"></pub-id></mixed-citation></ref><ref id="scirp.43038-ref25"><label>25</label><mixed-citation publication-type="other" xlink:type="simple">Jahnke, M., Thul, C. and Martini, P. (2007) Graph Based Metrics for Intrusion Response Measures in Computer Networks. Proceedings of the IEEE LCN, Dublin, 15-18 October 2007, 1035-1042.</mixed-citation></ref><ref id="scirp.43038-ref26"><label>26</label><mixed-citation publication-type="other" xlink:type="simple">Yu, S. and Rubo, Z. (2008) Automatic Intrusion Response System Based on Aggregation and Cost. International Conference on Information and Automation, Changsha, 20-23 June 2008, 1783-1786.</mixed-citation></ref><ref id="scirp.43038-ref27"><label>27</label><mixed-citation publication-type="other" xlink:type="simple">Papadaki, M. and Furnell, S.M. (2006) Achieving Automated Intrusion Response: A Prototype Implementation. Information Management and Computer Security, 14, 235-251. http://dx.doi.org/10.1108/09685220610670396</mixed-citation></ref><ref id="scirp.43038-ref28"><label>28</label><mixed-citation publication-type="other" xlink:type="simple">Haslum, K., Abraham, A. and Knapskog, S. (2007) DIPS: A Framework for Distributed Intrusion Prediction and Prevention Using Hidden Markov Models and Online Fuzzy Risk Assessment. 3rd International Symposium on Information Assurance and Security, Manchester, 29-31 August 2007, 183-188. http://dx.doi.org/10.1109/ISIAS.2007.4299772</mixed-citation></ref><ref id="scirp.43038-ref29"><label>29</label><mixed-citation publication-type="other" xlink:type="simple">Mu, C.P. and Li, Y. (2010) An Intrusion Response Decision Making Model Based on Hierarchical Task Network Planning. Expert Systems with Applications, 37, 2465-2472. http://dx.doi.org/10.1016/j.eswa.2009.07.079</mixed-citation></ref><ref id="scirp.43038-ref30"><label>30</label><mixed-citation publication-type="other" xlink:type="simple">Kanoun, W., Cuppens-Boulahia, N., Cuppens, F. and Dubus, S. (2010) Risk-Aware Framework for Activating and Deactivating Policy-Based Response. 4th International Conference on Network and System Security, Melbourne, 1-3 September 2010, 207-215.</mixed-citation></ref><ref id="scirp.43038-ref31"><label>31</label><mixed-citation publication-type="journal" xlink:type="simple"><name name-style="western"><surname>Kheir</surname><given-names> N.</given-names></name>,<name name-style="western"><surname> Cuppens-Boulahia</surname><given-names> N.</given-names></name>,<name name-style="western"><surname> Cuppens</surname><given-names> F. and Debar</given-names></name>,<name name-style="western"><surname> H. </surname><given-names>  </given-names></name>,<etal>et al</etal>. (<year>2010</year>)<article-title>A Service Dependency Model for Cost Sensitive Intrusion Response</article-title><source> Proceedings of the 15th European Conference on Research in Computer Security</source><volume> 6345</volume>,<fpage> 626</fpage>-<lpage>642</lpage>.<pub-id pub-id-type="doi"></pub-id></mixed-citation></ref><ref id="scirp.43038-ref32"><label>32</label><mixed-citation publication-type="other" xlink:type="simple">Denning, D. (1999) Information Warfare and Security. Addison-Wesley.</mixed-citation></ref><ref id="scirp.43038-ref33"><label>33</label><mixed-citation publication-type="other" xlink:type="simple">Northcutt, S. (1999) Intrusion Detection: An Analyst’s Handbook. New Riders Publishing.</mixed-citation></ref><ref id="scirp.43038-ref34"><label>34</label><mixed-citation publication-type="journal" xlink:type="simple"><name name-style="western"><surname>Lee</surname><given-names> W.</given-names></name>,<name name-style="western"><surname> Fan</surname><given-names> W.</given-names></name>,<name name-style="western"><surname> Millerand</surname><given-names> M.</given-names></name>,<name name-style="western"><surname> Stolfo</surname><given-names> S. and Zadok</given-names></name>,<name name-style="western"><surname> E. </surname><given-names>  </given-names></name>,<etal>et al</etal>. (<year>2002</year>)<article-title>Toward Cost-Sensitive Modeling for Intrusion Detection and Response</article-title><source> Journal of Computer Security</source><volume> 10</volume>,<fpage> 5</fpage>-<lpage>22</lpage>.<pub-id pub-id-type="doi"></pub-id></mixed-citation></ref><ref id="scirp.43038-ref35"><label>35</label><mixed-citation publication-type="other" xlink:type="simple">Tanachaiwiwat, S., Hwang, K. and Chen, Y. (2002) Adaptive Intrusion Response to Minimize Risk over Multiple Network Attacks. ACM Trans on Information and System Security.</mixed-citation></ref><ref id="scirp.43038-ref36"><label>36</label><mixed-citation publication-type="other" xlink:type="simple">Durst, R., Champion, T., Witten, B., Miller, E. and Spag-nuolo, L. (1999) Testing and Evaluating Computer Intrusion Detection Systems. ACM, 42, 53-61. http://dx.doi.org/10.1145/306549.306571</mixed-citation></ref><ref id="scirp.43038-ref37"><label>37</label><mixed-citation publication-type="other" xlink:type="simple">Saydjari, O.S. (2000) Designing a Metric for Effect. Presented at DARPA: IDS Evaluation Re-Think Meeting, Lake Geneva, 23-24 May.</mixed-citation></ref><ref id="scirp.43038-ref38"><label>38</label><mixed-citation publication-type="other" xlink:type="simple">Stolfo, S., Fan, W., Lee, W., Prodromidis, A. and Chan, P. (2000) Costbased Modeling for Fraud and Intrusion Detection: Results from the JAM Project. Proceedings of DARPA Information Survivability Conference and Exposition, Los Alamitos, 2, 130-144.</mixed-citation></ref><ref id="scirp.43038-ref39"><label>39</label><mixed-citation publication-type="other" xlink:type="simple">McHugh, J., Christie, A. and Allen, J. (2000) Defending Yourself: The Role of Intrusion Detection Systems. IEEE Software, 17, 42-51. http://dx.doi.org/10.1109/52.877859</mixed-citation></ref><ref id="scirp.43038-ref40"><label>40</label><mixed-citation publication-type="other" xlink:type="simple">Graf, I., Lippmann, R., Cunningham, R., Fried, D., Kendall, K., Webster, S. and Zissman, M. (1998) Results of DARPA 1998 Off-Line Intrusion Detection Evaluation. Presented at DARPA PI Meeting, Cambridge, 15 December.</mixed-citation></ref><ref id="scirp.43038-ref41"><label>41</label><mixed-citation publication-type="other" xlink:type="simple">(2012) Verizon Business Data Breach Investigations Report. http://www.verizonenterprise.com/DBIR/2013/</mixed-citation></ref><ref id="scirp.43038-ref42"><label>42</label><mixed-citation publication-type="other" xlink:type="simple">Widup, S. (2010) The Leaking Vault—Five Years of Data Breaches. Digital Forensics Association.</mixed-citation></ref><ref id="scirp.43038-ref43"><label>43</label><mixed-citation publication-type="other" xlink:type="simple">An Osterman Research White Paper (2011) Why You Need to Eliminate False Positives in Your Email System. http://www.ostermanresearch.com.</mixed-citation></ref></ref-list></back></article>