The model is “run” in an iterated player-versus-nature game with discrete time. In each round, the defender makes a security investment decision to define his configuration of defenses and extracts a net return from his asset before the attacker penetrates the system and, if successful, loots a fraction z of the asset. Gross returns are consumed or distributed so that the asset value does not accumulate over time. The defender seeks to maximize his expected return per round, where expectations are taken over the realizations of random variables and averages calculated over time.

Let elements of the binary column vector indicate whether a defense against the i-th threat is

implemented () or not, and let be the number of defenses in place.

The cost of defense in round t can be calculated from an upper triangular cost matrix to reflect possible interdependent defenses,

Diagonal elements hold the cost to implement a defense against the i-th threat, and off-diagonal elements indicate the extra costs if the i-th defense is implemented together with the j-th defense. If all off- diagonal elements are zero, then the defenses are independent:²

A nice property of the cost matrix is that for positive off-diagonal elements, decreasing marginal utility of defenses has become endogenous to our model. This compares favorably to, say, the Gordon-Loeb framework, in which this property appears as an assumption (A3 of [6] , p. 443).

While the defender may possess some intuition about the relative difficulty of carrying out the n threats, such knowledge may very well be blurred. To model this uncertainty, we order the threats by increasing expected cost of attack, where the expectation is taken from the point of view of the defender. This constitutes our notion of an attack profile. This rank order is also convenient because it allows us to ignore the threats deemed too unlikely when deciding what to defend. This way, one can think of the n threats as those anti- cipated by the defender.

We define a simple functional form for the expected attack costs of the i-th threat as follows,

The unknown true cost, however, is modeled as a Gaussian random variable with mean and standard deviation (truncated to values),

The realizations are drawn only once and kept constant over time. We can vary the level of uncertainty by adjusting parameter. Modeling uncertainty in this way is crucial to the model, since it captures the diffi- culty defenders face in anticipating which of the undefended threats turns out to be the weakest link exploited by the attacker. Figure 1 illustrates the role of uncertainty when ordering threats. The left graph plots a perfect matching between expected and realized costs of attack under certainty; under uncertainty, by contrast, the right graph shows that threat 4, not threat 3 as expected, is the weakest link if defenses 1 and 2 are in place.

The defender’s knowledge about the attack profile will naturally increase over time once attacks occur, and so her uncertainty decreases as the observed attacks reveal which threat is in fact the weakest link with respect to a given configuration.

So far we have assumed that the defender can upgrade his configuration at any time. This may be necessary in order to adjust to new information on the threat level or to changing risk appetite. For example, a start-up company is exposed to so many risks that is might tolerate a moderate level of information security risk. As the venture grows and develops a brand name, its risk aversion will increase to reflect the higher damage caused by potential reputation losses. Conversely, market competition (alongside herd behavior and short-sighted incen- tives to management) may drive firms to decrease their risk aversion for the sake of higher expected profits.

As is argued in [6] , security disinvestment is often possible to a certain degree, since personnel can be fired or equipment sold (though sometimes at high transaction costs). However, for many businesses, updating may be very expensive and the costs are spent irrevocably. For instance, the vast majority of the cost to incorporate new security features into bank notes or payment cards are borne when the changes are first made. Producing and distributing tailored tokens or devices to a large and dispersed community is expensive, and unlikely to be repeated often. As unrecoverable costs considerably affect the strategy of security investment in an iterated setting, we include them in our model as follows:

Hence, parameter controls the amount of unrecoverable costs, on an ex post basis sometimes referred to as sunk costs.

Our attacker model is very simple: the attacker identifies and exploits the weakest link, i.e., the threat least costly to the attacker. We do not require the attacker to make a trade-off between cost and potential gains by assuming the same utility is received for exploiting all threats. If the attacker succeeds, regardless of how, he profits, which is added to the defender’s cost. Unlike the defender, the attacker is certain of the cost for each realization of. The attacker does not operate indiscriminately; rather, he only attacks when it is pro- fitable to do so; that is, if the term is not negative.

These attacker assumptions are quite strong and probably unrealistic in some cases. For targeted attackers in particular, searching for the weakest link could be very expensive. So a more appropriate conception is the presence of many opportunistic attackers [7] who operate independently. Like in swarm intelligence, the chance that one of them finds the hole is quite high.

Moreover, the attacker’s gain from successful attacks may only represent a small fraction of the defender’s losses. But this is not crucial here since we do not consider attackers’ profits; we simply need an asset-dependent quantity to relate to the cost. A level shift of could easily account for the negative balance (destruction of wealth) after successful attacks. This assumption is also common in the literature (e.g., [8] ), and justified there by attackers who mainly care about inflicting harm on the defender. In summary, we consider it prudent to err in the direction of a more powerful attacker than an unrealistically weak one.

To make the analysis more tractable, we restrict the space of possible cost matrices by fixing all diagonal elements to a unit cost of value one and all off-diagonal elements, to the same constant. Hence, we can now derive from Equation (1) a simple expression for the cost per round to maintain k defenses,

Observe that if and if. Of course, if better knowledge on the cost of defenses is available in practical applications of the model, this assumption can be relaxed and populated with real data.

We further assume a risk neutral defender, that is, one who maximizes inter-temporal utility as a linear function of (expected) revenues and costs. We defer consideration of other attitudes towards risk or different time preferences to future work in order to keep the core model lean.

Our model can be applied to many scenarios. For now, we demonstrate its parameters in action by considering an imaginary online music store to demonstrate its parameters in action (we describe other scenarios in Section 5). The music store’s asset is the library of audio files and the associated property rights. Let the total value of the asset be one million dollars. To avoid large numbers, we count in thousands of dollars, so asset value a = 1 000. Every year, indexed by t, the net return from online sales amounts to of the asset value.

Surrounded by malicious competitors, professional cybercriminals and “curious customers”, the store is ex- posed to various threats (say,). In case of a successful attack, the store loses of the asset value, or $25,000, due to forgone sales and incurred damage. It is known (from industry sources) that, on average over all similar businesses, the cost of a successful attack is at least 15,000 dollars (). This could be the price for an attack kit against standard software on the underground market for cybercrime supplies [9] . Attack costs also rise with novelty and difficulty of implementation, on average by 1000 dollars for each level of sophistication. Hence, the gradient of the attack cost is. However, these are only average values and it is unknown how much the actual costs are to attack the particular business. This uncertainty can be expressed by the deviation of the attack costs around the average. To put this parameter in perspective, means that all attacks can be predicted perfectly, means 96% of the attacks can be predicted correctly,³ and for, predicting attacks is only 65% successful. This is still somewhat better than random guessing.

The store owner is not entirely helpless: he may spend on security measures to defend against each of the threats. Each defense costs 1000 dollars per round (this is a constant that defines the unit, so no symbol is required). On top of that, defense interdependence with parameter accounts for additional cost for every pair of installed defenses. As a consequence, every investment in security must be carefully considered, since countermeasures added later cost more due to increased complexity and interoperability challenges. Finally, parameter controls (optional) unrecoverable costs. It describes the cost to update the defense con- figuration as a fraction of the asset value a. In the example of a music store, unrecoverable costs are negligible (i.e.,) since security upgrades do usually not render existing investment useless. However, if the music store served a proprietary audio format targeted to specific playback devices, an exchange of all customers’ physical playback devices to upgrade the defense configuration could be modeled as unrecoverable cost ().

Table 1 summarizes the parameters of our model and their values used in the subsequent examples. The following section will show how defenders can optimize their security spending in this model. We are primarily interested in exploring how the optimal strategy changes based on the level of uncertainty; we also elaborate on the role of unrecoverable costs.

We consider a defender with defense vector where the lowest cost threats are defended, leaving the remaining threats unprotected: and. Let be the return function for this defender,

where q is an indicator variable that takes value one if the attacker is successful and zero otherwise. In the case of certainty, q is fixed for all rounds, so that two outcomes are possible:

The cost term disappears in Equation (8) because if we accept attacks anyway, then there is no need to spend money to make it more difficult for the attacker (i.e., k = 0). Otherwise, the minimal number of defenses to defeat all attacks can be calculated by rearranging Equation (2) and comparing it to the attacker’s gain,

Typically, the best strategy for the defender is to follow whatever is higher, Equation (8) or (9), with. When Equation (9) is lower than Equation (8), the defender finds it profitable to operate without investing in any security measures. We call this situation indefensible, which does not necessarily imply that the business is unsustainable. Rather like in self-insurance, the defender extracts positive gross return after deducting the losses due to successful attacks. Only when both equations are negative, the cost of attacks and the cost of defending them are both so high that the defender’s business becomes unviable.

In the case of uncertainty (), q becomes a random variable and a risk neutral defender maximizes, where expectations are taken over the realizations of q. For a static solution, it is still rational to use configurations in which lower-order threats are defended first. So we can calculate as a function of k as the probability that the cost of at least one of the unprotected threats falls below the threshold,

where is the cumulative distribution function of the Gaussian normal distribution. The product term is the probability that none of the undefended threats materializes, so one minus this term is the probability of at least one successful attack. Inserting into Equation (7), we obtain

Figure 2 depicts expected returns of the static solution as a function of k for parameters as specified in Table 1. Each curve corresponds to different levels of uncertainty; the utility-maximizing selection of for each uncertainty level is circled. In the case of certainty (), the optimal security investment amounts to defenses (Equation (10)). Increasing uncertainty affects the optimal strategy in a non-linear manner: while some noise in the predicted attack costs lets the defender take more proactive counter-measures (), the cost of this over-investment start to outweigh the protected revenues when uncertainty is too high (). Eventually, it becomes rational for more uninformed defenders to refrain from any defense and instead accept moderate losses in each round. In other words, knowing your enemy is a prerequisite for taking tailored countermeasures, and tailored defense is the only option if comprehensive defense (i.e.,) is prohibitively expensive.

When (uncertainty), the static solution is often sub-optimal because observed attacks reveal additional

information about the true cost of attack. This information could be used to reconfigure the defenses adaptively. In the dynamic case, uncertainty is repeatedly eliminated for targets revealed to be the weakest link. Con- sequently, this reduction in uncertainty for later rounds can lead to a better outcome for the defender than in the static case. In general, the inter-temporal outcome is the sum of all returns over a period of time,

and can be maximized by adjusting adaptively. At each point in time t, the defender learns about the materialized threat (if any) of period. In Equation (13), , and, are functions of the sequence of configurations and the random realization of. Due to this stochastic component and its subsequent path dependencies through adaptation, the analysis must be based on expectations over all possible realizations. We will first discuss the simpler case without fully recoverable costs (i.e.,) before moving on to the general case.

Our first argument focuses on the security of computers comprising the Internet follow an iterated weakest link model. Fundamental to the Internet’s design is the notion that any connected computer can communicate with every other computer. Attackers have relentlessly exploited this structure, constructing networks of compromised machines (so-called botnets) to pester legitimate users by emitting spam, distributing malware and hosting phishing websites. Internet service providers (ISPs) are usually well-placed to detect infection, because evidence of a user’s infection flows over an ISP’s communication systems. Moreover, large companies that act as Internet service providers have technical staff who can detect and clean up infected machines, while domestic users and small businesses are mostly helpless to even recognize when they are compromised; let alone to take appropriate remedial action. However, for every responsible ISP that keeps a clean network, there are countless others who find it more cost-effective to ignore the problem and let infections flourish.

There is substantial evidence that attackers concentrate their efforts at the most irresponsible ISPs, moving on to others once the ISP cleans up its act or is shut down. For instance, until 2007, a lot of the world’s malware was hosted by the Russian Business Network (RBN), which refused to comply with requests from international law enforcement [14] . After Russian Business Network suddenly went offline in late 2007, malware distribution shifted to other ISPs. In November 2008, a journalist from the Washington Post persuaded upstream bandwidth providers to shut off their connection to San Francisco-based McColo, which led to a temporary fall of almost 70% in the volume of spam worldwide [15] . Apparently, many botnet herders had been using McColo to host their control machines. Spam volume has since recovered as the spammers found new safe havens. In 2008 EstDomains, which served as the primary domain name registrar for malicious websites hosted by Russian Business Network [16] , became the first domain registrar to have its accreditation terminated by the Internet Corporation for Assigned Names and Numbers [17] . Unfortunately, other irresponsible registrars remain.

Attackers can often move on to exploit the next weakest link in the Internet far faster than defenders can knock them out. Moore and Clayton [18] showed how one leading group of online fraudsters, the rock-phish gang, operated. It registered many malicious web domain names to carry out attacks. Periodically, the gang picked a firm for registering Internet domain names that it had never used before and registered hundreds of web domains, using false names and stolen credit cards. These domains did not resemble normal bank names, so a registrar may not identify foul play, and the first domains to be used for false purposes were not removed quickly.

Figure 6 presents scatter plots of phishing site lifetime based on the date reported from data of [18] . Both .hk (Hong Kong) domains (left) and .cn (China) domains (right) lasted much longer in their first month of use than in later months. The gang targeted Hong Kong domains first in March 2007, followed by Chinese domains in May 2007 after the Hong Kong authorities wised up.

Such iteration over previously untargeted infrastructure operators is prevalent among many forms of cybercrime. Other researchers have found that websites hosting most malware move from one registrar to the next [19] . Liu et al. studied patterns in the registration and take-down of spam-advertised domains [20] , finding that miscreants register many domains at previously untargeted registrars, repeatedly moving on to new ones following crack-downs. McCoy et al. examined the use of payment processors by unlicensed online pharmacies [21] . They showed that the operators of unlicensed pharmacies initially relied on a small number of payment processors. Once banks were made aware of the criminal behavior, many severed ties with the pharmacies, who quickly made arrangements with new processors.

Financial transactions have long been subject to fraud, and banks and payment processors have recognized that fraud has to be managed rather than eliminated. In 2000, the UK banks decided to adopt the EMV card payment system designed by Europay, Mastercard and Visa, popularly known as “Chip and PIN” EMV. The EMV roll-out provides a compelling example of taking security decisions to tackle the weakest link, only to find that the attackers react by exploiting new holes previously ignored. A May 2007 internal report prepared by APACS (since rebranded as the UK Cards Association) compared the expected benefits from EMV adoption to fraud incidence over time. While the report was intended to be kept confidential, it was accidentally leaked and is now hosted on the Cryptome website [22] . We use the data gleaned from this report, as well as more recent public statistics [23] [24] , to demonstrate how the banks’ fight against card fraud has essentially been a sequence of patches to plug weak links.

Figure 7 plots the annual percentage of transaction turnover found to be fraudulent from 1972 to 2014.

Notably, there are significant swings in the rate. According to the report [22] , “the industry has responded to crises in card fraud by making significant investment in new preventative measures,” including “terminalisation in the early 80s and, later, a major increase in levels of authorisation during the mid 90s.” Such behavior is consistent with our model of iterating through successive weakest links.

Around 2000, a fear arose that another uptick in the fraud rate was imminent. The dashed line with triangle markers in Figure 8 plots the forecast losses due to fraud based on maintaining the status quo as of 2000. To stem this expected rise in fraud, UK banks decided to adopt the EMV standard (Chip and PIN) over the course of the next few years. The estimated cost of implementing EMV across the UK was just over £1 billion, shared between the banks and retailers. We can interpret this investment to roll out EMV as unrecoverable costs. The dashed line with diamond markers in Figure 8 plots the forecast losses following EMV adoption. The uptick to 2003 is due to the phased roll-out of the technology.

In fact, the actual losses look rather different (solid line in Figure 8). The anticipated explosion of fraud in the early 2000s did not materialize; strikingly, the full adoption of EMV (completed in 2005) did not significantly alter the fraud totals. Instead, a slow but steady increase was observed, both before and after EMV adoption.

Digging a little deeper, we can see that the switch to EMV did significantly alter attacker strategy, even if it did not significantly shift the aggregate fraud rates. Instead, as some types of fraud diminished, other techniques quickly filled the gap. For instance, EMV has caused a dramatic drop in face-to-face retail fraud, since forging a signature is easier than guessing the right PIN. In 2004, face-to-face fraud totaled £219 million; in 2006, after EMV became mandatory, it fell to £72 million (see Figure 9). However, card-not-present (CNP) fraud (where only the card number is used, as in online transactions) has sharply risen since the introduction of EMV, from £150 million in 2004 to £328 million in 2008, before falling modestly and reaching a new high of £332 million in 2014. This is not surprising, since PINs are not verified for CNP transactions.

Another example of attackers moving to weaker links can be found by looking at where ATM fraud takes place. With the adoption of EMV, ATMs in the UK stopped verifying PINs using magnetic stripes, requiring the PIN be verified using the more secure chip. However, many ATMs outside the UK continued to allow the less secure verification via magnetic stripe. As UK ATMs switched over during 2006, fraudsters immediately adapted to cashing out via less secure international ATMs. Figure 10 shows how this transformation happened. In December 2005, when most UK ATMs still allowed magnetic stripe PIN verification, £6 million out of the total £6.5 million stolen from ATMs took place within the UK. By the end of the year, after the switch to more secure verification, the total monthly fraud increased to over £9 million, with nearly 75% of the losses occurring overseas. The shift to overseas fraud that is easier to carry out has continued since the EMV rollout. In Figure 11, we see that the share of the total fraud occurring overseas rose from £93M (18% of the total) in 2004 to a peak of £230M (37% of the total) in 2008. In fact, the share of overseas-originated fraud on UK cards in 2008 exceeded the total UK-originated fraud in 2004, just prior to the EMV deployment. This is compelling evidence of how quickly attackers can shift to find the next-weakest link once the biggest hole has been plugged.

The EMV standard actually provides several options for implementation that vary in the cost and security achieved. While the UK EMV implementation was certainly not inexpensive, its designers did opt for many of the lower-cost implementations offered by the standard. For instance, PIN verification is normally done offline, be- tween the card’s chip and the PIN entry device (PED). This enables a relay attack whereby a small, legitimate transaction authorized by a compromised terminal can be used to approve a simultaneous, much larger fraudulent

transaction elsewhere [25] . Furthermore, the communications within the PED between the card’s chip and the PED’s processor are not encrypted. Combined with some shoddy tamper-proofed devices, this enables PINs to be read using a paper clip tapped inside the PED [26] . The same team later discovered an attack on the protocol that could enable a criminal to use a genuine card without knowing the PIN [24] [27] . While entirely practical, there is no evidence that these attacks are being used by attackers at present. Instead, they seem to be focusing on the more obvious weak links of card-not-present transactions and magnetic stripe verification at foreign

ATMs. If (and when) these holes are plugged⁵, one might expect these attacks to be deployed, which would trigger defensive countermeasures by the banks.

Figure 12 plots the expected and realized costs of different attacks for the case studies just presented. Figure 12(a) plots a sequence of top-level domains that potentially targeted by the rock-phish gang. First, many Chinese domains were registered by the gang, followed by domains from Hong Kong (as shown in Figure 6). The next top-level domain targeted―Austria―came as a surprise. Austria has an extensive Internet security community, and its law enforcement has long cooperated with cyber investigations from other countries. None- theless, the rock-phish gang registered many .at domains, and they were not removed for a very long time. It turns out that the Austrian domain registrar nic.at balked at removing the illicit domains without more extensive evidence documenting their illegality. The domains were only removed following a public row between the e-mail-blacklist operator Spamhaus and nic.at [29] . Finally, after .at domains began getting removed promptly, the Turkish top-level domain .tk was targeted.

Meanwhile, many top-level domains have not been targeted by the gang. This includes domains such as Paraguay (.py), which we might have expected to be an easy target for attackers⁶, as well as domains such as .se, .edu, and .gov that are harder to register.

The many threats to payment card security, along with their expected and actual costs, are shown in Figure 12(b). Face-to-face retail fraud might reasonably be seen as the weakest link in the payment card environment; the reduction in face-to-face retail fraud following the adoption of EMV supports this view. Similarly, the banks correctly anticipated that losses due to credit cards lost or stolen inside the UK (L&S in the figure) would drop once PINs were required for use. By contrast, fraud rates due to cards stolen outside the UK were lower than the banks expected. Card fraud at UK ATMs is a bit harder than retail fraud since a PIN is required. One area where the banks’ expectations were not met is with ATM fraud on UK cards outside the UK. It turns out that fraudsters can easily clone stolen UK cards and use them in foreign ATMs; hence the true cost of attack is lower than expected.

The banks’ losses due to card-not-present fraud were much higher than forecast; unsurprisingly, many banks have now decided to deploy readers that verify PINs in order to carry out online banking. Meanwhile, uncer- tainty remains regarding the attacks on PIN entry devices uncovered in [25] and [26] , though it is apparent that the costs are higher than for card-not-present fraud and foreign ATM fraud.