1. Introduction

OJPsych

Open Journal of Psychiatry

2161-7325

Scientific Research Publishing

10.4236/ojpsych.2018.83025

OJPsych-86397

Articles

Biomedical&Life Sciences

Examination of Screening of the Pachinko/Pachislot Playing Disorder Based on Gambling Disorder Scales

Akiyo

Shoun

¹^*Akira

Sakamoto

²Kumiko

Akiyama

³Yasunobu

Komoto

⁴Taku

Sato

⁵Naoyuki

Nishimura

⁶Kikunori

Shinohara

⁷Hitoshi

Ishida

³Nobuo

Makino

⁸

Yoshino Hospital, Tokyo, Japan

Graduate School of Humanities and Sciences, Ochanomizu University, Tokyo, Japan

Naruse Mental Clinic, Tokyo, Japan

Tokyo University of Science, SUWA, Nagano, Japan

Recovery Support Network, Okinawa, Japan

Faculty of Core Research, Human Science Division, Ochanomizu University, Tokyo, Japan

Japan Women’s University, Tokyo, Japan

The Nikkoso Foundation for Safe Society, Tokyo, Japan

* E-mail:g1370309@edu.cc.ocha.ac.jp(AS);

08062018

08033153279, June 201828, July 2018 31, July 2018

2014

This work is licensed under the Creative Commons Attribution-NonCommercial International License (CC BY-NC).http://creativecommons.org/licenses/by-nc/4.0/

Pachinko and pachislot are unique forms of entertainment that dominate gambling activities in Japan. In this study, a web survey and interview were conducted in order to adjust three scales (DSM-5, SOGS, and PGSI) to pachinko/pachislot and explore an appropriate cutoff point for each of them. Using the assessment result in the interview as the outcome, the ROC curve of each scale was examined. The cutoff points were 4, 7, and 8 and the area under the curves (AUC) were 0.730, 0.783, and 0.724 for the DSM-5, SOGS, and PGSI, respectively. The study found that the type and situation of gambling influenced cutoff points in different ways depending on the questions asked in the scale.

Gambling Disorder Pachinko/Pachislot Playing Disorder ROC Curve Cutoff Point

1. Introduction

There have been reports on individuals with the gambling disorder in Japan. Ozaki [1] conducted a study in Japan and concluded that Japanese study participants scored higher on the South Oaks Gambling Screen (SOGS) than other countries. More specifically, 8.8% of male participants and 1.8% of female participants scored 5 points or more in Ozaki’s study while 0% to 2% of them did so in other countries. In Japan, horse racing, boat racing, bicycle racing, motorcycle racing, lottery, and sports promotion lottery (toto) are the six legal types of gambling. At the same time, Japan has unique games called pachinko and pachislot. They work like gambling but legally are leisure activities [2] . According to the Leisure White Paper 2017, pachinko and pachislot together made a 21.6 trillion yen market in 2016, much greater than the 5.9 trillion yen legal gambling market (the four types and lotteries combined) [3] . Moriyama [4] studied his psychiatric patients with a gambling addiction and discovered that pachinko was the cause of the addiction in most cases. He claimed that this was characteristic of Japanese gambling addiction.

Pachinko is like pinball. Players rent small metal balls from a pachinko parlor, feed them into a pachinko machine, shoot them upward inside the machine so that they fall while bouncing on pins installed like a maze on the backboard. Then, some will enter a small space hitting a jackpot, dispensing a large number of balls, while some just fall into a gutter at the bottom of the machine. Players who win extra balls can exchange them for prizes at a counter inside the parlor [5] . These prizes, then, can be sold to a neighboring shop for cash.

Despite the necessity to understand factors influencing gambling activities and address gambling problems, only a few studies have been conducted on pachinko/pachislot. Pachinko/pachislot is one of the major gambling games in Japan, and therefore, there has been demand for accurately identifying individuals who meet the pachinko/pachislot playing disorder criteria.

Across the world, various types of scales have been developed to rate the gambling disorder. For example, the DSM-5 uses nine statements to diagnose a gambling disorder. An individual is considered to have the disorder if he/she answers that four or more statements apply to him/her [6] . Lesieur & Blume [7] developed the SOGS to screen individuals with problem gambling behavior. The SOGS defines individuals who score 5 points or more out of 20 points as pathological gamblers. This cutoff point was found valid after its false-negative and false-positive rates were examined against the cutoff point of the DSM-III-R, which was used as a reference figure (a score of 4 points or more defines a pathological gambler). Ferris & Wynne [8] developed the Problem Gambling Severity Index (PGSI) with nine screening items, which are from the Canadian Problem Gambling Inventory (CPGI). Based on the DSM-IV (a score of 5 points or more defines a pathological gambler), the PGSI was analyzed and adjusted to maximize its sensitivity and specificity and compared with the SOGS and clinical interview data. As a result, its cutoff point was 8, meaning that a score of 8 points or more would define a problem gambler [9] .

Williams et al. [10] reported the types and frequency of scales used in prevalence study. According to them, the CPGI, DSM, Gamblers’ Anonymous 20 Questions (GA20), Problem and Pathological Gambling Measure (PPGM), SOGS, and Victorian Gambling Screen (VGS) were the main scales used. While 242 scales were administered in 202 studies, the SOGS was used 42.6% of the time, the DSM 26.4% of the time, the CPGI 22.7% of the time, and others (e.g., PPGM, GA20, VGS) were used 8.3% of the time. Although the use of the SOGS has been decreasing in general, it is still used frequently in Japanese gambling studies [1] [4] .

Since it became clear that previous gambling studies mainly used the DSM, SOGS, and CPGI, we chose to use them in the present study. In this study, a web survey was administered to those who played pachinko/pachislot within the last 12 months to measure their scores of pachinko/pachislot playing disorder based on these three scales. Then interviews were conducted with these respondents using the DSM-5 to decide whether or not they had the pachinko/pachislot playing disorder. Using the interview diagnosis results as reference points, we explored cutoff points for the three scales administered in the web survey. Note that, while previous studies used sensitivity to examine the cutoff points for these scales, this study used the ROC curve because it is used to evaluate cutoff points in recent studies (e.g., [11] ).

2. Method2.1. Procedure2.1.1. Web Survey

A web survey was conducted for those who played pachinko/pachislot within the last 12 months. These participants answered the three gambling disorder scales, namely DSM-5, SOGS, and PGSI, and also answered socio-demographic characteristic questions.

2.1.2. Interview

Interviews were conducted with the web survey respondents who also agreed to participate in an interview at a later date. The DSM-5 was used in the interview to decide whether or not the interviewees had the pachinko/pachislot playing disorder.

2.2. Participants

For this study, participants were selected for a web survey from panel members registered for a web survey company. Individuals who were 20 years or older, lived in Tokyo, Saitama, Chiba, or Kanagawa prefecture, and had played pachinko/pachislot within 12 months were selected. Some of them were later asked to participate in an interview. They were selected so that their scores on the Pachinko-Pachislot Playing Disorder Scale, developed by Akiyama et al. [12] , were distributed evenly between the highest and lowest scores. In this study, the participants were measured with the Pachinko-Pachislot Playing Disorder Scale in addition to the three gambling disorder scales discussed earlier. This sampling method was considered appropriate for analysis because even score distribution was necessary to draw an ROC curve. For this study, 119 of them accepted the request and participated in the interview, but data from two of them was discarded due to incomplete responses. At the end, data from 117 participants (99 males and 18 females) was analyzed. Their age ranged from 29 to 67 years old. The average age was 46.7 and the standard deviation was 9.65.

2.3. Study Period

The web survey was conducted in February 2015, and the interview was in March 2015.

2.4. Study Details

The three gambling disorder scales were administered after all instances of the word “gambling” were replaced with “pachinko/pachislot.” For example, the question “When you gamble, how often do you go back another day to win back money you lost?” was modified to “When you play pachinko/pachislot, how often do you go back another day to win back money you lost?” In the web survey, the participants were asked to answer the DSM-5, the SOGS, and the PGSI questions. The interview was conducted based on the DSM-5. The time frame used in these scales was unchanged in the SOGS and the PGSI while it was changed for the DSM-5. The SOGS had no description of any set time frame in its questions, and the web-based SOGS administered in this study had no included time frame either. The PGSI had the time frame description “the last 12 months.” The same description was used in the web survey. The DSM-5 used the phrase “in a 12-month period” to specify a time frame. This was used as it was in the interview but was modified to “the last 12 months” for the web survey. This is because the expression “in a 12-month period” may be confusing to the participants when presented as the instructions.

2.4.1. Web Survey

1) DSM-5

For the nine items regarding the gambling disorder in the Japanese version of the DSM-5 [13] , all instances of the word “gambling” were replaced with “pachinko/pachislot.” The score was calculated based on the number of items that were true to the participant.

2) SOGS

Among the 20 items in the SOGS, developed by Lesieur & Blume [7] , 19 of them were translated for this study. They were first back-translated by a researcher specialized in English and a translation agency, and then used as a reference for modifying the wording. The item excluded from the study was a question whether or not the participant would “issue a bad check” as a way to borrow money. It was excluded because it did not match the Japanese custom and was against the ethical agreement with the web survey company. The same choices and the scoring method as developed by Lesieur et al. were used. The participants answered each question using either “Yes” or “No.” The answer “Yes” was worth 1 point. The possible score was between 0 and 19.

3) PGSI

The nine items developed by Ferris & Wynne [8] were translated into Japanese after the word “gambling” was changed to “pachinko/pachislot.” The same choices and the scoring method as developed by Ferris et al. were used. The participants answered the four-point scale ranging from “Never” (0 points) to “Almost Always (3 points). The possible score was between 0 and 27.

4) Social-demographic characteristics

The participants were asked about their sex, age, household size, most recent education, and annual household income. For the household size, they were asked if they lived alone or with someone (two-person household or larger). For the most recent education, they were presented with choices ranging from junior high school to graduate school. Finally, they reported their annual household income using a 7-point scale ranging from less than 2 million yen to 12 million yen or more.

2.4.2. Interview

One-on-one interviews between an interviewer and a participant were conducted. The DSM-5 was used to identify the pachinko/pachislot playing disorder. As with the case of the web survey, the DSM-5 used in the interview was the Japanese version of the DSM-5 modified for pachinko/pachislot by Takahashi et al. [13] . Each translated and modified item was examined and then written as a question by three psychiatrists and four researchers specialized in psychology, education, sociology, and physiology respectively [14] .

Due to the time pressure and in consideration of ease of recall, we asked the participants about their disorder experience at the one time point which they could remember the quickest. The disorder score was the total number of questions that the participants responded as being applicable to them. The possible score range was from 0 to 9, and those who scored 4 or more points were diagnosed as having the pachinko/pachislot playing disorder.

For this study, individuals who were a certified clinical psychologist, psychiatric social worker, or certified social worker with clinical experience at a medical institute or academic interview experience were advertised for, and five of them were selected as interviewers. Under the supervision of a psychiatrist with rich clinical experience in treating individuals with the gambling disorder, the interviewers participated in the study for three days including training on the first day. The interviewers were required to immediately seek decisions of the supervisor whenever they came across uncertainties. They had a meeting every day, before and after the interview, in order to minimize differences between their decisions. Note that they spent approximately 10 minutes per interviewee to ask the DSM-5 questions.

2.5. Analysis Method

An ROC curve is a graphical presentation of the relationship between both sensitivity and specificity and it helps to decide the optimal model through determining the best threshold for the diagnostic test [15] . In this study, ROC curves for the three disorder scales were obtained after dividing the participants into the Disorder and No disorder groups in accordance with the DSM-5-based diagnosis made in the interviews. The area under the curve (AUC) is an indicator of classification accuracy. It ranges from 0.5 to 1, and a value closer to 1 means a higher predictive ability of a scale.

Based on the score on each subject scale, the cutoff points were calculated in two ways for both minor and severe disorder levels [16] . In method (a), the cutoff point was obtained where the difference between true-positive and false-positive rates was the largest (Youden’s index). In method (b), the cutoff point was obtained where the distance was smallest from the ideal point, defined by a 1.00 true-positive rate (sensitivity) and 0.00 false-positive rate (1 − specificity). In method (a), in order to set the score farthest away from the AUC = 0.500 line on the ROC curve as the cutoff point, the score where the difference between the true-positive and the false-positive rates was the largest was obtained. This means that the score where the result of “sensitivity + specificity − 1” was the smallest was obtained as the cutoff point. In method (b), in order to set the score on the ROC curve closest to the top left corner of the plot area as the cutoff point, the score with the smallest distance from the coordinates (0, 1) was obtained. This means that the score where the result of “(1 − specificity)² + (1 − sensitivity)²” was the smallest was obtained as the cutoff point.

3. Result

Table 1 shows the socio-demographic characteristics of the 117 participants. The participants consisted of 99 males and 18 females. Their ages ranged from their 20 s to 60 s with a mean of 46.7 years old (SD 9.65). Single-person households accounted for 30%, and multiple-person households accounted for 70%. The most frequently reported annual household income was 4 to 6 million, and the median was also 4 to 6 million. As for education, university graduates accounted for the most at 62% followed by senior high school graduates at 22%.

Diagnosis during the interviews found that 85 participants were suspected to have the disorder and 32 participants were suspected not to have it. ROC analysis of the AUC of the three scales resulted in 0.730 for the DSM-5, 0.783 for the SOGS, and 0.724 for the PSGI (Table 2). Figures 1-3 show the ROC curve for each scale.

For each scale, Table 3 shows the mean, standard deviation, traditional cutoff point, and score range. It also shows the cutoff point for each scale obtained by the two analysis methods described above.

3.1. DSM-5

Table 4 shows the DSM-5 score distribution by diagnosis as an interview result. The AUC of the ROC curve was .730 (p < 0.001), and the cutoff point was the score of 4 in both methods (a) and (b). Therefore, the candidate cutoff point was the score of 4 (sensitivity: 62.4% and specificity: 78.1%).

3.2. SOGS

Table 5 shows the SOGS score distribution by diagnosis as an interview result. The ROC curve was obtained and its AUC was 0.783 (p < 0.001), and the cutoff point was either (a) the score of 8 (sensitivity: 57.6% and specificity: 84.4%) or (b) the score of 7 (sensitivity: 64.7% and specificity: 75.0%).

3.3. PGSI

Table 6 shows the PGSI score distribution by diagnosis as an interview result. The ROC curve was obtained and its AUC was 0.724 (p < 0.001), and the cutoff point was the score of 5 in both methods (a) and (b) (sensitivity: 72.9% and specificity: 62.5%).

Table 1 Social-demographic characteristics of the participants

		n	%
Sex	Male	99	84.62
	Female	18	15.38
Age group	20 s	3	2.56
	30 s	23	19.66
	40 s	48	41.03
	50 s	28	23.93
	60 s	15	12.82
Household size	Single person	35	29.91
	Multiple persons	82	70.09
Annual household income	Less than 2 million yen	7	5.98
Annual household income	2 to less than 4 million yen	23	19.66
	4 to less than 6 million yen	31	26.50
	6 to less than 8 million yen	21	17.95
	8 to less than 10 million yen	21	17.95
	10 to less than 12 million yen	7	5.98
	12 million yen or more	7	5.98
Education	Junior high school	1	0.85
	Senior high school	26	22.22
	Specialized or vocational training school	11	9.40
	Two-year or technical college	4	3.42
	University	73	62.39
	Graduate school	2	1.71

Table 2 AUC in each subject scale

	DSM-5	SOGS	PGSI
AUC	0.730	0.783	0.724

Table 3 Statistics, candidate cutoff points, traditional cutoff points, and possible score range of the subject scales

	M	SD	Candidate cutoff points	Traditional cutoff points	range
DSM-5	3.84	2.68	4	4	0 - 9
SOGS	7.28	4.57	7 or 8	5	0 - 19
PGSI	7.32	5.58	5	8	0 - 27

Table 4 DSM-5 score distribution by diagnosis as an interview result

DSM-5 score	Diagnosis		Total
DSM-5 score	Disorder not suspected	Disorder suspected	Total
0	10	8	18
1	2	6	8
2	3	9	12
3	10	9	19
4	4	11	15
5	1	12	13
6	1	9	10
7	0	6	6
8	0	11	11
9	1	4	5
Total	32	85	117

Table 5 SOGS score distribution by diagnosis as an interview result

SOGS score	Diagnosis		Total
SOGS score	Disorder not suspected	Disorder suspected	Total
0	5	1	6
1	3	1	4
2	5	6	11
3	1	4	5
4	4	8	12
5	3	6	9
6	3	4	7
7	3	6	9
8	2	8	10

9	2	8	10
10	1	6	7
11	0	5	5
12	0	5	5
13	0	3	3
14	0	2	2
15	0	5	5
16	0	5	5
17	0	0	0
18	0	2	2
19	0	0	0
Total	32	85	117

Table 6 PGSI score distribution by diagnosis as an interview result

PGSI score	Diagnosis		Total
PGSI score	Disorder not suspected	Disorder suspected	Total
0	4	3	7
1	5	2	7
2	2	5	7
3	3	5	8
4	6	8	14
5	2	7	9
6	2	8	10
7	2	7	9
8	2	5	7
9	1	8	9
10	0	4	4
11	1	0	1
12	0	5	5
13	1	5	6
14	1	1	2
15	0	1	1
16	0	1	1
17	0	3	3
18	0	0	0
19	0	3	3
20	0	0	0

21	0	1	1
22	0	0	0
23	0	1	1
24	0	1	1
25	0	0	0
26	0	0	0
27	0	1	1
Total	32	85	117

4. Discussion

For the three gambling disorder scales, the cutoff points appropriate for pachinko/pachislot were examined.

In the DSM-5, the score of 4 was traditional cutoff point. In this study, the DSM-5 was modified for pachinko/pachislot and used in the interview. The score obtained in this interview was then used as a reference figure to examine the cutoff point for the DSM-5 questionnaire. The result indicated that the appropriate cutoff point was the score of 4 for both the interview and questionnaire. Note, however, that the time frame was expressed differently in the questionnaire and the interview. While it was expressed as “in a 12-month period” in the web-based questionnaire, it was expressed as “the last 12 months” in the interview. Therefore, it should be kept in mind that the time frame expressions may influence the survey results.

For the SOGS modified for pachinko/pachislot, the scores of 7 and 8 were candidate cutoff points. These values were higher than 5, the traditional cutoff point for gambling in general. Therefore, a higher cutoff value was appropriate when rating the gambling disorder caused by pachinko/pachislot.

For the PGSI modified for pachinko/pachislot, on the other hand, the candidate cutoff point was the score of 5. This was lower than 8, the traditional cutoff point for gambling in general. Therefore, a lower cutoff value was appropriate when rating the gambling disorder caused by pachinko/pachislot. Note, however, that later studies reported that the PGSI better rated the gambling disorder when its cutoff point was 5 instead of 8. If the score of 5 was to be used for gambling in general, it means these later studies arrived at the same figure as the present study [17] . The PGSI modified for pachinko/pachislot also better rated the disorder when the cutoff point was 5.

One of the reasons why each scale had different cutoff points for gambling in general and pachinko/pachislot would be the content of the scales. In the SOGS, 10 out of the 20 total items asked about borrowing money (in this study, 9 out of 19 items were about borrowing money). Since approximately half of the items dealt with financial issues, they would strongly influence the SOGS measurement result. Meanwhile, the DSM-5 had 1 item (out of 9) asking about the willingness to borrow money to play pachinko, and the PGSI had 2 items about borrowing money out of 9. Since the SOGS used in this study had a total of 19 items, one less than the unmodified version, after excluding 1 and asked the respondents about their “lifetime” experience, it is not appropriate to make a simple comparison among these scales. Still, different scale contents may have resulted in each scale having a different cutoff value when it was modified for pachinko/pachislot. The changes in the cutoff points suggest that pachinko/pachislot players are more likely to have financial issues than general gamblers. Even individuals who are not suspected to have the gambling disorder may still respond that financial issues are applicable to them, possibly influencing the screening of pathological pachinko/pachislot players in Japan. For this reason, it is appropriate that the SOGS modified for pachinko/pachislot has a higher cutoff value than the unmodified version for gambling in general.

In all three subject scales the AUC was higher than 0.7, showing that they were able to predict the disorder with moderate accuracy. Calculation of candidate cutoff points for these scales in two ways, including the use of Youden’s index, resulted in multiple candidates for the SOGS. The obtained cutoff points must be used carefully for scales with a low AUC and multiple candidate cutoff points.

This study has some limitations. For example, its sample size could not be considered large. Also, the sample representativeness may not be sufficient since the participants were registered panel members for a web survey company who lived in areas not far from the interview venue. For future studies, a larger sample size and participants recruited from a wider variety of areas are desirable.

While gambling disorder scales have been translated and used in many places, how individuals engage in gambling should vary with place. The present study examined cutoff values for three scales modified for pachinko/pachislot and some were quite different from the traditional cutoff values used in the unmodified versions of scales. This result suggests that, application of disorder scales to different gambling situations requires careful consideration about the viability of using existing scales without modifying them. Accurate understanding of the characteristics of scales and the gambling disorder will allow appropriate responses to gambling problems.

Conflicts of Interest

Kikunori Shinohara is on the board of trustees of Nichiyukyo (Japan Pachinko Pachi-Slot industry association).

Cite this paper

Shoun, A., Sakamoto, A., Akiyama, K., Komoto, Y., Sato, T., Nishimura, N., Shinohara, K., Ishida, H. and Makino, N. (2018) Examination of Screening of the Pachinko/Pachislot Playing Disorder Based on Gambling Disorder Scales. Open Journal of Psychiatry, 8, 315-327. https://doi.org/10.4236/ojpsych.2018.83025

References1

Ozaki, Y. (2014) 2013 National Survey on Drinking by Japanese Adult Population—Comparison with the 2003 and 2008 National Surveys. FY2013 Study Report for Research Using the Health and Labour Sciences Research Grants, 19-28. (In Japanese) https://mhlw-grants.niph.go.jp/niph/search/NIDD00.do?resrchNum=201315050A

Hirano, K. and Takahashi, K. (2003) Trends of Japan’s Giant Leisure Industry: Pachinko. UNLV Gaming Research & Review Journal, 7, 55-56.

Japan Productivity Center (2017) 2017 Leisure White Paper. Japan Productivity Publications. (In Japanese)

Moriyama

,et al. (2008)Clinical Features of 100 Pathological Gamblers Clinical Psychiatry 50, 895-904.

Sibbitt

E.C.

,et al. (1997)Regulating Gambling in the Shadow of the Law: Form and Substance in the Regulation of Japan’s Pachinko Industry Harvard International Law Journal 38, 568-569.

American Psychiatric Association (2013) Diagnostic and Statistical Manual of Mental Disorders. 5th Editoin, American Psychiatric Association, Washington DC.

Lesieur, H.R. and Blume, S.B. (1987) The South Oaks Gambling Screen (SOGS): A New Instrument for the Identification of Pathological Gamblers. American Journal of Psychiatry, 144, 1184-1188. https://doi.org/10.1176/ajp.144.9.1184

Ferris, J. and Wynne, H. (2001) The Canadian Problem Gambling Index. Canadian Centre on Substance Abuse, Ottawa.

Wynne, H.J. (2003) Introducing the Canadian Problem Gambling Index. Wynne Resources, Edmonton.

Williams, R.J., Volberg, R.A. and Stevens, R.M. (2012) The Population Prevalence of Problem Gambling: Methodological Influences, Standardized Rates, Jurisdictional Differences, and Worldwide Trends. Ontario Problem Gambling Research Centre.

Himelhoch, S.S., Miles-McLean, H., Medoff, D.R., Kreyenbuhl, J., Rugle, L., Bailey-Kloch, M., Potts, W., Welsh, C. and Brownley, J. (2015) Evaluation of Brief Screens for Gambling Disorder in the Substance Use Treatment Setting. The American Journal on Addictions, 24, 460-466. https://doi.org/10.1111/ajad.12241

Akiyama, K., Sakamoto, A., Shoun, A., Komoto, Y., Sato, T., Nishimura, N., Shinohara, K., Ishida, H. and Makino, N. (2016) Development of Pachinko-Pachislot Playing Disorder Scale. Clinical Psychiatry, 58, 307-316. (In Japanese)

Takahashi, S., Ohno, Y., Someya, T., Kamba, S., Ozaki, N., Mimura, M. and Murai, T. (2014) Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (Japanese version). Igaku-Shoin. (In Japanese)

Akiyama, K., Sakamoto, A., Shoun, A., Komoto, Y., Sato, T., Nishimura, N., Shinohara, K., Ishida, H. and Makino, N. (2017) Study on Pachinko-Pachislot Disorder by Use of DSM-5: Measuring Samples of Ordinary Players. Japanese Journal of Addiction & Family, 32, 143-151. (In Japanese)

Zhu, W., Zeng, N. and Wang, N. (2010) Sensitivity, Specificity, Accuracy, Associated Confidence Interval and ROC Analysis with Practical SAS Implementations. NESUG Proceedings: Health Care and Life Sciences, Baltimore, Maryland. http://www.nesug.org/Proceedings/nesug10/hl/hl07.pdf

Akobeng, A.K. (2007) Understanding Diagnostic Tests 3: Receiver Operating Characteristic Curves. Acta Paediatrica, 96, 644-647. https://doi.org/10.1111/j.1651-2227.2006.00178.x

Williams, R.J. and Volberg, R.A. (2010) Best Practices in the Population Assessment of Problem Gambling. Faculty of Health Sciences.