This study assessed agreement between two measures of medicine use, self-report by mail and pharmaceutical claims data, for a national sample (N = 4687) of older women aged 79 to84 in2005, from the Australian Longitudinal Study on Women’s Health. Medicines used for common chronic diseases in older people were selected, with pharmaceutical claims data retrieval periods of three and six months. For six month retrieval, Kappa’s ranged between 0.44 (nervous system medicines) and 0.94 (glucose lowering medicines). For three month retrieval, aspirin (Kappa: 0.35) and folic acid (Kappa = 0.48) had lowest agreement. Women were least able to accurately report use of nervous system medicines (sensitivity < 50%), and most accurately report glucose lowering medicines use (sensitivity > 80%). Specificity was consistently high across all classes, suggesting women could accurately report using a medicine. Pharmaceutical claims data can assist evaluation of judicious medicines use, changes to availability and uptake of medicines, and track medicine expenditure for chronic conditions. Over-the-counter medicines, medicines not covered by pharmaceutical subsidies and those used on an as needed basis may be best measured by self-report, as use may be underestimated using pharmaceutical claims data.
Linking routinely collected administrative data on health and health service use to self-report data from surveys can enhance the breadth and efficiency of population health research. Record linkage also provides a mechanism to assess the accuracy and consistency of each data source. Researchers and clinicians need to be aware of the strengths and limitations of using various sources of data for decision-making. Pharmaceutical claims data can be used to evaluate the appropriate use of medicines, and the impact of changes to the availability and uptake of medicines. These data can also be used to track medicines expenditure, particularly for chronic conditions. Hence it is important to know how closely pharmaceutical claims data compare to self-reported medication use and whether the discrepancies between the two sources are more pronounced for some types of medications.
The benefits and limitations of collecting and analysing self-reported medicines use have been addressed in previous research [1]. For example, self-report can be used to measure use of over-the-counter medicines and to assess compliance behaviours such as adherence to timing directions, which are often not measured in pharmaceutical claims databases. However, the quality of survey data on self-reported medicines use depends on the accuracy of recall of participants. Inaccurate recall of medicines use can lead to misclassified medicines exposure and incorrect risk and prevalence estimates of medicines use in case control studies [2,3], randomised controlled trials and population and longitudinal studies. Accuracy and recall in the self-report of medicines use depend on a number of research design, medicines related and participant factors. Research design factors include question structure [4], interviewer skills and length of recall period. Important medicines related factors impacting on medicines recall are, for example, class of medicines [1], regularity and frequency of use [1,5] or seriousness of conditions for which used [5]. Recall can also depend on participant characteristics, such as income [6] and living alone [3,6] but generally gender has no impact [3,7-9]. The impact of participant characteristics on medicines recall generally varies according to the medicines class under investigation [3]. There have been mixed results on the impact of age on medicines recall with some studies reporting statistically significant associations [6] but most not [7-10], although participants with age-related memory problems tend to over-report medicines use [11]. It is important to understand the reliability and accuracy of self-reported medicines use amongst older people given they are the largest users of medicines and commonly have multiple medical conditions. Studies in Australia [1], the Netherlands [8,12], and the US [13] have concluded that pharmaceutical claims databases can be a useful tool to measure medicines exposure for prescription medicines amongst older people. However, study samples are generally drawn from very specific populations such as local general practice [1], inner city [12] or health maintenance organizations [13] which makes it difficult to generalise the results to population based studies. In this paper, we draw on a national sample of community-living older women that is broadly representative of the national population of older women [14] in Australia to estimate the agreement and accuracy of self-report survey medicines data by comparing it with pharmaceutical claims data as the reference standard.
2. MATERIALS AND METHODS2.1. The Australian Longitudinal Study on Women’s Health
The Australian Longitudinal Study on Women’s Health was designed to investigate multiple factors affecting the health and well-being of women. In 1996, women born in the years 1921-1926 (aged 70 - 75 years) were randomly selected from the national Medicare database, with over-representation of women living in rural and remote areas [14]. Medicare includes all Australian citizens and permanent residents, regardless of age or income. The women were sent a mail survey. The baseline survey, survey 1 (S1) was completed by 12,432 women and these women have now been surveyed five times over a 13-year period (1996-2008). Data on medicines taken by the women were available from two sources: self-report of their prescribed medicines and pharmaceutical claims data for the same year.
2.2. Coding of Self-Reported Medicines
This study uses data from Survey 4 which was conducted in 2005 (S4 n = 7158) when the women were aged 79 - 84 years. Parts of the methods and data have been reported elsewhere [15]. The women were asked: “Please write down the names of all your medicines prescribed by a doctor. Where possible, copy names from the packets, or obtain a list from your regular pharmacist and return it with your survey.” Participants recorded their medicines in open-ended text-format (see www.alswh.org.au). Coding occurred in two steps [15]:
1) Information provided by the women was entered into a database containing four fields: medicine name, dosage, frequency and any other information.
2) The medicines data were then coded according to two standard medicines classifications. First, each medicine was assigned a value for the pharmaceutical name. This information was derived from the Australian Statistics on Medicines (ASM) [16]. Second, all medicines were classified according to the Anatomical Therapeutic Chemical (ATC) Classification System 2001 [17]. This information was entered into a database as a medicines list. To enable analysis, each medicine was coded to an Anatomical Therapeutic Chemical (ATC) group according to the World Health Organisation (WHO) definitions. Each new medicine that was entered, was checked against the existing list of medicines in the database. If a new medicine was found, it was cross referenced with the Monthly Index of Medical Specialties (MIMS) [18], the Pharmaceutical Benefits Scheme and the World Health Organisation Anatomical Therapeutic Chemical Classification [15].
2.3. Coding of Pharmaceutical Claims Data
In Australia, pharmaceutical claims data are processed through Medicare Australia under the Pharmaceutical Benefits Scheme (PBS) and the Repatriation PBS (RPBS) for Department of Veterans’ Affairs cardholders. Pharmaceutical claims data have some limitations. First, when a person reaches a certain monetary threshold (Safety Net Threshold), they are allocated a new identifying number which applies to families and not individuals [1]. However, the majority of older women in the Australian Longitudinal Study on Women’s Health are concession cardholders (94%) [19] which improves the completeness of the dataset. Second, pharmaceutical claims data does not cover all medicines, specifically medicines that are provided in hospital, bought over-the-counter and complementary medicines. Additionally, pharmaceutical claims data exclude prescription medicines that are not subsidized through the scheme. If the cost of a medicine is below $5.60 for concession cardholders or $32.60 for nonconcession cardholders, the women pay for the full cost of the medicines, and the prescription will therefore not be recorded on the pharmaceutical claims database. Hence, pharmaceutical claims data provide an appropriate and objective, but not perfect, comparison for self-reported use of medicines. Medicines listed in the Pharmaceutical Benefits Scheme are coded on the basis of the purpose of the medicines (such as the patient’s diagnosis or prognosis) rather than its chemical composition. Each pharmaceutical item number for dispensed medicines was also coded to an Anatomical Therapeutic Chemical (ATC) group according to the World Health Organisation (WHO) definitions [15,17].
2.4. Selected Medicines
For this study, all medicines are covered under the Pharmaceutical Benefits Scheme but some medicines use may not be recorded for women who have reached the Safety Net, or when medicines such as aspirin that are also available over-the-counter are purchased without prescription. Appendix shows the selected medicines groups, generic names and matching ATC codes [15]. These medicines were selected because they are used for common chronic conditions such as diabetes, hypertension, depression and anxiety [20] Ethics approval was received from the University of Newcastle Human Research Ethics Committee.
2.5. Statistical Analysis
Agreement was assessed by determining whether selfreport and pharmaceutical claims data for each woman identified medicines in the same ATC class. Kappa coefficients and their 95% confidence intervals were calculated. The sensitivity, specificity and positive and negative predictive values of the self-report data were also calculated, with claims data as the reference standard. Use of these tests in combination with agreement data will often enhance the interpretation of data quality. Also, since it is important to investigate what effect different retrieval periods prior to the survey date have on the accuracy of self-report as determined by the pharmaceutical claims dataset, two cut-points for the time period were used to define “medicines used” based on the specific Australian situation: three and six months prior to the date of the woman completing the survey. Three and six months were chosen because repeat prescriptions are usually intended to last up to six months. Additionally, the majority of the selected medicines are taken regularly and unlikely to be discontinued without a replacement from another medicine in this class.
3. RESULTS
A total of 5494 women provided consent for their survey data and pharmaceutical claims data to be linked [15].
When sociodemographic factors were compared between women who consented and those who did not, no differences were found between consenters and nonconsenters in relation to self-rated health, diabetes, Body Mass Index, and number of general practitioner visits. Consenters were better educated and were more likely to be able to manage on their available income (P < 0.0001). There were also small but significant differences between consenters and non-consenters by area of residence (P < 0.0001). Compared to non-consenters, a higher proportion of consenters lived in regional areas and a lower percentage of consenters lived in outer regional and remote areas [15]. Of the 7158 women that returned the Survey, 6,495 (90.7%) completed the self-report medicine question [15]. Of these 6495 women, 4687 (66%) consented to the release of their pharmaceutical claims data. Of these women, 392 women (8%) did not record any medicines [15]. The pharmaceutical claims data showed that 338 (7%) women did not claim any medicines during a three months data retrieval period; and 233 (5%) of women did not use any medicines during a six months period (Table 1(a)). The longer the retrieval period the more likely it was that the women claimed a medicine subsidy from the PBS.
Table 1(b) shows the agreement between self-report and pharmaceutical claims data in terms of whether women use any medicines or not. Women both under and over-report medicines use when compared to pharmaceutical claims data. For example, under-reporting occurred for 240 women who reported that they did not use any medicines but pharmaceutical claims data (three months) revealed that they had prescriptions filled. ConverselyTable 1. Self-reported medicines data compared to pharmaceutical claims data with three and six months retrieval periods (n = 4687) [15].
*Sample excluded women that failed to return their survey in 2005.
over-reporting occurred for 186 women who reported that they were using a medicine at the time of the survey but none could be found in the pharmaceutical claims dataset at three months. However, this number declined to 104 when using a six month retrieval period. A total of 152 women (3.2%) had no medicine documented in either the survey data or the pharmaceutical claims data (three months) and 129 (2.8%) had no medicine documented in either the survey data or pharmaceutical claims data (six months). The pharmaceutical claims data showed a higher mean and median number of medicines used compared to self-report data.
Table 2 displays the agreement and accuracy of selfreport compared to pharmaceutical claims data. The prevalence of medicines use is higher in the pharmaceutical claims data for the six month retrieval period than selfreport data, except for aspirin [15]. The prevalence of medicines use in the pharmaceutical claims data for three month retrieval periods is also generally higher in pharmaceutical claims data, except for insulin, thiazide, aspirin and folic acid. In particular, aspirin use is twice as high in self-reported data than in pharmaceutical claims
Table 2. Prevalence of self-report (SR) and pharmaceutical claims data (PBS), kappa (K), sensitivity, specificity and positive and negative predictive values of self-report compared to pharmaceutical claims data for older women (N = 4687), 2005 [15].
*95% CI = 95% confidence interval, **Anxiolytics, hypnotics and sedatives.
data [15]. Kappas are generally higher for the six month retrieval period than the three month retrieval period, with aspirin, folic acid, medicines used for the nervous system and insulin having the lowest agreement.
Sensitivity declines with longer retrieval periods, whereas specificity increased with longer retrieval periods. Similarly, positive predictive values increased with longer retrieval periods whereas negative predictive values decreased. Sensitivity was lowest for medicines used for the nervous system (including anxiolytics, hypnotics and sedatives which may be used on an as needed basis) with a sensitivity of only 37% when using a six month retrieval period.
Women were most likely to accurately report glucose lowering medicines, thiazides, angiotensins, betablockers and statins (sensitivity > 80%). Specificity was consistently high across all medicines classes. Positive and negative predictive values measure how well a test performs in a given population with a given prevalence of use and are not fixed. Positive predictive values were lowest for aspirin, folic acid, and insulin (three months). The low positive predictive value of 56% for insulin means that amongst those who report insulin use only 56% are true positives according to pharmaceutical claims records and this may be explained by the low prevalence of insulin (0.9%). Negative predictive values reached high to very high levels.
4. DISCUSSION
The purpose of the present study was to estimate the agreement and accuracy of self-report survey medicines data by comparing it with PBS/RPBS claims data as the reference standard. Good to excellent agreement was found between self-report compared to the pharmaceutical claims dataset for repeat medicines. Aspirin, folic acid and medicines used for the nervous system had the lowest agreement and women were also least able to accurately report the use of these medicines. Women were most likely to accurately report glucose lowering medicines, thiazides, angiotensins, betablockers and statins (sensitivity > 80%). Specificity was consistently high across all medicine classes, suggesting that if women reported using a medicine this was likely to be valid.
Prevalence of medicines use was generally higher in pharmaceutical claims data except for aspirin intake at three and six months and folic acid, insulin and thiazides at three months and these medicines generally had low positive predictive values for the three month period. These effects could be accounted for by over-the-counter purchases of aspirin and folic acid which will not appear in the pharmaceutical claims data, if women received more than three months’ supply of the medicine each time it was dispensed, or if women had less than optimal compliance and so the prescription lasted longer than three months. Likewise medicines used on an as needed basis will also not necessarily show high levels of agreement or predictive values. Sensitivity declined with increased retrieval periods since women may no longer be using medicines prescribed in the more distant past.
Consenters were better educated and were more likely to be able to manage on their available income. However, most studies have not demonstrated a relationship between education [8,9,21] and income [21] with recall accuracy. Additionally, the bias will vary per medicine class. Haapea et al. [3] showed that more education only led to better recall for antidepressants but not for betablockers, antidiabetics, antiepileptics and antipsychotics. Poor health has also been shown to be related to poorer self-report of medicines amongst low-income older adults [5] but no differences were found in our study between consenters and non-consenters in relation to self-rated health, diabetes, Body Mass Index, and number of general practitioner visits which suggest that this potential bias may not be relevant in our study. Some of the differences in agreement may be explained by medicines that are not recorded in the pharmaceutical claims dataset, for example when women reach the Safety Net or their medicines fall below the co-payment threshold, or receive medicines while hospitalised.
Care should be taken when comparing the results with other studies given the variability in methods used such as differing recall period, study populations, question structure and comparators. The 8% of women who reported not using any prescription medicines is similar to another Australian study using the same methods, recall period, database and similar question structure in an older population [1,22]. The main difference was that this was a sample selected from a general practice population and included men and women. Although the medicines classes were slightly different, the accuracy for antihyperglycaemics was reasonably similar to the glucose lowering medicines category used in this study [1]. Both studies confirmed that older people report cardiovascular and glucose lowering medicines accurately. The prevalence for medicines used for the nervous system was similar between both studies (10%) but the Kappa was much lower in this community study when compared to the general practice study (0.47 versus 0.89) as was sensitivity (43% versus 74%). Finally, while the general practice population reported a much higher thiazide diurectics use (19%) than did the Australian Longitudinal Study on Women’s Health sample (9%), both studies showed very high agreement and validity for this commonly used class of medicines. Caskie [5] noted in an American study that agreement between self-report and pharmaceutical claims data was higher for medicines with a higher prevalence. This was in general not the case in the current study. It is possible that given this study has a much larger sample size that low prevalence becomes less of an issue and more precise estimates of sensitivity, specificity and predictive values can be determined.
The study had some limitations. The data did not allow for analyses of compliance issues. Additionally, the differences between national pharmaceutical claims databases can make comparison between countries difficult. Agreement and accuracy cannot be generalised to other medicines classes or other populations but can be useful for other researchers working with self-report data. A strength of this study was that this is the first study that has compared self-reported medicines use with pharmaceutical claims data for a nationally representative sample of older women.
The generalisability of the study findings is confined to older women living in the community. Measuring medicines use through pharmaceutical claims database has limitations. Care must be taken when using pharmaceutical claims data as a source of information about medicines that can be bought over-the-counter or that are used as needed. Medicines that are not covered under the PBS/ RPBS scheme will also be under-represented in pharmaceutical claims data and self-report may be a better source of information on the use of these medicines. Appropriate consideration must be given to the pharmaceutical claims data retrieval period to understand the accuracy of self-report data [1,12]. It is also important to understand the intricacies of the pharmaceutical claims data to determine whether pharmaceutical claims data or self-report produce more valid results. For example, if a researcher knows that a medicine falls under a co-payment threshold, it may be better to use self-report rather than claims data.
This study demonstrated that medicines that can be bought over-the-counter (aspirin and folic acid) should be measured by self-report rather than claims data. Medicines that may be used sporadically and that can not be bought over-the-counter such as benzodiazepines benefit less from self-report especially if the recall period increases. Medicines that are used daily, and can not be bought over-the-counter (glucose lowering drugs, thiazides, angiotensins, beta blockers, and statins) can accurately be self-reported by older women.
In conclusion, for several medicine classes, high agreement and accuracy were demonstrated for self-reported use of medicines of older women when compared with pharmaceutical claims data. Pharmaceutical claims data and self-reported medicines use can be useful in the evaluation of medicines use. Pharmaceutical claims data can also be used to track medicines expenditure, particularly for chronic conditions requiring repeat prescriptions. Studies that require measurement of medicines that can be bought over-the-counter or that are not covered by pharmaceutical claims data or medicines used on an as needed basis would benefit from using self-report methods and must take care when using pharmaceutical claims data to measure use of these types of medicines.
5. ACKNOWLEDGEMENTS
The research on which this paper is based was conducted as part of the Australian Longitudinal Study on Women’s Health, the University of Newcastle and the University of Queensland. We are grateful to the Australian Government Department of Health and Ageing for funding and to the women who provided the survey data. Researchers in the Faculty of Health at the University of Newcastle are also members of the Hunter Medical Research Institute.
Funding: SP was supported by the Australian National Health and Medical Research Council via an Australian Research Training Fellowship Part-time (ID: 511998). The funding body played no role in the design, execution, analysis and interpretation of data, or writing of the study.
REFERENCESAPPENDIX
Medicines groups, generic names and ATC-codes [15].
NOTESReferencesPit, S.W., Byles, J.E. and Cockburn, J. (2008) Accuracy of telephone self-report of drug use in older people and agreement with pharmaceutical claims data. Drugs & Aging, 25, 71-80.
doi:10.2165/00002512-200825010-00008Boudreau, D.M., Daling, J.R., Malone, K.E., Gardner, J.S., Blough, D.K. and Heckbert, S.R. (2004) A validation study of patient interview data and pharmacy records for antihypertensive, statin, and antidepressant medication use among older women. American Journal of Epidemiology, 159, 308-317. doi:10.1093/aje/kwh038Haapea M., Miettunen J., Lindeman S., Joukamaa M. and Koponen, H. ,et al. (2010)Agreement between self-reported and pharmacy data on medication use in the Northern Finland 1966 birth cohort International Journal of Methods in Psychiatric Research 19, 88-96.Klungel, O.H., de Boer, A., Paes, A.H., Herings, R.M., Seidell, J.C. and Bakker, A. (1999) Agreement between self-reported antihypertensive drug use and pharmacy records in a population-based study in The Netherlands. Pharmacy World & Science, 21, 217-220.
doi:10.1023/A:1008741321384Caskie, G.I.L. and Willis, S.L. (2004) Congruence of self-reported medications with pharmacy prescription records in low-income older adults. The Gerontologist, 44, 176-185. doi:10.1093/geront/44.2.176Cotterchio, M., Kreiger, N., Darlington, G. and Steingart, A. (1999) Comparison of self-reported and physician-reported antidepressant medication use. Annals of Epidemiology, 9, 283-289. doi:10.1016/S1047-2797(98)00072-6West S.L., Savitz D.A., Koch G., Strom B.L., Guess H.A. and Hartzema, A ,et al. (1995)Recall accuracy for prescription medications: Self-report compared with database information American Journal of Epidemiology 142, 1103-1112.Sjahid, S.I., van der Linden, P.D. and Stricker, B.H. (1998) Agreement between the pharmacy medication history and patient interview for cardiovascular drugs: The Rotterdam elderly study. British Journal of Clinical Pharmacology, 45, 591-595.
doi:10.1016/S1047-2797(98)00072-6Brown, D.W., Anda, R.F. and Felitti, V.J. (2007) Self-reported information and pharmacy claims were comparable for lipid-lowering medication exposure. Journal of Clinical Epidemiology, 60, 525-529.
doi:10.1016/j.jclinepi.2006.08.007Grimaldi-Bensouda, L., Rossignol, M., Aubrun, E., El Kerri, N., Benichou, J. and Abenhaim, L. (2010) Agreement between patients’ self-report and physicians’ prescriptions on cardiovascular drug exposure: The PGRx database experience. Pharmacoepidemiology and Drug Safety, 19, 591-595. doi:10.1002/pds.1952Johnson, T. and Fendrich, M. (2005) Modeling sources of self-report bias in a survey of drug use epidemiology. Annals of Epidemiology, 15, 381-389.
doi:10.1016/j.annepidem.2004.09.004Lau, H.S., de Boer, A., Beuning, K.S. and Porsius, A. (1997) Validation of pharmacy records in drug exposure assessment. Journal of Clinical Epidemiology, 50, 619-625. doi:10.1016/S0895-4356(97)00040-1Johnson R.E. and Vollmer, W.M. ,et al. (1991)Comparing sources of drug data about the elderly Journal of the American Geriatrics Society 39, 1079-1084.Brown, W.J, Bryson, L., Byles, J.E., Dobson, A.J., Lee, C., Mishra, G., et al. (1998) Women’s health Australia: Recruitment for a national longitudinal cohort study. Women & Health, 28, 23-40.
doi:10.1300/J013v28n01_03Byles, J., Loxton, D., Berecki, J., Dolja Gore, X., Gibson, R., Hockey, R., et al. (2008) Use and costs of medications and other health care resources: Findings from the Australian Longitudinal Study on Women’s Health. Report to Department of Health and Ageing.
http://www.alswh.org.au/images/content/pdf/major_reports/2008_major_report_c_r144.pdfDrug Utilisation Sub-Committee (DUSC). (2004) Australian statistics on medicine 2001-2002. Department of Health and Ageing, Canberra.WHO Collaborating Centre for Drug Statistics Methodology. (2001) Anatomical therapeutic chemical (ATC) classification index with defined daily doses (DDDs). WHO Collaborating Centre for Drug Statistics Methodology, Oslo.MIMS Australia. MIMS. St Leonards MIMS Australia.Berecki-Gisolf, J., Hockey, R. and Dobson, A. (2008) Adherence to bisphosphonate treatment by elderly women. Menopause, 15, 984-990.
doi:10.1097/gme.0b013e31816be98aAustralian Bureau of Statistics (2009) National health survey: Summary of results, 2007-2008. ABS Catalogue No. 4364.0, ABS, Canberra.Metlay, J.P., Hardy, C. and Strom, B.L. (2003) Agreement between patient self-report and a veterans affairs national pharmacy database for identifying recent exposures to antibiotics. Pharmacoepidemiology & Drug Safety, 12, 9-15. doi:10.1002/pds.772Pit, S.W., Byles, J.E. and Cockburn, J. (2008) Prevalence of self-reported risk factors for medication misadventure among older people in general practice. Journal of Evaluation in Clinical Practice, 14, 203-208.
doi:10.1111/j.1365-2753.2007.00833.x
