TITLE:
Calibration of a Confidence Interval for a Classification Accuracy
AUTHORS:
Steen Magnussen
KEYWORDS:
Overall Accuracy, Producer’s Accuracy, User’s Accuracy, Area Proportions, Semi-Systematic Sampling, Post-Stratification, Stratified Random Sampling, One-Stage Cluster Sampling, Two-Stage Cluster Sampling
JOURNAL NAME:
Open Journal of Forestry,
Vol.11 No.1,
January
20,
2021
ABSTRACT: Coverage of nominal 95% confidence intervals of a
proportion estimated from a sample obtained under a complex survey design, or a
proportion estimated from a ratio of two random variables, can depart
significantly from its target. Effective calibration methods exist for
intervals for a proportion derived from a single binary study variable, but not
for estimates of thematic classification accuracy. To promote a calibration of
confidence intervals within the context of land-cover mapping, this study first
illustrates a common problem of under and over-coverage with standard
confidence intervals, and then proposes a simple and fast calibration that more
often than not will improve coverage. The demonstration is with simulated
sampling from a classified map with four classes, and a reference class known
for every unit in a population of 160,000 units arranged in a square array. The
simulations include four common probability sampling designs for accuracy
assessment, and three sample sizes. Statistically significant over- and
under-coverage was present in estimates of user’s (UA) and producer’s accuracy
(PA) as well as in estimates of class area proportion. A calibration with Bayes
intervals for UA and PA was most efficient with smaller sample sizes and two
cluster sampling designs.