To maintain methodological consistency with Perkins’ skill score calculation described in the next subsection and adapted from Perkins et al. (2007) [28] , daily values of minimum and maximum temperature and mean precipitation from the dynamically downscaled results and observed data were extracted for the period 1970-1999 from grid points located inside and within a half-degree of the boundaries of the two Southeast sub-regions (east and west). Kjellstrom et al. (2010) [18] found the same skill score metric based on daily data excluding dry events, or monthly data including both wet and dry events, gave very different results in terms of model ranking relative to precipitation output. This was most likely a result of a difference in the underlying probability density functions (PDFs). The same daily data values extracted for the Perkins method were also used for calculation of Willmott’s index of agreement, RMSE, and MAE. For comparison of the NARCCAP ensemble members to the NARR and NCEP-driven RCM hindcasts, daily data from 1979-1999 were extracted from the same grid points. A half-degree buffer was chosen due to the spatial resolution of the data and that the data points may not completely lie on or within state boundaries, yet are representative of some portion of the region and sub-region. By extracting daily values at each grid point, it is possible to gauge a model’s ability to simulate day-to-day variability rather than long-term averages, giving a proper gauge for testing a model’s worth.

Although individual grid points were extracted within the six states classified as the Southeast U.S., all analysis was conducted by aggregating data to the sub-regional scale (rather than grid point by grid point) for several reasons. First, the PDF skill score metric was designed by Perkins et al. (2007) [28] for determination of GCM skill across all of Australia (for the entire country rather than at individual grid locations) and is used in the same manner for regional climate model skill assessments [15] [29] . To maintain methodological consistency with others studies using Perkin’s method, aggregation was necessary. Additionally, although Kjellstrom et al. (2010) [15] display skill scores spatially throughout Europe, they average individual grid skill scores to obtain a single skill score for each of their defined sub-regions. Second, aggregating data into sub-regions allows for meso- and synoptic-scale pattern recognition which may not be evident when analyzing at each grid location. To further this point, GCMs were not meant to represent conditions at a single grid point but are more representative of the synoptic-scale (100 s to 1000 s of kilometers). Although RCMs are still not representative of a single grid location they are representative of the meso-scale (10 s to 100 s of kilometers). Third, most crop and hydrological assessments aggregate information from the single grid point scale to a field (or multi-field) or watershed-scale [30] [31] and assessing skill and bias on a sub-regional level is adequate. Lastly, although the information contained within an individual RCM grid better represents conditions within the grid point, models run at 50 km resolution will still have issues with pinpointing information at a specific location (e.g., town, city, weather station, etc.).

Characterization of climate model forecast skill begins with determining how closely a model’s output matches observations. Model skill was measured against the Maurer gridded observed dataset [25] for the Southeast U.S. for monthly minimum and maximum temperature and mean precipitation (utilizing daily data) during a historical reference period, 1970-1999. Skill was determined through calculation of four metrics: probability density function overlap [28] , an index of agreement [32] , root mean square error, and mean absolute error.

Another important aspect of this research was to determine why a model may lack skill. A first step towards this is to determine whether bias occurs in a particular part of a variable’s frequency distribution. To this end, Kjellstrom et al. (2010) [18] computed area-average model bias at the 1^st, 5^th, 10^th, 25^th, 50^th, 75^th, 90^th, 95^th, and 99^th percentiles for each of their sub-regions, variables, and models. This method allows for determination of where, within the probability density function (or distribution), a model under or over predicts a given variable, leading to degradation in skill. For example, a model with a warm temperature bias in the lower percentiles (the coldest days) may decrease the number of cold air outbreaks and potential decrease snow cover in the region during winter. A model with a warm summer bias in the upper percentiles could cause over prediction of heat waves and drought. In this study, model bias at each percentile was computed for each combination of RCM-NCEP output from NARCCAP Phase I, RCM-GCM results from NARCCAP Phase II, and the three GCMs from the 4^th IPCC assessment.

Each member of the NARCCAP ensemble replicates observed minimum temperature well. Perkins skill scores for most RCMs (with the exception of the WRFG models, and RCM3-GFDL and ECP2-GFDL) remain above 0.7 for each month. Index of agreement values generally exceeds 0.5 in each month in both eastern (Figure 2(a) and Figure 2(b)) and western portions (Figure 3(a) and Figure 3(b)) of the study region. RMSE and MAE values typically remain below 3˚C for most models (Figure 2(c) and Figure 2(d); Figure 3(c) and Figure 3(d)). For the WRFG RCMs, ECP2-GFDL, and GFDL-timeslice, July and August show the worst skill. The degradation in skill for the WRFG RCMs and ECP2-GFDL are attributed to a cold bias between 1˚C to 3˚C as illustrated for July in Figure 4(c). Additionally, the WRFG model illustrates a slight cold bias in the same months for the NCEP-driven runs (e.g., Figure 5(c)). The ECP2-GFDL’s cold bias in July and August must be a function of the GFDL LBCs, as the NCEP-driven runs of the ECP2 model illustrate a warm bias in all months between 1˚C and 4˚C (Figure 5(c)). The GFDL-timeslice, on the other hand, exhibits a warm bias of 2˚C to 5˚C from the 50^th through 99^th percentiles. The cold bias presented in the GFDL GCM over the Southeast U.S. is noted in Freidenreich and Ramaswamy (2011) [57] by a deficit in modeled downward shortwave radiationflux compared to observations. Further, the authors conclude the deficit in downwelling shortwave radiation is due to an over prediction of total cloud amount when compared to observations. The best performing models are the RCM3- CGCM3 and CRCM-CGCM3 which show very little warm or cold bias, remaining within ±2˚C of observations across all percentiles and months, and only a 1˚C to 2˚C warm bias in the NCEP-driven runs. Although a bias of 1˚C to 2˚C may seem rather large, it falls within a commonly known and accepted threshold in climate models [58] [59] .

Interestingly, the two models driven by GFDL exhibit degradation in skill in the winter and early spring, an observation not found in the other ensemble members. Percentile plots reveal these RCMs suffer from significant cold bias between 4˚C and 8˚C. This strong cold bias is passed from the GFDL model to each RCM (Figure 6). Conversely, the CCSM GCM exhibits a warm bias between 1˚C and 5˚C in most months which is tempered

slightly in the downscaling process by both the MM5I and CRCM, resulting in a slight warm bias of not more than 2˚C in any given month.

Percentile plots for the west sub-region reveal the RCM3-GFDL and ECP2-GFDL models have a more pronounced cold bias than the eastern sub-region, with cold biases from December through March on the order of 4˚C to 10.5˚C for the lower 50^th percentiles. The cold bias must be a function of the GFDL LBCs as both the ECP2 and RCM3 RCMs have a warm bias of 1˚C to 3˚C from December through March with the NCEP-driven runs. Conversely, the GFDL-timeslice’s warm bias observed in the east sub-region (2˚C to 5˚C warm bias in above-median temperatures) is even greater in the west sub-region (3˚C to 8˚C warm bias in above-median temperatures), the result of a strong warm bias in the same percentiles with the driving GCM. This bias during the warmest months of the year, especially in the upper percentiles, may have the potential to lead to an increase in nocturnal evaporation from the soil [60] . Enhanced nocturnal evaporation aids rapid daytime temperature increases and contribute to an increased number of heat waves, leading to a positive feedback wherein evaporation is increased leading to drier soil (assuming little to no replenishment) which perpetuates the warm cycle [61] .

Maximum temperature is less skillfully predicted, showing higher bias in both sub-regions. The RCM3-GFDL and ECP2-GFDL models, in particular, show strong cold bias, especially as measured by RMSE and MAE (Figures 7-9). The low skill values are attributed to a cold bias in the GFDL GCM (Figure 10) from winter through mid spring between 3˚C and 7˚C. Although two models with the same LBCs show relatively little skill

in replicating daily maximum temperature and have high RMSE and MAE values, the fact the RCM3-CGCM3 illustrates relatively little skill points to a systematic error within the RCM3 itself (Figure 11), and is not simply an issue of the GCM providing biased LBCs. Conversely, the CRCM-CCSM suffers a reduction in skill from July through September, a direct result of the model exhibiting a strong warm bias from the 50^th percentile to the 99^th percentile and a lesser warm bias in the 1^st through 50^th percentiles. Part of the warm bias can be attributed to the CCSM GCM which shows a warm bias from the 50^th through the 99^th percentiles and a slight warm bias of less than 2˚C from the 1^st through 50^th percentiles while the other part is attributed to the strong warm bias observed with the NCEP-driven runs of the CRCM observed over the same period.

Percentile plots (Figure 10) highlight the significant cold bias found within the GFDL-based NARCCAP ensemble members, leading to low skill scores and high RMSE and MAE values. Most months exhibit a cold bias on the order of 4˚C with values as low as 9˚C and 10˚C in winter. Four of the nine NARCCAP members have a cold bias greater than 4˚C during the months encompassing boreal winter, spring, and mid to late fall. The cold bias observed with the GCM-driven runs is partially attributed to the persistent cold bias observed in most months (and percentiles) for each RCM driven with NCEP LBCs. Percentile plots also reveal the CRCM-CCSM suffers from a warm bias, most pronounced from June through September with the largest warm bias occurring from the 50^th to 99^th percentiles.

An interesting comparison between the GFDL-driven (RCM-based runs) and the GFDL-timeslice reveals the GFDL-timeslice typically outperforms the RCM-driven GFDL runs, with higher skill-based values and lower error/bias values. This demonstrates how the use of observed sea-surface conditions in the timeslice experiment results in more accurate projections than using GCM-based sea-surface conditions. Additionally, it should be

noted that the separation of skill and bias metrics of the GFDL-driven models are not significantly different with respect to minimum temperature but are (in some instances) with respect to maximum temperature. This may suggest that the inclusion of observed sea-surface conditions in a regional climate model impacts maximum temperatures more than minimum temperatures. Solman et al. (2008) [62] noted in their RCM-timeslice experiment that maximum temperatures are better represented than minimum temperatures when compared to observations.

The most consistent and least bias model in both sub-regions is the MM5I-CCSM with Perkins skill scores and Willmott values exceeding 0.75, and low RMSE and MAE values between 0.5˚C and 2˚C. Percentile plots concur with the MM5I-CCSM’s skillfulness by showing the model rarely deviates beyond ±2˚C from observations across all percentiles (Figure 9). The only months in which the MM5I-CCSM observes a small deviation in skill are December, August, and November when the model exhibits a consistent bias throughout all percentiles (slight cold bias in December and November; slight warm bias in August). The MM5I-CCSM is the only model (with respect to temperature) to attain Perkins skill scores of greater than 0.9 for at least eight months, Willmott values of 0.85 for at least eight months, and RMSE and MAE values less than 1.5˚C for 10 months.

Skill associated with NARCCAP output of mean precipitation in the Southeast varies by metric. No discernible relationship between Perkins/Willmott scores and RMSE/MAE for precipitation was found, indicating these metrics quantitatively measure non-Gaussian distributions much differently, with outliers having a much larger impact in the calculation of RMSE and MAE than in Perkins or Willmott’s methods. Very rarely is a models’ Perkins or Willmott skill score below 0.8, indicating the models are able to adequately reproduce the daily data distribution found in observations, especially considering most data points are contained within the left tail of the distribution, with a very small number comprising the right tail of the distribution. However, a high RMSE and MAE, coupled with high Perkins skill score, indicates the model either under- or over-predicts the quantity and/or frequency of precipitation events in the upper percentiles (from 75^th through 99^th percentile). Although the upper percentiles account for an extremely small portion of the data distribution (while still having a large impact on daily total rainfall), their impact on RMSE and MAE is significant because these values are large outliers from the mean. With few exceptions, Perkins skill scores for most models exceed 0.8, with several in the 0.85 to 0.95 range for both sub-regions (Figure 12 and Figure 13). Additionally, Willmott index of agreement values mostly fall within the 0.82 to 0.96 range.

The only models to exhibit degradation in Perkins and Willmott skill scores in the east sub-region are the CRCM-CGCM3 and CRCM-CCSM in September and October, with Perkins skill scores between 0.7 and 0.8 (save for CRCM-CGCM3’s September Perkins skill score between 0.8 and 0.85) and Willmott values between 0.75 and 0.85. Percentile plots (Figure 14) reveal the CRCM RCMs exhibit a dry bias across all percentiles, but is most pronounced from the 50^th through 99^th percentiles and ranges between 30% and 60% below observations. A similar dry bias is observed from the winter through mid spring and fall for the NCEP-driven runs of the CRCM (Figure 15). In terms of real numbers, assume the 75^th percentile value from observations was 100 mm∙day⁻¹, a 30% to 60% dry bias would mean the model predicts the 75^th percentile value to fall between 40 and 70 mm∙day⁻¹. RMSE and MAE values are high because of the magnitude of model bias, and because of the range of values across which model exhibit bias.

The west sub-region performs slightly worse than the east sub-region with regards to Perkins and Willmott scores, moreover, RMSE and MAE values are higher for most models, particularly in the winter and spring months. Percentile plots reveal, from December through May, almost all models have a dry bias in the lower 50^th percentiles between 5% and 30%. Although Perkins and Willmott scores are respectable during this period (between 0.8 and 0.9 for both, respectively), indicating the models are able to replicate the daily precipitation pattern, they fail to generate lighter precipitation found in the bottom half of the distribution.

GCM-based percentile plots (Figure 15) illustrate all three models stay within a bias of ±15% below the 50^th percentile, steadily increasing to a dry bias between 20% and 65% at the 99^th percentile. Most RCMs, except those run by the CCSM GCM, show less dry bias at the higher percentiles. However, some models overcompensate and what was once a dry bias within the GCM becomes a wet bias of the same magnitude in the RCM. This finding is attributed to the wet bias observed consistently in the ECP2, MM5I, and WRFG NCEP-driven models (Figure 16) for all months. Additionally, the RCM3-NCEP model illustrates a pronounced wet bias in the highest percentiles from spring through mid summer.

More than half of the models have a wet bias in the 50^th through 99^th percentiles and is a function of a wet bias in the same percentiles for each RCM driven by NCEP boundary conditions. These precipitation amounts do not adequately replenish water supplies and sub-surface moisture due to high runoff rates. Even more troubling is the dry bias observed from most models during the summer across all percentiles. The dry bias within the GCMs noted for the east sub-region is slightly greater in the west sub-region which is maintained, and in some cases enhanced, within the RCMs. The dry bias must be attributed to the GCMs because each RCM run with NCEP LBCs illustrates a wet bias during the summer, with the exception of the WRFG from the 5^th through 50^th percentiles and the CRCM from the 90^th to 99^th percentiles. Although observations show extreme precipitation (both high and low) is increasing across the Northern Hemisphere [63] and U.S. [64] , the models exhibit a propensity toward high-end exaggeration.