Statistics of different public forecast products of temperature and precipitation in Estonia

Day (0600–1800 UTC) maximum and night (1800–0600 UTC) minimum temperature forecasts as well as prediction of the occurrence of precipitation are evaluated for different sites in Estonia: southern coast of the Gulf of Finland (Tallinn), WestEstonian archipelago (Kuressaare), and inland Estonia (Tartu). The forecasts are collected from Estonian weather service. Several traditional verification methods are used, first of all reliability (root mean square error (RMSE)) and validity (mean error (ME)). Detailed analysis is carried out by means of the contingency tables that enable the user to calculate percent correct, percent underestimated, and percent overestimated. The contingency tables enable the user to calculate conditional probabilities of the realizations of certain forecasts. The paper is user-oriented and does not analyse the forecast technique. On the other hand, attention is drawn to the subjectivity of such evaluation, as the results may depend on the forecast presentation style and/or on the choice of the features of the meteorological parameter under consideration. For the current case study (the coldest hour during night and the warmest hour during day chosen to validate the temperature forecast, the temperature validation bin size 3 degrees, precipitation forecast validated in three categories based on the 12 h precipitation sums) one may say that the RMSE of the shortterm prediction of night minimum and day maximum temperature is 1.5 °C...3.1 °C. It was also noticed that Estonian weather service predicts lower night minimum temperature than it follows in reality. The skill of the temperature forecast is estimated by comparison of its RMSE with that of the persistence forecast (next night/day will be similar to the previous one). The RMSE of the 1st day/night forecast is by 1.3 °C...1.4 °C less for Tallinn and Tartu and 0.3 °C...0.8 °C for Kuressaare than that of the persistence forecast. For the precipitation forecast, percent correct is 60...70, the probability that dry weather forecast is followed by no precipitation is 70%...80%. At the end of the paper the long-term forecasts of two international weather portals www.gismeteo.ru (Russia) and www.weather.com (USA) are briefly analysed.


*
Weather forecast is based on the predictability of atmospheric processes that is tightly related to the chaos theory.It is widely known that there are fundamental limits on the deterministic predictability of the atmosphere, as the development of the processes is sensitive to the variations in the initial conditions.Additional errors to the forecasts may be introduced by the schemes of numerical weather prediction due to the approximate simulation of atmospheric processes.Therefore, estimation of the forecast quality is necessary.This can be carried out by means of theoretical considerations or by comparison of the given forecasts with the values of the meteorological parameters that were really measured.Due to the large dimensionality, the task is not trivial and, as a rule, the predictability of different components is of different quality.Even more, the results depend substantially on the details (range, bin size, intensity, occurrence, etc.) of the meteorological parameters under consideration.
The simplest way to get information on the forecast quality is calculating root mean square error comparing the predicted values with the measured ones.We speak about a good forecast when the RMSE is small and about a bad forecast when it is large.The other simple characteristic of the sample is mean error or bias.On the other hand, such knowledge is of low value when we want to estimate risk that needs two components: probability and cost.To evaluate the forecast of dichotomous weather events, several economically sensible characteristics are offered [1][2][3].These estimates are based on the contingency table that permits the user to calculate the accuracy in the form of percent correct (PC): the ratio of hits plus correct negatives to the total number of forecasts.A similar score can be used also when the forecast is given for a range of meteorological parameters, e.g., temperature, precipitation sum, wind speed, etc.In this case PC is calculated as the ratio of the number of correct predictions to the total number of predictions.In a similar way also the percentage of under-and overestimations can be calculated that is somewhat better indicator than bias.
Heideman et al. [4] confirm that although numerous methods to estimate skill and accuracy of forecasts are available, there exists no single measure to evaluate the forecast quality, as different users have different interests.This is especially evident for precipitation forecasts as some users need quantitative precipitation data [5] while others are interested in the rainfall threshold and spatial scale [6].Special procedures should be used to provide profit-oriented enterprises with the necessary information to make proper decisions [7,8].On the other hand, economic benefits from improved forecast accuracy should be estimated to justify the investments into the weather service [9].
Stewart et al. [10] have claimed that even the most accurate forecast is of little value if it is not well used.In most cases the decisions must be made in the conditions of imperfect weather forecasts and the user must get maximal profit or minimal loss from its skilful application.One method to help the decision-makers to account the cost-loss balance is calculating conditional probabilities from the contingency tables: what is the probability that the predicted situation takes place?Together with an economic value statistics such conditional probabilities might help to make the best decision [2,11].
Private users are mostly interested in the precipitation and temperature forecasts while the most important aspects are when and where precipitation will occur [12].On the other hand, people do not always understand the language of weather forecasts adequately and effective communication of the weather forecast uncertainty needs special skills [13].
The present paper is written from the viewpoint of the user of the temperature and precipitation forecasts, uniting therefore assessment of the prediction skills with presentation of the forecast.The outputs of different forecast systems are taken as input material and the forecast technique is not analysed.The aim of the paper is to give a meteorological background for the decisionmakers who can use such output statistics to a further analysis of their specific economic or social problems.This is especially important at the risk calculations in conditions of uncertain forecast.

MATERIAL AND METHODS
Weather forecasting at the Estonian weather service is based on the HIRLAM and ECMWF systems but the products of the numerical weather prediction (NWP) models are complemented with the analysis of surface and higher atmosphere weather maps.Therefore, the human experience is considered to be important [14,15].The forecaster determines the direction of the main air flow and pressure tendencies.Air moisture content is decisive by the forecast of cloudiness and precipitation.The isotherms on the 850 hPa level and the height of the air layer are important by the forecast of the winter precipitation phase.The analysis of higher level maps gives a possibility to estimate the development of the pressure systems and their motion.Additional information is drawn from the forecast charts of other countries that are available in the Internet.
The forecasters send material to the telecommunication department where forecast maps, provided with icons, are prepared for the public use.In the present study, the files of the forecasters are used.
Estonian weather service gives official weather forecast every day at 1200 UTC.Night minimum and day maximum temperature is predicted and archived for three following nights and days.For these forecasts, night means 1800-0600 UTC (2000-0800 East European winter time) and day means 0600-1800 UTC (0800-2000 East European winter time).One should keep in mind that in this context night minimum does not necessarily mean daily minimum, as sometimes the lowest temperature is measured early morning after 8 a.m.The same should be said about day maximum that is not necessarily the highest daily temperature.
Forecasts for three Estonian sites were used in the present study (Fig. 1): Tallinn on the northern coast of Estonia, Tartu as an inland site, and Kuressaare in the West-Estonian archipelago.
To estimate the quality of the local forecasts, a threeyear period of 2009-2011 was used.
For the values of the measured night minima (day maxima) the average temperature of the coldest (warmest) hour was taken.These data were drawn from the archives of Estonian weather service.Tallinn is represented by Tallinn-Harku meteorological station that is situated some kilometres to the west of the city.For Tartu, measurements at Tartu-Tõravere meteorological station were used.Tõravere is located about 25 km to the southwest of the city.Kuressaare is represented by measurements at the Roomassaare coastal hydrological station that is situated near the port of Kuressaare.The temperature measurements were carried out by means of the automatic weather stations.
The RMSE and mean error (ME) were calculated for the temperature forecasts with the lead times of next night and day, the second night and day, and the third night and day.These estimates were compared with the persistence forecast that predicts next night minimum (day maximum) as that of the previous one.From the forecasts and measurements, contingency matrices were calculated for predicted and realized temperatures.From these matrices percent correct as well as the percentage of over-and underestimation of the day maximum and night minimum temperature were calculated.The profitoriented customers can easily use these matrices to calculate necessary conditional probabilities.
Precipitation, cloudiness, and atmospheric phenomena forecast of the Estonian weather service is given in the form of icons that refer to 27 possibilities (www.emhi.ee).The icons are related to the 12 h precipitation sums as shown in the first two columns of Table 1.To verify the forecast, the precipitation sums measured at Tallinn-Harku, Tartu-Tõravere, and Roomassaare stations during the respective night and day were used.To validate Tallinn and Tartu forecasts, the manual measurements were used.At Roomassaare the manual measurements were not available.Therefore the data from automatic weather station were used to validate Kuressaare forecast.The evaluation was carried out for three gradations: dry, light precipitation, and moderate or heavy precipitation.The selection of such bins is justified by the characteristics of the icons used in the forecast.From the numerous popular weather portals, www.gismeteo.ru(Russia) and www.weather.com(USA) were chosen to estimate the long-term forecasts they offer for a wider public.Temperature and precipitation forecasts for Tartu and Tallinn were read every third evening with the lead times of 7 and 10 days (www.weather.com)and 7, 10, and 14 days (www.gismeteo.ru).The data were collected during one year from 11 September 2010 to 10 September 2011.This data base is too small to get stable statistical estimates.Therefore these results should be regarded as tentative ones.

ACCURACY OF THE TEMPERATURE FORECASTS
The RMSE and ME values for the local temperature forecasts, calculated from the comparisons of a three year period (2009)(2010)(2011), show an increase with the increase of the lead time (Tables 2 and 3) -a result that could be expected.The RMSE of the persistence forecast that predicts next night minimum or day maximum as that of the previous one is the largest in all cases, exceeding even the RMSE of the 3rd night/day forecasts.The difference between the Estonian weather service next night and day forecast and the persistence forecast is the smallest at Kuressaare -a nice demonstration of the stabilizing role of the sea in weather processes.It can also be seen that the accuracy of the forecast for the day maximum is better than that of the night minimum.This result could be expected, as the variance of the air temperature in winter is larger than in summer.The ME of the forecast of night minimum indicates systematic underestimation (predicting colder).Day maximum is predicted with nearly negligible bias.Figure 2 demonstrates scatter plots for two cases.The best forecast was given for the 1st day maximum temperature in Tallinn and the worst for the 3rd night minimum temperature in Tartu.For comparison, two cases of the persistence forecast are shown in Fig. 3 the best (day maximum at Kuressaare) and the worst (night minimum at Tartu).Both forecasts use the idea that next night (day) will be similar to the previous night (day).

UNDER-AND OVERESTIMATION OF THE TEMPERATURE
RMSE, ME, and scatter plots characterize only the general features of the forecast quality for the whole range of possible temperatures that is from -35 °C to + 35 °C in the present data series.The user needs specific and more detailed information that can be given by means of contingency tables.Next the values of percent correct (PC), percent underestimated (PU) and percent overestimated (PO) are analysed for two temperature ranges: • forecast for the night minimum -11 °C and lower • forecast for the day maximum 22 °C and higher.These boundaries were chosen to describe the quality of predictions in frost and heat conditions.If -11 °C was chosen at the authors' discretion, then forecast of 22 °C can be followed by day maxima up to 25 °C that might cause health problems in the marine climate of Estonia.
Traditionally, the temperature forecast is considered to be successful when the deviation of the measured temperature is ± 1 °C.This was taken into account at the calculation of the described characteristics.PC means that the real measurements fall into the boundaries of the forecasted temperature ± 1 °C, e.g., in case the next night minimum forecast for Tallinn was -12 °C, the forecast was labelled 'correct' when the real temperature was -13 °C…-11 °C.PU shows the percentage of the cases when the forecast showed lower temperature than the measured temperature minus 1 °C, PO shows the percentage when the forecast showed higher temperature than the measured temperature plus 1 °C.
Table 4 shows that in frost conditions Estonian weather forecast strongly underestimates coming night minimum, showing the temperature lower than it will be measured.That is somewhat justified as warnings.
Table 5 shows that coming day maxima in summer are forecast better than night minima in winter.The highest PC of the next day maximum (71%) is shown in Tartu, but it is slightly less than 70% also at the other sites.On the other hand, day maxima are overestimated for longer lead times that can also be justified as preliminary warnings, but corrected later for shorter lead times.Underestimation of the day maxima can be noticed only for Tallinn, in other sites the percentage of such forecasts is less than 10.

CONDITIONAL PROBABILITIES FROM CONTINGENCY TABLES
Table 6 shows the results of the evaluation of local temperature forecast for the next night in detail around the critical freezing temperature.The range of the temperature forecast was chosen to cover all possibilities when the temperature during 3 years was 0 °C or lower.Such information in the form of contingency tables could be used for calculations of conditional probabilities of realized situations for fixed forecasts, e.g., in case the temperature 0 °C of night minimum is predicted in Tallinn, the probability is 50% that it will be warmer: 12 + 6 + 5 + 2 cases of total 51 cases is 50%.Knowing this probability, the user can decide whether to take expensive measures against freezing (roads, vegetable, fruit trees, etc.) or is it cheaper to suffer losses.A similar calculation shows that for the forecast of + 1 °C in Tallinn, there is still probability of 20% (4 + 6 cases of 51) that the temperature will be 0 °C or lower.In case a similar forecast is given for Kuressaare, the probability of freezing is 40% (3 + 18 cases of 52).Table 6 demonstrates also that the result of the evaluation of a forecast depends on the choice of the evaluation conditions.In case the forecast is considered to be correct in the boundaries of the measured temperature ± 1.5 °C, we get PC of 73% for the forecast -2 °C in Tallinn.Requiring the realization of the predicted temperature exactly, PC is only 25%.
It can also be seen that the values of PC are in this temperature range much higher than in the range of low temperatures, although the nights are still predicted somewhat colder than they really are.

ASSESSMENT OF THE PRECIPITATION FORECASTS
Table 7 presents part of the results of the examination of local precipitation forecasts.For validation, the icons are divided into three groups: no precipitation (labelled as 'Dry'), 0…(2)3 mm during 12 h ('Light'), and more (see Table 1).Table 7 enables one to estimate the accuracy of the forecast as PC (the number of correct forecasts divided by the number of all forecasts) that is 74% for next night and 63% for next day.The accuracy is lower for the sites that are situated on the coast (not shown in this paper).On the other hand, the difference between the accuracies of the night and day forecasts is the largest at the inland station of Tartu and the lowest at the test site of the West-Estonian archipelago.

LONG-TERM FORECASTS OF THE INTERNATIONAL WEATHER PORTALS
The time period, during which the data from the weather portals were collected, is short and the data set contains about 120 observations.Nevertheless, it could be interesting to calculate from these data the same statistical parameters that were calculated from the Estonian weather service data.
The RMSE for the temperature forecasts drawn from the weather portals during the period 11 September 2010 -10 September 2011 is shown in Table 8.It should be mentioned that these two weather portals present the temperature forecast differently.The www.weather.comcharacterizes the temperature by "low" and "high".We decided that "low" belongs to the night and "high" to the day.The www.gismeteo.ruoffers four values for temperature: night, morning, day, and evening.The data read from the weather portals were compared with the minimal and maximal temperature of the date under consideration.
It can be seen that for the lead time 7 nights/days the RMSE of www.weather.com is somewhat smaller than that of the www.gismeteo.com,but in case the lead time is 10 days, both show RMSE around 4 °C.The international weather portals describe precipitation differently.The www.weather.comgives chance of precipitation in per cent, but www.gismeteo.ruuses words like 'light rain', 'heavy snow', etc. Therefore it was not possible to compare the forecasts in detail and only the accuracy of the forecast 'no precipitation' was estimated (Table 9).This is calculated as the percentage of dry situations that followed the forecast 'dry'.In Table 9, similar percentage of the local forecast 'no precipitation', is shown for shorter lead times.It can be seen that www.gismeteo.rupredicts dry weather in Tallinn somewhat better than www.weather.com,but local short-term forecasts show no better quality than international long-term forecasts.

DISCUSSION
The paper offers some possibilities to estimate the forecasts of temperature and precipitation from the viewpoint of the user.The case study is based on three sites in Estonia, but the results have a wider context, as the Estonian local weather service acts according to the WMO rules and international weather portals pretend to forecast the weather all over the world.Traditionally, the lead time of the forecasts of the local weather services is up to 5 days.The weather portals offer predictions for the next 10 to 15 days.Therefore, such a comparison permits one to assess the differences in the quality of the forecasts at least in the regions where the variation of the weather systems is similar to Estonia.
To satisfy the users, decision-makers should be provided with the details of the forecast and its evaluation.Both components are necessary to understand the forecast properly.The members of the public who interpret "a 30% chance of rain tomorrow" in many different ways [16] should be educated, more competent users, whose activities depend on the weather, should be provided with unambiguous information on the forecast accuracy.
There are many possibilities to evaluate the weather forecast and weather services offer a wide variety of numbers that should describe the accuracy of the prediction of some meteorological parameter or the whole forecast text.Such evaluations are of low value, unless it is not described what is compared with what and how the estimates were calculated.
It is rather widely known that the results of weather forecast evaluation depend not only on the available meteorological information, model quality or forecaster skills, but also on the forecast presentation technique, choice of a specific feature of the meteorological parameter under consideration and dividing the continuous range of it into bins.In the present paper the measured coldest hour during night and the warmest hour during day were chosen to validate the temperature forecast.Somewhat different results could be obtained when absolute minima and maxima or night and day averages or midnight and noon data were chosen.The temperature validation bin size in the present paper was 3 degrees.We get worse percentage of the correct forecast when we reduce the bin size and better results when we increase it.Sometimes the accuracy estimates are extremely sensitive to the chosen input data.In case we calculate the value of fraction correct of the precipitation forecast validating it against manual measurements, we get somewhat different estimate than validating it against automatic weather station measurements.Therefore, a profit-oriented person, who must use the weather forecast, must first choose the appropriate meteorological parameter that might affect his/her activities.Next he/she must decide which material to use for validation of the forecast -to choose the appropriate meteorological element, the range of its values, allowed tolerance in time and space, etc.In this case the contingency matrices act as a tool of the risk estimation.

Fig. 2 .Fig. 3 .
Fig. 2. Forecast and realization of the 1st day maximum temperature in Tallinn and the 3rd night minimum temperature in Tartu.

Table 1 .
The names of the icons, the intensity of precipitation, and the gradation for validation

Table 2 .
The reliability and validity (in degrees) of the night minimum temperature forecast of Estonian weather service for different lead times and the RMSE of the persistence forecast

Table 3 .
The reliability and validity (in degrees) of the day maximum temperature forecast of Estonian weather service for different lead times and the RMSE of the persistence forecast

Table 4 .
PC, PU, and PO for the Estonian weather service forecasts for night minimum when the forecast was -11 °C and lower

Table 5 .
PC, PU, and PO for the Estonian weather service forecasts for day maximum when the forecast was 22 °C and higher

Table 6 .
Number of cases that followed the predicted next night minimum temperature.Forecast is in the first column, measurements in the first row

Table 7 .
Number of cases that followed the predicted precipitation for the next night and day in Tartu.Forecast is in the first column, measurements in the first row

Table 8 .
The reliability (RMSE in degrees) of the night minimum and day maximum temperature forecasts of the weather portals for different lead times

Table 9 .
The percentage of the dry weather forecast that was followed by no precipitation from international weather portals and local weather service