We develop a graphical interpretation of ternary probabilistic forecasts in which forecasts and observations are regarded as points inside a triangle. Within the triangle, we define a continuous colour palette in which hue and colour saturation are defined with reference to the observed climatology. In contrast to current methods, forecast maps created with this colour scheme convey all of the information present in each ternary forecast. The geometrical interpretation is then extended to verification under quadratic scoring rules (of which the Brier score and the ranked probability score are well-known examples). Each scoring rule defines an associated triangle in which the square roots of the score, the reliability, the uncertainty and the resolution all have natural interpretations as root mean square distances. This leads to our proposal for a ternary reliability diagram in which data relating to verification and calibration can be summarized. We illustrate these ideas with data relating to seasonal forecasting of precipitation in South America, including an example of nonlinear forecast calibration. Codes implementing these ideas have been produced using the statistical software package R and are available from the authors.
Forecasts are often given in probabilistic terms. For example: ‘There is a 60 per cent chance that rainfall before 12.00 tomorrow will exceed 15 mm’; ‘The odds for the football match are 6-to-1 against a home win and even money for a draw’; ‘The climate model predicts that mean daytime temperature in summer 2100 will be normally distributed with mean μ and variance σ2’. In each of these examples, the forecast consists of a probability distribution. The sample spaces for the observable events of interest consist of two elements (exceed/not exceed), three elements (home win/draw/away win) and the real line . Consequently, these examples relate to binary, ternary and continuous probabilistic forecasts, respectively.
This paper is concerned with two types of visualization. In physical space, visualization constitutes colour maps of probabilistic forecasts over a geographical region. In probability space, visualization will constitute a geometrical representation of forecasts and observations in our proposed ternary reliability diagram.
For binary forecasts, visualization tools such as the reliability diagram, the sharpness diagram and the relative operating characteristic (ROC) curve are well known . Our aim in this paper is to develop analogues of these ideas for ternary forecasts and to develop a geometrical intuition for verification and recalibration.
The structure of this paper is as follows. Section 2 contains a discussion of probabilistic forecasting, and illustrates how a continuous forecast distribution can be identified with a ternary forecast. Section 3 then introduces the idea of representing each ternary forecast as a point in barycentric coordinates, and hence assigning to it a unique colour. In §4, the quality of a forecasting system is quantified using quadratic scoring rules. This is then used in §5 to propose an algorithm for recalibrating probabilistic forecasts which is illustrated with seasonal forecasting data. Conclusions are presented in §6.
2. Probabilistic forecasting
The generic situation to be considered is as follows: a forecasting system (perhaps a suite of climate models with different initial conditions) can be used to produce probabilistic forecasts (that is, probability distributions) for some variable of interest at a number of points within a geographical area. The subsequent challenge is to display in map form as much as possible of the information contained in this spatial array of probability distributions. The most natural thing to do, perhaps, is to associate with each forecast distribution a set of scalars (such as the distribution's moments or a selection of its quantiles) and produce a separate map for each scalar. The disadvantage of this procedure is that multiple maps are needed to convey the information contained in the forecast distributions. If information on the skill of the forecasting system were available, yet another map would be required.
Consider probabilistic forecasting of a scalar climate variable x, such as temperature or precipitation. The forecasting system produces a cumulative distribution function (CDF) F(x) at each spatial location (figure 1). The CDF expresses the forecast probability that the variable of interest will not exceed x. The forecast distribution F(x) can be interpreted as the forecasting system's state of knowledge about the future value of the variable x. In contrast to a deterministic forecast xf (which is a scalar), a probabilistic forecast F(x) is (in principle at least) an analytic function and hence an infinite-dimensional object. For this reason, it is not immediately obvious how to visualize spatial probabilistic forecasts.
Often, the variable x has associated with it an observed climatology represented by the CDF G(x) (figure 1). The climatology can be interpreted as the historical distribution of the observable quantity. Typically, a climatology is calculated by aggregating data from an observational record over at least the past 30 years.
It is important to note that both the climatology G(x) and the forecast F(x) are continuous probability distributions and so it is quite possible for the forecasting system to issue the climatology G(x) as its forecast. This is a perfectly valid probabilistic forecast but not, perhaps, a very useful one. It is usually hoped that the forecasting system's state of knowledge about the future extends beyond knowledge of the climatology alone. For this reason, it will be helpful to view the climatology as a benchmark distribution with which all other forecasts should be compared. Thus, in the discussion below, a colour will be assigned to the forecast F(x) by considering the ‘distance’ between the forecast F(x) and the climatology G(x).
(a) Ternary probabilistic forecasts
In order to reduce the dimensionality of the problem, it is useful to project the continuous distributions G(x) and F(x) onto ternary distributions q′=(qB,qN,qA) and p′=(pB,pN,pA) (figure 1). We adopt the convention that vectors are column vectors and that ′ denotes a transpose. The real line is divided into three ordered categories , N=[xB,xA] and whose labels have the following rationale: B, ‘below-normal’; N, ‘near-normal’; and A, ‘above-normal’.
The ternary distributions q(G) and p(F) encode the probabilities with which the variable is forecast to lie in the categories B, N and A (figure 1), and so all of their elements are non-negative with qB+qN+qA=pB+pN+pA=1.
For given quantiles xB and xA, it follows that the ternary climatology q and ternary forecast p are given by 2.1
An equivalent interpretation is that for a specified ternary climatology q the categories B, N and A are defined by the quantiles 2.2A natural choice for the ternary climatology is the uniform distribution . In this case—which we shall regard as the default—the categories B, N and A are defined by the terciles of the climatology G(x) and observations over a long period would be expected to lie with equal frequency in each of the three categories. It should be stressed, however, that any choice of quantiles can be made when defining the ternary climatology. For example, the choice would imply that categories B and A had been chosen to be the lower and upper quartiles of the climatology.
3. Barycentric coordinates
In this section, a geometrical representation of a ternary forecast p, ternary climatology q and ternary observation o is considered.
(a) Scoring functions
Scoring functions quantify the past skill of a forecasting system. The idea is to compare a ternary forecast p with the corresponding ternary observation o made after the event. Since the categories are exclusive and complete, it follows that a ternary observation o can take one of three values oB′=(1,0,0), oN′=(0,1,0) and oA′=(0,0,1) according to whether the observable is found to lie in category B, N or A.
The skill of a forecasting system can be quantified for a set of forecast–observation pairs by a score function S. The score is a measure of the difference between forecasts and observations. It follows that the lower the score, the more skilful the forecast.
In this paper, attention is restricted to scores defined by quadratic forms 3.1where the 3-by-3 matrix L defines the particular scoring rule being used (appendix Aa), L′L is assumed to be positive definite and the overbar denotes an average over all forecast–observation pairs. The Brier score (appendix Aa) and the ranked probability score (appendix Ab) are well-known examples of quadratic scores [2–6].
The scoring matrix L can be used to define a 2-by-3 matrix (equation (A4)) which maps a ternary forecast to a corresponding point in the plane. This transformation maps ternary forecasts to points within a triangle and so each scoring matrix L induces a corresponding triangle in . In this paper, the matrix is defined so that a score S corresponds to a (mean) squared distance between forecasts P and observations O (considered as points inside an appropriate triangle): 3.2
The corners of this triangle are the points , and associated with the three possible values of a ternary observation. The point associated with the climatology lies within the triangle. The default choice for the scoring matrix is that associated with the Brier score, in which case the associated triangle is an equilateral triangle with unit sides. In the case of the default climatology , the point Q lies at the centre of this triangle (figure 2a). This sort of visualization receives different names in different disciplines. In mathematics, it is known as a plot in barycentric coordinates while in applied sciences it is known as a ternary phase diagram.
Barycentric coordinates yield an intuitive geometrical interpretation of ternary forecasts. As well as allowing scores to be visualized as squared distances they allow any scheme for colour assignment to be visualized as a triangular colour palette. For example, figure 2b illustrates the algorithm which assigns colours according to which of the three possible outcomes is considered in the forecast to be the most probable. Here and subsequently, it is assumed for illustration that the forecast variable x is precipitation. It therefore seems sensible to assign the ‘dry’ colour red to category B and the ‘wet’ colour blue to category A. Note that both the ternary forecasts p′=(1,0,0) and p′=(0.34,0.33,0.33) would be assigned the colour red in figure 2 even though they differ greatly. In the former case, the forecasting system is certain that x will lie in category B while in the latter case, the forecast is barely different from the climatology .
Since scores are squares of distances within the triangle, it is helpful to consider the root-score . Figure 2c shows contours of constant root-score in the case of the Brier score, when the observation o takes each of its three possible values. These contours show the set of forecasts which are equally good (in terms of score) as predictions of the subsequently observed value. Figure 2d shows contours of constant root-score in the case of the ranked probability score, for which the induced triangle is a right-angled triangle with sides , and 1 (appendix Ab).
(b) Current colour schemes for ternary forecasts
Current methods for assigning colours to ternary forecasts discretize barycentric coordinates into a finite number of regions, meaning that information present in the forecast is not present in the assigned colour . For example, consider the ‘most likely tercile’ scheme of figure 2b used for displaying precipitation forecasts produced in the EURO-BRazilian Initiative for improving South American seasonal forecasts (EUROBRISA) project.1 Precipitation forecasts produced in EUROBRISA are Gaussian distributions obtained from a calibration and combination procedure known as forecast assimilation . These ‘integrated’ forecasts are a combination of four coupled ocean–atmosphere general circulation models (European Centre for Medium Range Weather Forecasting (ECMWF) , UK Met Office , Météo France  and Centro de Previsão de Tempo e Estudos Climáticos (CPTEC) ) and an empirical model that uses Pacific and Atlantic sea surface temperatures as predictors for South American precipitation .
A feature of current visualization methods is that the algorithms by which colours are assigned tend to be described algebraically. In the example considered here, colours are assigned to forecasts based on the following algebraic definitions of five regions of forecast space:
(dry): ( and and ).
(dry or normal): ( and ) or ( and ).
(normal): ( and and ).
(wet or normal): ( and ) or ( and ).
(wet): ( and and ).
The use of barycentric coordinates allows the meaning of these definitions to be visualized easily (figure 3). It also reveals something that is not immediately obvious from the algebra. There is a region in barycentric coordinates (here coloured grey) that is not included in these definitions. This region of ternary forecast space—at the base of the triangle—corresponds to ternary forecasts in which category N is assigned a low probability, but the outlying categories B and A are assigned relatively high probability. Such forecasts can arise when the forecast distribution F(x) has variance much greater than that of the climatology G(x) (§3e).
In summary, many current methods for assigning colours to forecasts lose information by discretizing ternary forecast space. The same colour is assigned to more than one ternary forecast and the colours do not convey a sense of how much the forecast differs from climatology. A ternary forecast close to climatology provides little gain in information—the forecast has not told us much that we did not already know. On the other hand, a ternary forecast far from climatology provides a large gain in information. In such a case, the forecasting system assigns high probability to the variable lying in one particular category and not the others.
The aim now, therefore, is to assign a continuum of colours to ternary forecasts, viewed as points in barycentric coordinates. This will retain all information present in the ternary forecast and so it will be possible to identify a ternary forecast uniquely from its assigned colour. The choice of colour assignment will, in particular, take account of the information gain in each ternary forecast.
(c) Comparing forecasts with the climatology
Probabilistic forecasts are regarded here as measures of belief and so a measure based on entropy  is preferred. A sensible measure of information gain E(p;q) is 3.3This can be interpreted as a scaled version of the Kullback–Leibler divergence between p and q. Contours of constant E(p;q) are plotted in the unit triangle in figure 4. Note that the climatology q corresponds to an information gain of 0 (E(q;q)=0) and that the corner furthest from the climatology corresponds to an information gain of 1 (E(o;q)=1 for some o).
Motivated by the idea of assigning colours according to the most likely tercile, the continuous angular measure θ(p;q) will be referred to as the dominant category. This measures the angle in barycentric coordinates of the forecast p with respect to an origin at the climatology q. It follows that the information gain E(p;q) and the dominant category θ(p;q) define an alternative coordinate system for ternary forecast space (figure 2).
(d) A new colour scheme for ternary forecasts
In the red–green–blue (RGB) representation of colour (used in colour televisions), colours are represented by sets of three numbers in the range [0,1] corresponding to the brightness of the three primary colours. Thus, RGB=(1,0,0) is bright red, RGB=(0,1,0) is bright green, and so on. In this paper, colours are assigned using the more intuitive hue–saturation–value (HSV) representation. Geometrically, the RGB system defines a unit cube in colour space in which shades of grey occur on a grey-line running from the black corner RGB=(0,0,0) to the white corner RGB=(1,1,1). The HSV system describes colours as points in a cylindrical coordinate system with this grey line as its axis. In this system, value∈[0,1] is a measure of distance parallel to the grey-line axis, saturation∈[0,1] is a measure of distance perpendicular to the axis and hue ∈[0,1] is an angular measure around this axis.
Here, a colour is assigned to the forecast p by associating the dominant category θ(p;q) with the hue and the information gain E(p;q) with the saturation. The proposed algorithm for colour assignment is 3.4where the functional forms h(θ) and s(E)=Em are illustrated in figure 5. Suggested default choices for the parameters are m=0.7 and θ0=0, which together yield the colour palette shown in figure 6. Note that the climatology is always assigned the colour white by this system. The nonlinear hue function h(θ) has been chosen to minimize the region of barycentric coordinates that is assigned the colour green and hence to minimize difficulty for readers with green-weak colour blindness [15,16]. The parameter θ0 can be chosen in order to rotate the palette in the triangle about the climatology, while the exponent m controls the rate at which colour saturation changes away from the climatology.
An example of a forecast map using the proposed colour scheme is shown in figure 7. In this illustration, the data consist of integrated seasonal precipitation forecasts for South America produced by the EUROBRISA project. At each spatial location, the probabilistic forecast is compared with the terciles of the local climatology. In other words, the ternary climatology (shown by a cross in the colour palette at bottom right) has been used as the benchmark ternary forecast. The map shows clearly that high probability is assigned to low rainfall in northern Brazil (red) while there are regions in southeast Brazil and the Southern Ocean for which the forecast barely differs from climatology (white). There are also regions where rainfall in the near-normal category is forecast with high probability (yellow) and low probability (purple).
(e) Special case: Gaussian distributions
To gain insight into the colours assigned by equation (3.4), it is helpful to consider the special case in which the climatology G(x) and the forecast F(x) are both Gaussian distributions. The space of Gaussian distributions is two-dimensional (each distribution defined by its mean and variance) and so there is a natural one-to-one mapping between a (Gaussian) forecast F(x) and its ternary representation p.
Specifically, suppose that the climatology is and the forecast is N(μ,σ2). It follows that 3.5where Φ(z) is the CDF of the standard normal distribution N(0,1). It is helpful to normalize the forecast mean and standard deviation with respect to those of the climatology 3.6It follows that the forecast is more sharply peaked than the climatology when and less sharply peaked than the climatology when . The ternary forecast p can be calculated from equation (2.1) by noting that 3.7
Figure 8a illustrates the colours that are assigned to a forecast and climatology when both are Gaussian. When the forecast is much more sharply peaked than the climatology () the ternary forecast assigns nearly all of its probability mass to one of the three categories. When the forecast mean is much less than the climatological mean () this probability is assigned to category B and the assigned colour is a strong red. Similarly, a strong yellow is assigned when the means are comparable () and a strong blue is assigned when the forecast mean significantly exceeds the climatological mean ().
The situation is rather different when the forecast has much higher variance than the climatology (). In this case, the ternary forecast assigns little probability mass to the central category N but rather splits it approximately evenly between the two extreme categories. In this case, the assigned colour is purple. This sort of situation might arise in an ensemble climate forecast in which some members of the ensemble predict that the climate variable increases while other members predict that it decreases. The purple colour might also arise if the forecasting system predicted an increase in the frequency of extreme events in both tails of the distribution.
Figure 8b shows and in barycentric coordinates, and hence the Gaussian distributions of climatology and forecast that would yield given a ternary forecast p. Consider the case in which the forecast and the climatology have similar variances, as appears often to be the case. It follows that and so the ternary forecast p lies close to the contour in figure 8b. The forecast mean μ then controls the colour assigned to the forecast, with red assigned when , white assigned when and blue assigned when .
4. Verification of ternary probabilistic forecasts
The previous sections have considered the visualization of a set of probabilistic forecasts. The aim in this section is to visualize the difference between a set of forecasts and the corresponding set of observations. This is known as forecast verification . In the next subsection, the standard score decomposition of Murphy  is reviewed and re-interpreted geometrically.
(a) Decomposition of the score
Suppose that the scoring matrix L has been specified so that the score of an individual forecast–observation pair is given by a squared distance in the appropriate triangle (equation (3.2)). As in previous sections, the Brier score is the default choice but the results hold for any quadratic scoring function.
The triangle representing ternary forecast space can be discretized into a finite number of bins with centres Pk. For simplicity, each forecast P that lies in the bin with centre Pk will be reassigned the central value Pk . Following Murphy , consider the identity 4.1where is the mean observation associated with forecasts in bin Pk. If the climatology is defined to be the mean of all the observations , it follows that a decomposition of the mean score with binned forecasts is 4.2Equation (4.2) constitutes the standard decomposition of the score S into uncertainty U, reliability R and resolution Z . It is important to stress that this decomposition applies only when all forecasts within a bin are assigned the central value .
It has already been remarked that the score is the mean square distance in the triangle between forecasts and observations, and it is clear that the uncertainty is the mean square distance between observations and climatology. In order to see how uncertainty varies with climatology q (figure 9), it is helpful (appendix Ac) to consider the particular climatology q0 (shown by a dot in figure 9) for which the uncertainty gains its maximum value U0=U(q0). The uncertainty U(q) is then 4.3where ΔU(q) represents the reduction in uncertainty between q0 and q. Figure 9 illustrates that the root-uncertainty-reduction can be visualized as the distance in the triangle between the climatology of interest q and the climatology q0 of maximum uncertainty.
Similarly, the resolution in equation (4.2) is the mean square distance between climatology Q and the mean observations conditional on the forecasts . The reliability is the mean square distance between forecasts Pk and the mean observations conditional on the forecasts .
It follows that the root-score is the r.m.s. distance between forecasts and observations, the root-uncertainty is the r.m.s. distance between climatology and observations, the root-resolution is the r.m.s. distance between climatology and mean conditional observations and the root-reliability is the r.m.s. distance between forecasts and mean conditional observations. These geometrical interpretations as r.m.s. distances are illustrated in the triangle in figure 10.
The aim of verification is to quantify the root-score (figure 10, black lines) and the aim of recalibration is to minimize the root-score. Since the forecaster cannot change the observations (figure 10, red circles), recalibration proceeds by changing the forecasts Pk (figure 10, black circles) in order to minimize the root-reliability (figure 10, red lines).
The diagram at the top left of figure 10 contains a proposed graphical interpretation of the score decomposition. The root-uncertainty is independent of the forecasts and defines a semicircle (grey) of diameter (blue). The root-resolution is also independent of the value of the forecasts (provided the ‘binning’ of the forecasts remains unchanged) and defines the larger of the two right-angled triangles. This triangle has sides of length (blue), (green) and (purple). It follows from Pythagoras' theorem that all three corners of this triangle lie on the semicircle, with two on the diameter. The smaller of the two right-angled triangles contains components of the score decomposition that can be altered by recalibration of the forecasts. The smaller triangle has sides of length (black), (red) and (purple) and equation (4.2) constitutes Pythagoras' theorem. The aim of recalibration is to minimize the root-score by minimizing the root-reliability . It follows that the dashed lines in the decomposition diagram represent limiting values of the root-score . A forecasting system that always issued the climatology as its forecast would have resolution Z=0 and hence have a root-score indicated by the blue dashed line. This represents the worst performance that might be expected from a skilful forecasting system. On the other hand, the best possible performance is indicated by the purple dashed line. This would occur for a forecasting system with perfect root-reliability .
(b) Ternary reliability diagrams
The ideas of the preceding subsection can now be illustrated with an example. The data in this example are seasonal precipitation hindcasts at one month lead-time from the EUROBRISA integrated forecast model. The data relate to the years 1981–2005, the season January–February–March and a spatial region 72.5° W–42.5° W, 12.5° S–2.5° N for which seasonal forecasts show reasonable skill.
Figure 11a contains one of our proposed ternary reliability diagrams. The large triangle contains information on the reliability of the forecasts. Bins of size were chosen and the central forecast in each bin Pk—to which all forecasts in that bin are set equal—is plotted in the triangle as a black circle. The mean observation conditional on a forecast is plotted as a red circle and the two are joined by a red line in order to form a ‘dipole’. The length of each dipole represents the root-reliability of the forecasts in each bin. The number of forecasts in each bin is shown graphically by the ternary sharpness diagram at the top right. (Dark colours, high density; light colours, low density; grey, no data.) The climatology is shown by a blue cross. Dipoles for bins containing fewer than 10 forecasts (indicated in the diagram by the text ‘threshold =10’) are omitted for clarity. An advantage of this sort of visualization of the data is that coherent patterns in the reliability dipoles are immediately obvious. Such patterns would indicate regions of ternary forecast space in which the forecasting system consistently assigned the wrong probability to one of the three categories.
The decomposition diagram (top left) illustrates geometrically the decomposition of the root-score in this dataset. Thus, the root-score is seen to be rather poor compared with the best-possible root-score that could be attained through recalibration.
(c) Forecast maps including verification data
The map of figure 7 contains information about forecasts but not about the skill of the forecasting system. In order to incorporate verification data into a forecast map, the array of coloured squares can be replaced by an array of coloured circles whose radii are a measure of forecast skill in a verification dataset. One possibility would be to set radius proportional to (Z−R)/Z which is a ‘standard’ definition of skill. In this paper, however, the convention has been to consider quantities like and which can be interpreted as distances in a triangle rather than squared distances (figure 10) and so is preferred here. (It is straightforward to relate these two possibilities via the identity .) Making the choice 4.4it follows that larger circles will be plotted in regions of high skill, and no circle will be plotted if the forecasting system has less skill than a climatological forecast (i.e. when the assigned radius is negative). A map produced in this way is shown in figure 12. This map shows clearly that, for the season shown, the skill of this forecasting system is greatest near the northern coast of South America. Maps of this type should prove useful in communication of operational forecasts because the reader's eye is drawn to areas where the forecasting system has been shown to perform well.
5. Recalibration of ternary probabilistic forecasts
The decomposition diagram in figure 11a illustrates that significant improvement can sometimes be made to the mean-score by recalibrating the forecasts. The score of the original forecasts is and so the aim is to produce recalibrated forecasts , calculated from the original forecasts via some specified functional form, whose score is minimized. As an example, consider recalibrated ternary forecasts which are a quadratic function of the original ternary forecasts (appendix Ad). Given a dataset of forecasts and observations, standard numerical optimization techniques can be used to find coefficients in the quadratic function which minimize the score of the recalibrated forecasts. The recalibration function can be interpreted by comparing figure 11a,b. Recalibration minimizes the score by changing the reliability (that is, the r.m.s. length of the red dipoles). It is important to note that recalibration changes the forecasts (black dots) but not the observations (red circles) and so the r.m.s. length of the red dipoles has been reduced from to solely by moving the position of black dots. The effect of this reduction can be seen in the decomposition diagrams (top left). Recalibration changes neither the root-uncertainty nor the root-resolution , but reduces the root-score from to .
In this case, it is immediately clear that the recalibrated forecasts (figure 11b, black dots) are clustered around an arc passing close to the climatology (blue cross). It is interesting that this arc is similar to the contour in figure 8b which applies when both the climatology and the forecast are Gaussian with equal variance.
Finally, the sharpness diagram (figure 11b, top right) shows the distribution of the recalibrated forecasts. It is produced by passing the sharpness diagram of the original forecasts (figure 11a, top right) through the quadratic recalibration function (appendix Ad). Regions of the sharpness diagram coloured grey indicate possible forecasts that were not issued originally. Regions coloured white indicate forecasts that can never be issued under this recalibration.
This article has outlined a novel procedure for visualizing ternary probabilistic forecasts. The proposed maps convey through colour all of the information present in a ternary forecast. A colour palette printed next to the map can be used to deduce the ternary forecasts directly from the colour used.
The proposed colour scheme always assigns the colour white to a forecast close to climatology, and strong colours to forecasts that differ greatly from climatology. The examples considered in this paper have concerned precipitation and so the ‘wet’ colour blue has been assigned to the above-normal category A and the ‘dry’ colour red to the below-normal category B. It is straightforward to reverse these conventions if desired for other forecast variables (e.g. temperature). The default palette (with θ0=0) assigns strong red, yellow and blue, respectively, to forecasts that assign high probability to one of the three categories. Yellow has been chosen in preference to green in order to assist colour blind readers. A variety of palettes can be created by varying the parameters m and θ0 in equation (3.4).
A novel visual interpretation of ternary forecasts has also been suggested for verification and recalibration under quadratic scoring functions. It has been shown that Brier scores correspond to squared distances in an equilateral triangle and ranked probability scores correspond to squared distances in a right-angled triangle. Thus, the root-score, root-uncertainty, root-resolution and root-reliability can all be interpreted as r.m.s. distances. Verification data can be visualized using the proposed ternary reliability diagrams (figure 11), which incorporate a decomposition diagram (top left) and a sharpness diagram (top right) to aid interpretation. The geometrical interpretation can also be applied to nonlinear recalibration of probabilistic forecasts.
It is hoped that the procedures outlined here will prove useful in the interpretation and communication of operational weather and climate forecasts. When no information on skill is available a map like that of figure 7 can be produced. If skill information is available, it can be incorporated as in figure 12.
This work was supported by the EUROBRISA network project (F/00 144/AT) kindly funded by the Leverhulme Trust. The dynamical ensemble forecast data were kindly provided by ECMWF as part of the EUROSIP project. Three forecasting centres are the partners in EUROSIP, these are ECMWF, the UK Met Office and Météo-France. C.A.S.C. was supported by Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP), processes 2005/05210-7 and 2006/02497-6. R.L. and D.B.S. thank Aidan Slingsby of City University for useful discussions.
Let the triangle in induced by the matrix L have vertices BNA (figure 13) and sides of length b, n and a: A1From the cosine rule, the angle ϕ is given by A2It follows that a ternary forecast and the associated point within the induced triangle are related by A3where the transformation matrices are defined by A4
(a) Brier score
The Brier score is defined by A5It follows that, for the Brier score, A6The triangle induced by the Brier score is an equilateral triangle with unit sides b=n=a=1.
(b) Ranked probability score
The ranked probability score [3,4,20] is defined by A7It follows that, for the ranked probability score, A8The triangle induced by the ranked probability score is a right-angled triangle with sides , n=1.
(c) The uncertainty of a quadratic score
For a quadratic scoring rule (equation (3.1)), the uncertainty U(q) is equal to the expected score when the climatology q is issued as the forecast. It follows that A9It follows that A10In the particular cases of the Brier and ranked probability scores: A11
(d) Quadratic recalibration
An original forecast p′=(pB,pN,pA) is defined by pB and pA since pN=1−pB−pA. Define a recalibrated forecast by A12Numerical values for the parameters C1,…,C12 can be found by minimizing the mean score of the recalibrated forecasts.
One contribution of 13 to a Theme Issue ‘Climate predictions: the influence of nonlinearity and randomness’.
- This journal is © 2012 The Royal Society