| How reliable are these predictions? |
|
Each day a check is made on the predictions of the previous day. The following figures and statistics describe the reliability of the predictions published to date on the web, based on these checks. (Note that a summary of the predictions is provided here.)
|
|
Total days: 529 Event days: 51 Mean (f): 0.194 Mean (x): 0.096 Median (f): 0.097 Std dev (f): 0.187 Std dev (x): 0.295 Mean (f|x=1): 0.349 Mean (f|x=0): 0.177 Median (f|x=1): 0.301 Median (f|x=0): 0.089 Std dev (f|x=1): 0.226 Std dev (f|x=0): 0.174 Discrimination: 0.172 ME: 0.098 MAE: 0.223 MSE: 0.101 Linear assoc: 0.272 Skill: ***** |
|
Figure 1: Reliability plot for M-X event prediction, and associated
statistics.
|
|
Total days: 529 Event days: 12 Mean (f): 0.042 Mean (x): 0.023 Median (f): 0.013 Std dev (f): 0.067 Std dev (x): 0.149 Mean (f|x=1): 0.221 Mean (f|x=0): 0.038 Median (f|x=1): 0.259 Median (f|x=0): 0.013 Std dev (f|x=1): 0.127 Std dev (f|x=0): 0.059 Discrimination: 0.182 ME: 0.020 MAE: 0.055 MSE: 0.019 Linear assoc: 0.407 Skill: 0.148 |
| Figure 2: Reliability plot for X event prediction, and associated statistics. |
|
In more detail, each plot may be understood as follows. The forecasts for all days are binned in increments of 0.05. For each bin, all the days on which predictions were made in the range of that bin are examined. The observed number of those days on which at least one event did occur is used to estimate the underlying probability of an event on those days. That is the vertical value of the plot, for that bin. Similar "reliability" plots for predictions issued by the US National Oceanic and Atmospheric Administration are given here and here. Next to the two plots above are additional statistics on the reliability of the predictions. In these tables, f refers to the forecast, and x to the observations. For each day x is either one or zero, i.e. one or more flares did or did not occur. For each day f is the value of epsilon for the forecast, i.e. the assigned probability for the occurrence of at least one flare. The statistics of f and x are shown.
Figure 3 shows plots of running "skill scores" for the predictions of M-X
events (upper panel) and X events (lower panel). The skill score is a
measure of how good the predictions are by comparison with a simple
forecast consisting of the average of the observed value. A skill score of
one is perfect prediction, and a positive/negative skill score represents
better/worse prediction than the simple forecast. The final values on these
plots are the skill scores for all days of predictions and observations to
date, which are also listed in the tables to the right of Figures 1 and 2.
|
|
|
| Figure 3: Running plots of skill scores for the predictions to date. |
|
|
|
Main flare prediction page
More detail on how today's predictions were made Summary of all predictions made to date The prediction for 4 November 2003 A test of the method on historical data Links to other pages related to flare prediction Mike Wheatland's home page |
|
|
|
|
| Page maintained by m.wheatland@physics.usyd.edu.au | Page last updated Wednesday, 01-Dec-2004 15:39:06 EST |
