5-day weather forecasts are available from a number of sources now - but how accurate are they, really?

My reason for asking is this:

If I make any kind of prognostication about something supposed to happen 5 days hence, and I have unlimited opportunity to change my prediction, right up until the morning of the day, doesn't that grant me licence to say whatever I like now, knowing I can bother with the trifling business of being accurate when we get closer to zero hour and proper prediction is that much easier

An Audit

So let's test the value of the 5 day forecast. What I intend to do is record the weather forecast for each day, as it progresses toward the present, through the 5-day forecast on the Met Office website.

I'll also be recording the actual weather outcome for the same local area.

As the data builds, we'll hopefully be able to see two things:

• How often, and by how much, the 5 day weather forecast changes as it progresses through to the present
• How accurately and how reliably can the weather be predicted, from 5 days out

Metrics

These are the various different weather types as defined by the Met Office - I've arranged them here into a sort of periodic table of weather - and it's this grid I'll use to score the forecasts on reliablility and accuracy.

Each day's forecast will be given a consistency penalty as follows:

• One point for each time it moves a column from the previously forecast weather for the day
• Two points for each time it moves a row from the previously forecast weather for the day
• Four points if the final forecast for the day is different from the initial forecast (this penalty avoided if the forecast reverts to the original one)
• Eight points if the forecast on the day is different from all of the previous forecasts

 

Each day's forecast will be given an accuracy penalty as follows:

• One point for each column away from the actual observed local weather on the day
• Two points for each row away from the actual observed local weather on the day
• Four points if the actual observed local weather on the day is different from the initial forecast
• Eight points if the Actual observed weather on the day is different from all of the forecasts

 

Location

I'll be monitoring two geographical points - Portsmouth for weekday forecasts and Southampton for weekend ones (because that's work/home - so I can make the observations on the day).

So, without further ado, let's start measuring...

Sunday, 14th June 2009 - Southampton

Forecasts

On 10th	On 11th	On 12th	On 13th	On 14th	Actual

Analysis
Well, this is a fine start - confirming my every suspicion so far - that is, the five-day forecast was completely useless - predicting heavy showers. Only the forecast on the morning of the actual day was correct - so the service provided by this forecast was approximately equivalent to that of an ordinary window.

Penalty scores: Inconsistency: 18, Inaccuracy: 17. Pretty poor.

Monday, 15th June - Portsmouth

Forecasts

On 11th	On 12th	On 13th	On 14th	On 15th	Actual

Analysis
Again, not a particularly useful or precise forecast - changing every single day, then being wrong on the day.

Penalty scores: Inconsistency: 15, Inaccuracy: 21. Shocking.

Tuesday, 16th June - Portsmouth

Forecasts <

On 12th	On 13th	On 14th	On 15th	On 16th	Actual

Analysis
OK, not bad. Only one fairly small change, and a reasonably accurate and reliable forecast over all.

Penalty scores: Inconsistency: 7, Inaccuracy: 13. Still room for improvement.

Wednesday, 17th June - Portsmouth

Forecasts

On 13th	On 14th	On 15th	On 16th	On 17th	Actual

Analysis
Forecast changing every single day, but not by much - getting it right on the morning of the 17th and once on the 15th, but maybe more by chance than anything else.

Penalty scores: Inconsistency: 15, Inaccuracy: 11.

Thursday, 18th June - Portsmouth

Forecasts

On 14th	On 15th	On 16th	On 17th	On 18th	Actual

Analysis
Reasonably consistent throughout the 5-day run up, but just never actually right.

Penalty scores: Inconsistency: 4, Inaccuracy: 29.

Friday, 19th June - Portsmouth

Forecasts

On 15th	On 16th	On 17th	On 18th	On 19th	Actual

Analysis
I suppose it had to happen sooner or later - the forecast from 5 days was right! - although it did get a bit shaky in between.

Penalty scores: Inconsistency: 10, Inaccuracy: 5. Hey, not bad.

Saturday, 20th June - Southampton

Forecasts

On 16th	On 17th	On 18th	On 19th	On 20th	Actual

Analysis
Reasonably consistent, but just wrong - it rained, and this was completely unpredicted. That's possibly the most inconvenient kind of inaccuracy - I could have spent the last five days planning something that was reliant on dry weather.

Penalty scores: Inconsistency: 11, Inaccuracy: 33. Atrocious.

Conclusion

I'm well aware of the dangers of drawing a firm conclusion based on small samples, but what sort of picture is painted the results above?

The first thing to note is that in this test, the actual daily weather was not particularly variable - less variable from day to day than some of the five day forecasts. Despite that, the general level of consistency and accuracy in the forecasts was pretty poor.

Maybe I should take another set of samples during a period of more changeable weather - perhaps in the autumn, but for now, my feeling is that my suspicions are beginning to have been confirmed - that 5-day weather forecasts are rather meaningless and useless, being likely to be superseded by different forecasts nearer the time.