Where it stands
Two very different questions. Only one of them has an answer yet, and pretending otherwise would defeat the point.
Reading the record...
How it works
Three steps. The order is the entire safeguard, and it is the only thing you really need to understand about this project.
- Predict tomorrow, tonight. The prediction is published to a public website the evening before the day it covers. An outside company, GitHub, records the time it arrived, so the timestamp is not something this project can fiddle with.
- Once the day starts, it is sealed. The software will not let a prediction be written or edited for a day that has already begun. If a day gets skipped, it stays skipped and is shown as a blank in the record, rather than being quietly filled in afterwards.
- Six weeks later, mark it. Earthquake records get revised for weeks after the event as scientists refine their measurements, so the scoring waits 45 days for the numbers to settle. Then the prediction is compared with what really happened, using the standard tests earthquake forecasters use on each other. Good result or bad, it gets published.
Every prediction, and how far ahead it was made
One row per prediction. Reading left to right is reading forwards in time.
- the moment the prediction was written down and sealed
- the head start: time between writing it down and the period starting. This is the part that cannot be faked
- the period being predicted, a day or a week
Reading the record...
Why the gap is the point
Every forecasting portfolio shows a backtest. Backtests are cheap and unfalsifiable: fit a model to history, tune until the number looks good, publish the number. Nobody can tell what was tried and discarded.
This cannot do that. A forecast is written to git with a content hash and a timestamp GitHub records on its own servers, hours before the window opens. Publishing into a window that has already started raises an error instead of writing a file, so a missed window stays missed and shows below as a permanent gap in the record.
The same thing, as a list
Every period that has begun so far. Periods still in the future have already been predicted, and appear here once they start. Each receipt links to GitHub's own record of when the prediction arrived.
| Period predicted | Method | Covers | Which quakes | Written down | Head start | Result | Receipt |
|---|---|---|---|---|---|---|---|
| Reading the record... | |||||||
Do not take this page's word for it
This page is produced by the very project it reports on, which is exactly the situation where you should want an independent check. So there is one, and anyone can run it.
git clone https://github.com/jjxobi/ex-ante cd ex-ante python scripts/audit_history.py --online
Those three lines download the whole history of the project and check it. Not the current state of the files, the entire history, so a prediction that was edited and then edited back still gets caught. It also asks GitHub directly when each prediction arrived, rather than believing the timestamps this project wrote about itself.
It is deliberately built the way a stranger would build it, sharing no code with the forecasting side, because a checker made from the same parts would inherit the same mistakes. If it cannot do a proper check, it refuses to run rather than report a clean result it has not earned.
the technical version
Reads git history rather than the working tree, verifies manifest checksums at every commit that touched a forecast, confirms no forecast file was modified after its window opened, checks a single grid hash across the record, and refuses to run against a shallow clone. Every check is mutation tested: disabling any one breaks at least one test.
What is being predicted, and what is not
This does not predict individual earthquakes. Nobody can do that.
What it predicts is how many. New Zealand is divided into a grid of small squares, and for each square the forecast gives the number of earthquakes to expect over the coming day or week. Think of it like rain: not a claim that it will rain at 3pm on your street, but an expected amount over an area. Only quakes of magnitude 3 and above are counted, roughly the point where people start to feel them.
Quakes are also split into shallow and deep, above and below 41 km down, and predicted separately. They behave differently enough that mixing them would hide errors in both.
The grid, the size cut-off and the depth split were all fixed in advance and cannot be changed now. That matters: if you can move the target after seeing the result, you can make almost any forecast look good.
the technical version
Expected count of M3.0+ events per cell per window over a frozen 4,100 cell grid at 0.1 degree resolution, two depth strata split at 41 km, fitted separately. Grid, threshold and boundary are hash locked. Scoring uses the CSEP N, S, M and L consistency tests, with information gain as the skill score against the baseline.
A flaw published on purpose
Over the stretch of past data it was built on, this method expected about a third more earthquakes than actually happened: 15.2 a week against 11.4. That was measured before any of this went live, and left in place rather than quietly fixed.
That figure is not a fixed error waiting to be subtracted off, and it is not a promise about the future. The method assumes earthquakes arrive at a steady rate. They do not: one large quake sets off weeks of aftershocks. So it reads high through quiet stretches and would read low during an active one. Whether it runs a third high on days it has never seen is precisely what nobody knows yet, and precisely what the results from September onwards will show.
It stays because a forecast claiming no known weaknesses is the least believable kind, and because this is the bar a better method has to clear. Correct it now and there is nothing honest left to measure the next attempt against.