I’m inclined to the opinion that biologists should be banned from using statistics. The practice of drawing inferences from gathered data is full of pitfalls; those who collect and analyse data on fish and river fly life seem especially prone to stumbling into them. Such data have many potential shortcomings and limitations as I pointed out in a previous post, yet to read the reports from the Riverfly Partnership and SmartRivers (part of Wildfish) you’d think the gods themselves were illuminating our understanding. Sadly not. We know the condition of our rivers is deteriorating and we know broadly the reasons: sewage, agriculture and now noxious pharmaceuticals, some from our own bodies. No fly monitoring needed to tell us that.
If all the long-term fly monitoring were able to tell us whether things are obviously improving or getting worse that would be useful. My attention has lately been drawn to some news regarding the Hampshire Avon. Apparently monitoring has revealed a steep decline in bugs over the last ten years. Numbers don’t lie, do they? No, and nor do statistics, but misusers of statistics can present us with questionable or false conclusions which the data just don’t support.
In the report SmartRivers claims that fly life on the Hampshire Avon has declined over the last ten years. There is little argument that aquatic fly life has fallen in most if not all rivers in the country, but this started well before the last decade. Still, if this slide in populations is continuing, we should know about it. So what does this report actually tell us? One of the problems of these reports, and those which appear in the Wild Trout Trust annual, Salmo Trutta, concerns the kind of publication they’re trying to be. Technical research paper or report for a general readership? They end up being neither. Jargon is scattered throughout together with some rather slapdash analysis, certainly not up to journal standard (and as far as I’m aware, the content of this report has not been published elsewhere). So not a science paper but not a read for the layman either. The technical references to inferential statistics put paid to that.
As often the case, the statistical testing is applied rather automatically without much attention paid to the obvious features of the data. One of the reasons is that computer packages allow the copious production of statistical calculations at a click rather than the hours of laborious work once necessary. Another is that all the other biologists are doing it. The main purpose of such tests is to decide whether what you see in the data is telling you something important or is just a fluke of sampling. The thinner the sampling, the more likely the result is down to chance. Riverfly sampling is pretty thin. Here a handful of sites are sampled twice a year. Quoting a figure called a p-value, a probabilistic measure widely misused, is not sufficient to decide whether the Avon fly population has gone down since 2015.
The observed changes are clearest in the graph below, taken from the report. The text will be hard to follow for most; even some abbreviations go undefined.
Whether the spring and the (usually lower) autumn counts should appear on the same graph is open to question. But what is clear is that 2015 counts are higher than 2024. The straight lines fitted to the data points suggest a steady decline, except that the actual figures do not really show this. Counts were obviously higher in 2015 than subsequent years. If anything, the data suggest a cyclic component to the observed counts. Remove the 2015 figures and suddenly the decline vanishes, or at least becomes a lot less clear. The Environment Agency noted this feature of the data but the report’s authors respond that ‘this is simply the dataset we have’. Hardly a justification for the conclusion they draw. If they had looked at the appropriate statistical figures, they should have concluded that evidence for decline in riverflies these last ten years is weak at best. The biologist’s best analysis tool is their eyeballs.
Why are the 2015 figures higher? We don’t know. Maybe it was a good year for fly, or maybe the sampling was more thorough. There are data going back further in the Riverfly database, so I’ve pulled out these graphs for the Avon, though as you’ll see from the four riverfly abundances, regular sampling only began around 2010, and probably was less regular then.
You can see the seasonal variation, and there is some evidence of cyclicity over the years. But is there any long-term trend? You decide.
The Secret Angler 