The biggest problem I have when writing a science article is figuring out the right time to mention how valid I think the results are. Sometimes I come across a story which is popular, but might have some problems that mean that some of the conclusions don't stand up to scrutiny.
As such, I feel compelled to question them, to expose said problems. Which gives me another dilemma. Once I say there is one flaw in a paper, that becomes the story. Which is a big issue, because most papers have flaws in them. It doesn't mean those papers can't be useful, it usually means you have to be careful with what you take away from them.
So why report on flaws at all ? It's not just because I feel we need to hold our scientists and journals to the best standards available, but that it's useful to have these kinds of discussions in the open. It's also about trust. If I report on an exciting new scientific discovery that I have reservations about, I feel that I am doing you a disservice by not reporting those reservations.
So I've boiled down my criticisms of a single paper into a boxout at the end of my articles, with a simple five star rating.
This rating reflects only my own personal views. I am not an all knowing computer that can automatically analyse every error that crosses my path. I am human, and I can make mistakes. It's a work in progress, and I will be updating it as I go on.
The Validity Rating scale
5/5- This paper has no statistical errors, and presents exemplary evidence to back up its assertions.
4/5 - The paper has some statistical errors, or no statistics at all, but the evidence is still strong enough to back up the assertions of the paper. Or the paper is fine statistically, but there are still more experiments needed before it's conclusions can be taken as read.
3/ 5- There are statistical errors, and problems with the data analysis indicates weak evidence for one or more assertions of the original paper.
2/5 - There is enough problems with the paper that at least one of the assertions is completely wrong, but there is still something useful that can be salvaged from this paper.
1/5 - There are huge problems with this paper, enough to offend me on a personal level. The errors are enough to colour the validity of all of the results presented. But there is a kernel that can be salvaged from this paper.
0/5- The experimenters clearly have no idea what they are doing, and cannot be trusted on any level, because of terrible experiments or obvious plagiarism or there is no data, and thus no evidence.
U - The data/ theory is unavailable, or not presented in a peer reviewed scientific publication. Treat as highly suspicious.