8 Questions for reading someone else's report

1. How much did it cost to collect this data?
How does it compare to the magnitude of the decisions that will be drawn from it?

2. Did they "scrub" the data -- data hygiene?
Are you looking at all the data, or someone else's preconceptions of what the data should look like?

3. Did they plot the data?
90% of what you conclude is usually visible in simple plots.

4. If there was a time series, was data plotted against time?
Always be on the look out for omitted variables -- time is number 1.

5. Were they looking for statistical significance against an irrelevant value?
T-tests and p-values are meaningless unless you know what the hypotheses were. Usually stat packages default to testing that the parameter is 0 and you may not be interested in the default.

6. Were they R-squared challenged?
Has someone found a great R-squared then assumed that everything else was OK because of that?

7. Is it clear how categorical variables have been coded?
Computer packages usually have a default categorical variable coding. You need to know what it is to interpret the coefficients.

8. Did they do prediction out of sample? In-sample prediction may lead to a false sense of accuracy, and the problem gets worse as you increase the complexity of the model.