During the week of September 19th, NRC provided pre-release access to its long-delayed “Data-Based Assessment of Research-Doctorate Programs in the United States,” scheduled for public release during the week of September 26th.
We, along with colleagues in other computer science programs nationally and colleagues in programs in other fields at the University of Washington, quickly discovered significant flaws of three types in NRC’s data:
- Instances in which the data reported by NRC is demonstrably incorrect, sometimes by very substantial margins.
- Instances in which the accuracy of the data cannot easily be checked, but it does not pass even a rudimentary sanity check.
- Instances in which institutions interpreted NRC’s data reporting guidelines differently, yielding major inconsistencies.
The widespread availability of the badly flawed pre-release data within the academic community, and NRC’s resolve to move forward with the public release of this badly flawed data, have caused us and others to urge caution. For our program – and surely for many others – the NRC assessment is based on data that is clearly erroneous and the results are therefore meaningless.
Quoting the Computing Research Association: “NRC Doctoral Rankings for Computer Science not ready for prime time … CRA has serious concerns about the accuracy and consistency of the data being used in the evaluation of the Computer Science discipline.”
See further information here.