Machine Learning an Experimental Science (Revisited)

  1. (PDF, 212 KB)
AuthorSearch for:
ConferenceProceedings of the Evaluation Methods for Machine Learning Workshop of the Twenty-First National Conference on Artificial Intelligence, July 16-20, 2006., Boston, Massachusetts, USA
AbstractIn 1988, Langley wrote an influential editorial in the journal Machine Learning titled “Machine Learning as an Experimental Science”, arguing persuasively for a greater focus on performance testing. Since that time the emphasis has become progressively stronger. Nowadays, to be accepted to one of our major conferences or journals, a paper must typically contain a large experimental section with many tables of results, concluding with a statistical test. In revisiting this paper, I claim that we have ignored most of its advice. We have focused largely on only one aspect, hypothesis testing, and a narrow version at that. This version provides us with evidence that is much more impoverished than many people realize. I argue that such tests are of limited utility either for comparing algorithms or for promoting progress in our field. As such they should not play such a prominent role in our work and publications.
Publication date
AffiliationNRC Institute for Information Technology; National Research Council Canada
Peer reviewedNo
NRC number48550
NPARC number8913623
Export citationExport as RIS
Report a correctionReport a correction
Record identifierd5dd1ff4-a6c9-47ab-9297-129cb23ea5d7
Record created2009-04-22
Record modified2016-05-09
Bookmark and share
  • Share this page with Facebook (Opens in a new window)
  • Share this page with Twitter (Opens in a new window)
  • Share this page with Google+ (Opens in a new window)
  • Share this page with Delicious (Opens in a new window)
Date modified: