PASS logo

2021 Summit video library

Lessons from Spreadsheet Data Horror Stories

Patrick O'Beirne

There are many pitfalls in using Microsoft Excel to processing important data. I present published reports of horror stories and draw lessons from them. Firstly, how unskilled Excel users corrupted gene name records; researchers decided to rename genes to suit Excel! The lesson is to learn how to import CSVs into Excel correctly. Then I describe the Public Health England debacle, where they lost Covid lab results. That illustrates the risks of automated data imports without safety controls and the lesson is a basic reconciliation technique. Finally I focus on data insecurity as illustrated by privacy regulators’ penalties for data leakage. The lesson is to understand how your analysis and reporting tool really works.