Researchers have been studying the productivity effects of remote work and work from home for years, and generally the results have trended in a positive direction. Stanford professor Nicholas Bloom, for example, talks about how we’ve now got a “working from home economy” that coexists with the gig economy and information economy, and why we need to better understand and design for working from home because

Without this historic switch to working from home, the… economy would have collapsed, forcing us to return to work, reigniting infection rates. Working from home is not only economically essential, it is a critical weapon in our fight against COVID-19 – and future pandemics.

One of the things researchers have argued is that productivity among people who work from home is generally higher: for example, in a 2014 study (“Does Working From Home Work? Evidence From a Chinese Experiment”), Bloom and his colleagued found that at one Chinese travel company, Ctrip,

Home working led to a 13% performance increase, of which 9% was from working more minutes per shift (fewer breaks and sick days) and 4% from more calls per minute (attributed to a quieter and more convenient working environment). Home workers also reported improved work satisfaction, and their attrition rate halved, but their promotion rate conditional on performance fell.

Working from home is often described as increasing productivity, but why is that? Is it because people are better able to manage their time, have more energy because they don’t have long commutes, have fewer distractions, or what? Being able to tease apart the strands is actually important because if we’re going to be working from home, we need to understand how to make it better. But work is often a hard-to-measure, slippery thing, which makes improving it tricky.

(This isn’t just a long-term issue: Bloom argues that work from home could set off “a time bomb for inequality” in which “educated, higher-earning employees are far more likely to work from home… [and] get paid, develop their skills and advance their careers,” while “those unable to work from home… face bleak prospects.”)

So I was really interested to see a study from some German researchers that looked at a highly specific kind of working from home situation: playing chess for money. Steffen Künn and Christian Seel of Maastricht University, and Dainis Zegners of Erasmus University, have constructed an elegant natural experiment from the lockdown, which they describe in their new article, “Cognitive Performance in the Home Office – Evidence from Professional Chess.”

As they explain,

During the recent COVID-19 pandemic, traditional (offline) chess tournaments were prohibited and instead held online. We exploit this as a unique setting to assess the impact of moving offline tasks online on the cognitive performance of individuals. We use the Artificial Intelligence embodied in a powerful chess engine to assess the quality of chess moves and associated errors.

One of the virtues of world-class tournament chess is that it’s eye-wateringly well-documented: every professional’s move is recorded, the results of tournaments are online, players’ ranks are known, and judges (and now AI) evaluate whether particular moves are especially smart or dumb. For people interested in work, it’s also good because chess is both really complicated, and a very well-defined set of tasks with clear outcomes. As the researchers explain,

Comparing online with offline chess tournaments offers several advantages for assessing the impact of moving tasks online on cognitive performance. First, playing chess is a purely cognitive task, which requires complex strategic decision making under time constraints. Therefore, chess offers a unique setting for studying performance in a cognitive task, which is important in many modern professional, managerial, technical, and creative occupations (Autor and Price, 2013). Second, although until very recently high-stakes chess tournaments were almost exclusively conducted with players competing face-to-face in physical playing halls, most chess players are very familiar with unincentivized online chess on various chess platforms. Due to the recent COVID-19 pandemic, several online tournaments are being organized in which many of the world’s elite players are participating, usually playing from their homes. These tournaments offer significant amounts of prize money to the players providing them with high incentives for performance. Third, using the Artificial Intelligence embodied in modern chess engines makes it possible to construct a benchmark of individual performance that is based on fine-grained move-by-move data with a high degree of objectivity and accuracy. This benchmark makes it possible to analyze both the probability and magnitude of making mistakes during a chess game. Fourth, since all major global chess events were canceled during the pandemic, we are able to observe a representative sample of elite players competing both online and offline, ruling out selection-effects influencing our results.

While you can’t argue that chess is exactly like your job, still all this makes for a really interesting study: it’s the same people, playing the same game, under the same tournament and time rules, against similar opponents. The “only” thing that’s different is that they’re playing at home, not in some conference center or hotel ballroom.

So what do they find?

Analyzing 27,267 individual moves played during 441 games in a regression model with player fixed effects, we provide evidence for a statistically and economically significant decrease in performance when the same players compete online compared to competing offline. While the probability of playing the best move as suggested by the chess engine is not statistically different online and offline, we find that conditionally on making an error, the magnitude of the error is 16.8% larger online for the sample player…. Thus, the cognitive performance of chess players is impaired when playing online.

The conclude, “Our results suggest that teleworking might have adverse effects on workers performing cognitive tasks.”

This is not to say that the productivity gains reported in other studies are illusory, or that everyone should now head back to the office. What this does suggest is that immediate productivity gains from working at home may not come from being able to perform specific tasks more rapidly or accurately, but from broader changes that boost performance: having more energy to devote to tasks when you’re not fighting traffic and crowds for 45 minutes before you get into work, having fewer interruptions, etc.

One of the implications of this work is that if the performance improvements come more from environmental factors than from cognitive improvements, then we can use work from home to improve office work: to redesign offices of the future to be more productivity-supportive places, rather than the carnivals of distraction that they’ve become.