Can we automatically assess the understandability of a given snippet of code? This is the question that the paper “Automatically Assessing Code Understandability: How Far Are We?” authored by Simone Scalabrino, Gabriele Bavota (Software Institute), Christophere Vendome, Mario Linares-Vásquez, Denys Poshyvanyk, and Rocco Oliveto tried to answer.

The paper has been selected as the recipient of an ACM Sigsoft Distinguished Paper Award at the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2017). ASE is considered one of the top software engineering venues and this year accepted 65 out of the 314 submitted technical papers (21% acceptance rate). Only three of the accepted papers have been awarded.

The preprint of the paper is publicly available, while the abstract is reported in the following:

Program understanding plays a pivotal role in software maintenance and evolution: a deep understanding of code is the stepping stone for most software-related activities, such as bug fixing or testing. Being able to measure the understandability of a piece of code might help in estimating the effort required for a maintenance activity, in comparing the quality of alternative implementations, or even in predicting bugs. Unfortunately, there are no existing metrics specifically designed to assess the understandability of a given code snippet. In this paper, we perform a first step in this direction, by studying the extent to which several types of metrics computed on code, documentation, and developers correlate with code understandability. To perform such an investigation we ran a study with 46 participants who were asked to understand eight code snippets each. We collected a total of 324 evaluations aiming at assessing the perceived understandability, the actual level of understanding, and the time needed to understand a code snippet. Our results demonstrate that none of the (existing and new) metrics we considered is able to capture code understandability, not even the ones assumed to assess quality attributes strongly related with it, such as code readability and complexity.