A mistake in numerical codes causes retraction of our Nature paper on Quantum Moves
A mistake in the numerical codes of our 2016 Nature paper on Quantum Moves has recently been identified and it is therefore now being retracted. In this blog post we review what the paper was about, and why the error had to lead to retraction. Finally, we address the question if this means that the efforts of all the Quantum Moves players are now wasted and give three reasons why we believe this is not the case.
As explained in ScienceAtHome Rewinds Part 1 and Part 2, the entire ScienceAtHome project originated from a desire to gamify some of the quantum challenges related to our experimental work on developing quantum computers and simulators. In the years 2012-2015 we iteratively developed the citizen science game, Quantum Moves, and finally had enough player data to start comparing to the performance of our conventional optimization algorithms. Although players did not have the fine motor skills to solve the tasks perfectly, we found among the data instances of surprisingly fast and decent quality solutions in a regime where our algorithms performed rather poorly. This inspired us to introduce a new collaborative paradigm where the player solutions were used as input for a subsequent thorough machine optimization. Here, players supplied the overall shape of the solution and the algorithm took care of the methodological fine adjustment necessary to gradually transform the initial guess into increasingly good solutions. In this way, we obtained dramatically improved results which were, in 2016, published in the prestigious scientific journal Nature under significant media attention.
In the scientific process, published scientific results are often later explored by other researchers both as i) inspiration for new research directions and as ii) exploration of the domain of validity of the reported findings. In particular, when unconventional and surprising research results are presented, this latter process of systematically putting claims to the test is an essential part of ensuring that scientific progress stays on track. In our case, Dries Sels and Allan Grønlund from Boston University and Aarhus University, respectively, took on the task of methodically testing our reported findings. Their analysis seemed at odds with our results and in December 2019, Grønlund discovered that the reason for this discrepancy was an erroneous implementation of our optimization algorithm.
Simply speaking, the error meant that we had inadvertently compared player performance to a severely handicapped algorithm. When we corrected the mistake and reran the comparison, it turned out that the purely numerical approach performed roughly on par with, and not worse than, the human-computer results we had originally reported in Nature. This meant that many of our conclusions in that paper were no longer valid. Subsequently, we contacted the editors of Nature and provided them with a detailed analysis of the implications of the error. Based on this analysis, the editors concluded that the best path forward was a full retraction of the paper.
As an author, it is of course not a pleasant experience to have made a mistake that leads to the retraction of a scientific paper. In fact, it is probably among the most dreadful experiences for any scientist. On a personal level, the past months have been extremely tough because as a scientist, your most valued resource is your personal integrity. Both as individual scientists and on behalf of science at large it is crucial that our statements can be trusted both by the general public and by our academic peers. In hindsight, it is excruciating to think of each of the small numerical tests we now know that we could have performed at the time to have brought the error to light before publication. Similarly, we must admit that, while we always believed to have solid arguments to explain any discrepancies, we should have been more open to the possibility of the existence of an error in our numerical codes.
While we cannot exclude errors from happening in the future, we have now introduced methodological, industry-standard software testing procedures to test new units of code, and we have benchmarked our new algorithms with respect to the results obtained by other groups on several state-of-the-art challenges. In addition, wherever we are not limited by GDPR considerations, we now strive to make new codes openly accessible for both peer review and inspiration. The process has been a stark reminder to us that scientific progress is not just about making new discoveries but just as importantly about questioning your established beliefs.
Although ideally the trial and error process should be considered a natural part of scientific exploration, making mistakes in science is in practice always painful. In the end, however, we can only conclude that when scientists make mistakes, it is a testament to the robustness of the scientific approach that follow-up efforts by the community can bring such mistakes to light. In our case, we would like to thank Allan Grønlund and Dries Sels for taking on this, often thankless, task.
Does this mean that the efforts of the players in Quantum Moves were wasted and that the approach of gamifying quantum tasks is useless?
We do not believe so but before we answer this question, let us first give some more background. Due to the immense growth of artificial intelligence (AI) applications in society, there is an intense interest in understanding and pushing the boundary between human intelligence and AI. Due to this zeitgeist, our work has often been put in the context of either presenting evidence that human problem-solving will or will not still have a role in the future of work and research. Also, our work has sometimes been presented as controversial evidence that complex challenges could be solved effortlessly by amateurs using only the power of their human intuition, indicating that perhaps in the future there would be no need for long and tedious education and training. Popular media headlines like LA Times’ “Take that AI: video gamers solve quantum physics mystery using human intuition” have definitely spurred on these exaggerated interpretations. In retrospect, some of our early communication may also have been unclear in this regard (in part because our own understanding of the underlying mechanisms is continuously evolving). In our more recent work, we try to communicate the possible interpretations and limitations of our results more clearly.
In what follows, we give three main reasons why we do not believe that the efforts of the players in Quantum Moves were wasted or that the approach of gamifying quantum tasks is useless.
Reason 1: Our Quantum Mechanics gamification followed in the footsteps of related efforts in other fields
As mentioned above, it was surprising that players seemed to be able to help solve complex quantum challenges in quantum physics without any formal training in the field. This might have seemed less incredible if we had made it much more clear that our work was a natural extension of preceding efforts in protein and RNA folding. Here, projects like FoldIt and EteRNA have clearly demonstrated that a suitably-defined gamified interface allows players to make contributions to computationally difficult optimization problems. Therefore, the question addressed by the Quantum Moves project is not if the general public can contribute through playing games, but which scientific domains have problems suitable for such a gamified approach. Our effort with Quantum Moves was the first major effort outside the domain of molecular biology. This made the efforts of our players an important first step to investigating this question within quantum physics.
Reason 2: This work inspired others and allowed us to move forward with designing better quantum games
The fact that quantum scientific challenges could be gamified and the general public interested in participating, provided strong encouragement for scientists to try similar methods on other quantum problems. For instance, in 2018 an international team of researchers launched a video game allowing 100,000 players to provide live input to 12 quantum experiments over five continents. Similarly, IBM has launched a comprehensive effort to develop quantum games that interact directly with their quantum computers. Finally, the group of Prof. Sabrina Maniscalco in Finland has launched an ambitious effort to develop a Quantum Black Box allowing “game developers to create citizen science projects helping topical research questions without a need to understand physics behind it”.
In our research group, we have pursued a number of other quantum research games. In 2017 we developed a gamified interface allowing the opening up of our quantum experiment in Aarhus to remote optimization by both the general public and fellow researchers. In our 2018 publication of the results, we reflected critically on the question: “Why could players without any formal training in experimental physics manage to find surprisingly good solutions?” and speculated that the answer could be that “the players may be performing better not because they have superior skills, but because the interface they are using makes another kind of exploration "the obvious thing to try out" compared to the traditional experimental control interface”. From this point of view, much of the challenge is actually solved in the process of creating the right kind of interface allowing this change in perspective by the players. This explains why, perhaps, the people who learned the most from our efforts were our quantum researchers who worked so hard with the game developers to iteratively design the interface.
Reason 3: We learned from it
The work with the original Quantum Moves left us with a number of observations. Most importantly, i) the core scientific challenge was not computationally complex, ii) the design of the game did not steer the players to explore the most difficult part of the problems, and iii) it seemed inefficient and disengaging that players first provided their solutions, which were then subsequently optimized algorithmically by our science team. In response to this, we created Quantum Moves 2: Crowdsourcing human common sense for quantum control, which has been live since December 2018 (allowing for both benchmarking against and collaboration with new optimization algorithms independently verified to match state-of-the-art performance).
In a recent research paper we pose five important questions:
- Q1: Can a suitable gamified interface allow citizen scientists to solve quantum control problems entirely on their own, using their own hardware to run the required computation (e.g. using an in-game optimizer), with a quality on par with traditional expert-driven optimization?
- Q2: Can the player-generated solutions, in combination with algorithms, provide an edge against purely algorithmic solutions?
- Q3: Could the citizen science games be first steps towards playful, explorative tools for domain experts, also in quantum physics? (as is being currently spearheaded within molecular biology)
- Q4: If the games are sufficiently challenging for humans, can they be used for systematic studies of human problem-solving?
- Q5: If larger portions of the player base can, using a gamified interface, make non-trivial contributions to several classes of research problems, could this potentially contribute to the solution of one of the major roadblocks in the path to domain-general AI according to Prof. Gary Marcus, author of Rebooting AI: the crowd-sourcing and algorithmization of human common sense?
Our recent and ongoing work with Quantum Moves 2 appears to confirm Q1 and gives ground for cautious optimism on Q2, whereas Q3-Q5 are still entirely open research questions to be pursued by our research teams and our many international colleagues from many disciplines. We hope you will join us on this exciting journey in the coming months and years ahead :)