DeepMind’s new AI coding engine scores among top human programmers • Slash

The field of autonomous coding has seen a major development recently with the creation of the AlphaCode system by DeepMind AI research lab. The system works on a model based on the Transformers encoder-decoder architecture, and it has proven its ability to solve complicated programming challenges usually dealt with by human coders.

Table of Contents

Training and testing

Before reaching that stage, AlphaCode had undergone unsupervised and supervised learning phases. First, the deep learning system was given data extracted from GitHub and taught to predict missing parts of a language or code piece. Then, the DeepMind researchers used the lab’s annotated dataset of problem statements and test cases, Code Contests, on which AlphaCode was trained to transform the description of the challenges into source code.

After that, DeepMind considered the system to be ready for a competition with human coders.

The competitions, in which AlphaCode featured, were curated by Codeforces. Coders participate in weekly challenges on the platform, and based on the results, the platform ranks those most successful. The challenges are comprehensive as they combine logic, math, and coding expertise. The coders need to be knowledgeable in theoretical concepts and algorithms in computer science to be able to take on the Codeforces problems. For AlphaCode, another layer of difficulty was that it had to solve problems it has never seen before.

No alterations were made to the challenges for AlphaCode – the 10 tasks were the same that the 5,000human developers got. In a wholly automated process, the AI system completed the steps that homo sapiens coders would, i.e. ran the code and check the output to select possible answers. According to DeepMind, AlphaCode generates thousands of solutions and then filters them with a clustering algorithm that puts the working solutions in the same group. That way, it was easier for the system to make the selection.

The result was impressive: in addition to securing a rank among 54% top human coders, AlphaCode also found itself among around 30% of the platform’s best users of the six-month period preceding the test.

The challenges of AI coding systems

This result has encouraged DeepMind, whose aim is to create an AI program that can solve problems currently dealt with exclusively by human coders. Currently, the project is still in its early day. The challenges it worked with during the testing fall in the domain of competitive programming, which is not really the sort of problems that human coders usually solve. Besides, the problem statements AlphaCode can work with need to be well-formulated and explicit, otherwise the system will not be able to solve them problems. Another factor is the number of samples the system needs to be able to generate code – 1 million to get around 35% of a 20-line program correct.

Other projects like AlphaCode are in a similar position. Microsoft and OpenAI’s GPT-3 autocomplete program for code and GitHub’s Copilot AI programming tool cannot replace human coders. In addition, there are some specific problems with the use of these programs, such as bugs and security vulnerabilities.

Getting buggy code from a program that is supposed to make your job as a coder easier is irksome, plus sometimes these programs use copyrighted code, because they are trained of public sources. The security challenge is far more serious, of course. If the program does not fully understand the context and the purpose of the code, for example, it will produce problematic code. Vulnerabilities that occur in the code due to the program’s lack of capacity to prevent them are bad enough, but a worse possibility is that someone can share code with hidden backdoors on purpose, knowing that AI tools will transport those to new programs.

Looking ahead

At this stage of development, despite learning fast, AI programming tools remain helpers of human coders, not trusted yet to substitute them. As such, these systems can be very useful in terms of increasing the productivity of programmers. The time that they spend now on generating code can be allocated to less mundane work, such as checking the code and formulating problems.