GitHub Copilot is now generally available to all developers. GitHub surveyed 2,000 Copilot users in the US to find out how AI support affects developer productivity.
The core thesis behind GitHub-Copilot is: AI can help developers be more productive and happier while they code. GitHub wanted to test this thesis and find out whether their copilot actually leads to higher productivity. For this purpose, survey data was collected from more than 2,000 developers in the USA and combined with anonymized data. According to GitHub, further studies on GitHub Copilot will follow soon.
The word “productivity” has a wide range of possible practical meanings in development. Ideally, do developers want to save on keystrokes or avoid searches on Google and StackOverflow? Should GitHub Copilot help them stay in the flow by providing them with high-precision solutions for mechanical tasks? Or should he inspire them with speculative stubs that can help them when they get stuck?
In total, more than 2,000 developers in the US were asked the following three questions and their answers compared to user data from the same period.
- Do people feel that GitHub Copilot makes them more productive?
- Is this feeling reflected in objective usage measurements?
- What usage metrics best reflect that sentiment?
The results: Using GitHub Copilot correlates with improved developer productivity. Developers reporting the highest productivity gains using GitHub Copilot also accept most of the code suggestions presented. When users were divided into different quartiles based on how useful they found GitHub Copilot, there was one notable difference between these groups: the completion acceptance rate was much higher among those who reported the greatest productivity gains. Also, the interviewed developers didn’t really care if they needed to revise the proposal, as long as GitHub Copilot gave them a good starting point.
GitHub: “Our Copilot isn’t designed to help you build software yourself. It’s designed to provide helpful suggestions that make it easier to stay in the flow. In other words, GitHub Copilot gives developers the pieces, but lets them do the finished assembling and designing the product.”