How AI Raised Our Efficiency by 27%
We’ve been using multiple AI tools since their inception. We subscribed to ChatGPT and Midjourney and found them extremely useful in many scenarios. While ChatGPT proved its capabilities as a general writing assistant it wasn’t that accurate and up-to-date in coding. Midjourney turned out to be the best generator of illustration images - just check the one generated for this post. Now I’m writing about Github Copilot, the "AI pair programmer". Our initial goal was to experiment and evaluate to learn what’s the real value behind the hype.
We’ve been using Copilot since February 2023 when we enabled it in our whole Github organization. Before that, we had only a few developers who tried it, but from then it became a default assistant for writing code more efficiently for the whole development team. It’s integrated into the developer environments. First, it feels like a code autocompletion tool on steroids, but it can do more. It can write full comment blocks based on code and vice versa. We still don’t find it useful to generate complete apps or classes, but on function level it is very helpful. As ChatGPT also evolves we check back from time to time and we use it for more conceptual questions, for example, it is great to generate boilerplate code tailored to the actual requirements. Then for the real implementation level, we still rely on Copilot.
We regularly discuss our experiences using AI and recently I asked our developers the following question:
“What’s your perceived increase in efficiency thanks to Copilot?”
When I first asked this question, the average response was less than 10%.
This time, responses varied between 20% and 40%, with an average of 27%.
That means it more than doubled during the last 10 months!
What are we using Copilot for?
Mostly for Drupal development and for some frontend apps in React/Nextjs and Vue/Nuxtjs.
- PHP
- JavaScript
- Twig
- CSS, SASS
What are the IDEs we use?
- PHPstorm
- VS Code
- Neovim
Some feedback from our team members
I use it for almost everything: TWIG, CSS, SASS, JS. It also helps in readme and markdown. I haven't turned it off anywhere, it tries to help everywhere.
It writes comments and texts very well.
It can write very good code from comments.
I noticed that it learns when it is turned on.
It can deduce very well what I might want from code context.
I only wrote the comment for a function with a date display, and I didn't even have anything to code in my head, and it was coded quite well.
Notes and frills that are required for the coding standard are written, they just need to be improved a little. The documentation above the dependents is written very well.
It has improved a lot in the past six months, it says much less nonsense.
Final thoughts
We know that many people think AI is overrated and unfortunately, we don’t have an objective methodology for measuring efficiency. For now, we have to be satisfied with these subjective, perceived numbers. I’m already happy even if it was mostly the DX that has been improved by 27% and not the real coding efficiency. But I believe it’s much more than the DX when it comes to enabling developers to focus on more complex problems and let the Copilot do the less creative work for them.
It is doing already more than that by trying to show alternative solutions and figuring out what the developer was about to write, sometimes surprisingly well.
What’s even more fascinating is the growth in efficiency. AI is expected to deliver exponential growth. I don’t know where we are on the curve, but the growth is already impressive and we will try to measure and get back on this topic. We will also continue experimenting with alternatives as not only ChatGPT is evolving constantly, but Google is pushing hard to catch up and even JetBrains introduced its own ai assistant available in PHPStorm.
