My Week With GitHub Copilot: AI Pair Programming Review

ProfilePicture of Rafael Goulart
Rafael Goulart
Senior Full-stack Developer
A developer using GitHub Copilot to help review code

Before text and code editors, developers were tasked with having the correct code passed to a compiler. Now, support tools like syntax highlighting, function syntax assistance, autolinting, and Artificial Intelligence (AI) assistance are available directly in your Integrated Development Environment (IDE) or the editor where you code. 

With these tools, developers can focus on building great software and spend less time completing tedious tasks or combing through code to discover minor errors. For example, ubiquitous code highlighting reminds developers to finish their strings. There’s no need to keep a programming language reference open in a browser tab now that developers can start the function name and the syntax parameter will pop up. If developers forget to close a bracket, the code formatting tool Prettier triggers a reminder to enforce a consistent code style. For those who use tabs or spaces for indentation, now whatever is set up on a project is automatically applied when the file is saved. Overall, these tools help developers reduce errors and improve the quality of their code.

Now, there’s a tool that promises to improve the developer experience even more: GithHub Copilot, an AI pair programmer. After a year of closed beta, GitHub Copilot is now publicly available to developers and is gaining a lot of attention for its ability to enable developers to harness AI while writing code. According to GitHub, developers can write a comment describing the logic they want and GitHub Copilot will automatically generate a code suggestion for their solution. It sounds pretty cool, right? Our team thought so, but we had some questions, for instance: 

OK, that last one might be a bit philosophical, but the others felt testable. So, to get some answers, I put GitHub Copilot to the test. In this article, I’ll share my experience after one week using the tool and dive into key findings from this trial period. I’ll also use my 20+ years of experience as a Senior Full-Stack Developer to compare it with my day-to-day before Copilot. Hopefully, my experience will give other developers a better understanding of how this tool works and if it lives up to the hype.

Table Of Contents

What Is GitHub Copilot?

To understand how Copilot came to be, it’s first important to understand pair programming.  

Pair programming is an extreme programming technique where two developers work together on one computer, one acting as the pilot and the other the copilot. The pilot is on the keyboard while the copilot helps navigate, checking the logic and patterns, while weighing against alternative solutions. While two developers producing one piece of code sounds expensive, the quality of the final work can help avoid added costs of code review and QA. 

It was from this idea that AI pair programming came to life. Officially launched in June 2022, GitHub Copilot is an AI pair programming editor that suggests code in real time to help developers work faster and more efficiently. It’s powered by a new AI system, OpenAI Codex, and translates natural language to coding prompts as developers write a comment or a line of code. While it’s free for students, it’s $10/month or $100/year for developers (with a 60-day trial).

Now, with that background covered, let’s dive into some key findings from my week with GitHub Copilot. I’ll run through the installation, testing and how it looks in action.

Testing GitHub Copilot with JavaScript and PHP in Visual Studio

Before we get started with the tool, let’s take a look at how compatible it is with different languages. According to GitHub Copilot’s official documentation, the tool is “optimized to help you write Python, JavaScript, TypeScript, Ruby, Go, C#, or C++. You can also use GitHub Copilot to generate suggestions in other languages and a wide variety of frameworks.”

For this test, we’re using a JavaScript front-end and a mixed PHP/JavaScript back-end. While GitHub Copilot isn’t optimized for PHP, it should be smart on the JavaScript/React front-end side.

It’s important to note that, for the sake of this project, l only learned the basic information needed to use the tool. I didn’t read the FAQ or any additional documentation from GitHub before testing it out on my own. Instead, I let the tool work and adhered to the suggested flow in order to observe its use without forcing it too much. This way, I was able to work as usual and evaluate Copilot, without biasing my conclusions about the tool.

GitHub Copilot Installation in Visual Studio Code

GitHub Copilot has support for many IDEs: documentation cites extensions for Visual Studio Code, Visual Studio, Neovim, and the JetBrains suite. I’m a Visual Studio Code (VSCode) user, so I used the VSCode extension for this project.

The installation consists of three main steps:

For an initial test, create a file with the extension for your programming language (like test.js for Javascript) and start typing:

function calculateDaysBetweenDates(begin, end) {

Screenshot of the first GitHub Copilot suggestion on Visual Studio Code
Screenshot of the first GitHub Copilot suggestion on Visual Studio Code

When you type this function declaration into your js file, you will see that GitHub Copilot starts to fill in what it believes to be the required implementation. The code that GitHub Copilot provides converts the begin and end inputs into dates, determines the days between the two dates, then returns the result. This initial test demonstrates how GitHub Copilot is able to help suggest and generate code. 

GitHub Copilot Shortcuts

I was expecting to get an overflow of information and techniques from GitHub Copilot’s documentation, but getting started is pretty straightforward: 

OSSee next suggestionSee previous suggestion
macOSOption (⌥) or Alt+]Option (⌥) or Alt+[

You can find everything we’ve covered up to this point in the GitHub Copilot’s Visual Studio Code documentation. That’s all you need to know to use the tool. Now, let’s get started.

Initial Tests 

To have a consistent testing environment, I set up two directories: one with plain JavaScript (no dependencies) and one with a Next.js application, so I could play around with React and an established React framework briefly. 

Plain JavaScript

In an empty .js file, I started typing to create a simple function to convert from hex color to rgb. This is a common feature – most of us know the concept of converting hex numbers to integers – but let’s be honest, most developers don’t have solutions memorized. And unless there’s a library in the system, we’ll search for the simplest solution, validate and move ahead.

Using Copilot, here’s how it can be achieved:

VSCode with GitHub Copilot suggesting a Javascript hex color to rgb function
A VSCode GIF animation of GitHub Copilot suggesting a Javascript hex color to rgb function

Cool, right? Notice that I ignored the first suggestion as it looked too complex. I was writing a basic test with a sample color and I only had to hit Tab.

React in Next.js

This is a more complex test, but let’s keep it simple for now. Here’s a test page, but nothing specific from Next.js:

VSCode with GitHub Copilot suggesting a simple React test page
A VSCode animation of GitHub Copilot suggesting a simple React test page

There’s a mix between GitHub Copilot and the IDE’s code completion. Do you notice a small delay to display the suggestion? This may be because it’s too vague, but it’s hard to tell. GitHub Copilot’s suggestion is okay, and it also offers an export default, but it isn’t needed in this case.

The code above is enough to test the environment. Out of curiosity, let’s see how it looks if we hit Ctrl + Enter:

VSCode with GitHub Copilot suggesting a simple React test page in a separated tab
A VSCode animation of GitHub Copilot suggesting a simple React test page in a separated tab

You can also use Alt + Tab, of course.

We’ve completed the dummy tests. Next, let’s see GitHub Copilot in action. 

Real-Life Action

My current project, where I’ll be using the tool, has Laravel on the back-end and React on the front-end. 

Starting with Back-end Code 

I only had back-end work on my first day using Copilot, so my expectations were low (no optimization for PHP, remember?)

Note that the tasks were related to the Laravel framework: a modern PHP framework and VSCode with some very established extensions that are able to complete “use” statements and pop up PHP function parameters. Prettier, the code formatting tool I mentioned earlier, alerts me about silly syntax mistakes at saving. In my day-to-day, I purposely avoid indenting code; that way,  if Prettier doesn’t auto-adjust for me at save, I know it’s not working and I have something to fix.

Complete the Line

The first effect is “filling in the blanks” suggestions. This is similar to Google’s auto-suggest function, where typing in Gmail or Google Docs will recommend words or phrases to finish a  sentence for you. Copilot shows obvious completions, like end-of-line semicolons or closing brackets/parentheses. 

Verdict: GitHub Copilot is helpful in this case, but not impressive. To compare, Prettier does the same at saving, while also automatically indenting your code.

Variable Filling

When you use a function, GitHub Copilot suggests what variable to fill in depending on your context.

A VSCode screenshot of GitHub Copilot variable suggestion
A VSCode screenshot of GitHub Copilot variable suggestion

This suggestion is 99% of the way there in my context, but the correct answer is:

A VSCode screenshot of the correct answer for variable suggestion
A VSCode screenshot of the correct answer for variable suggestion

Verdict: Okay, GitHub Copilot, you won this one. 

Message Suggestion

This is a common pattern. You make an early condition to check user access and throw an exception returning a message:

A VSCode screenshot of GitHub Copilot suggesting an exception message
A VSCode screenshot of GitHub Copilot suggesting an exception message

This doesn’t come from external code, but it used my context to suggest an especially useful message. So GitHub Copilot not only knows I need to fill a message, but also an authorization message that should be inferred by the previous code. 

Verdict: For those not used to Laravel, it’s a pretty common pattern to get the current user. So this is amazing functionality!

Library Discovery

I was working with the Cloudinary API through their PHP library. The code is supposed to be different depending on the mimetype. Then I get:

A VSCode screenshot of GitHub Copilot suggesting a Cloudinary-related code
A VSCode screenshot of GitHub Copilot suggesting a Cloudinary related code

The case syntax isn’t impressive as any IDE snippets plugin can do this for any language. But, the mime type goes deeper: it considers the function name to infer the related mime type (video/mp4). The cloudinary_php suggestion looks impressive, but it’s not correct. I could actually write a code like this in some situations, but the solution I need is very different. And so I still have to check the API reference to get my final solution.

Verdict: Developers won this round. Sorry, Copilot.

Model/Database Suggestion

In this example, there’s a switch syntax building a custom filter for the Booking model in Laravel. After pressing Enter, Copilot suggests a real field. But wait, this isn’t on the active file!

A VSCode screenshot of GitHub Copilot suggesting a Laravel model field on a switch control structure
A VSCode screenshot of GitHub Copilot suggesting a Laravel model field on a switch control structure 

When I accept by hitting Tab I get:

A VSCode screenshot of GitHub Copilot suggesting content of switch case entry considering Laravel database builder
A VSCode screenshot of GitHub Copilot suggesting content of switch case entry considering Laravel database builder

This is pretty much the answer, except $value[‘id’] should be $value.

Verdict: It’s not 100% correct, but it saves time because I need to type less and only requires a minor tweak. It’s pretty useful.

Not Optimized for PHP?

GitHub states that PHP is not one of Copilot’s strengths, but my expectation for JavaScript is high considering the tool is optimized for it. Let’s see how it performs. 

Front-end and the Optimization in Action

Finally, I had some front-end work in React that I could use Copilot for. I did some initial tests in React that looked promising, but let’s see how it performs in my real workflow.

For this, I was working on a hook, returning an object to render row actions in a table. I needed a permission check to restrict actions depending on the user’s context. However, for now, I just need to define permissions for each action as processing is done in another function.

A VSCode animation of GitHub Copilot suggesting permissions considering the object context

The first suggestion (modify) was smart (based on the label attribute above), but far from what I need. I have to type the correct value, but then on the Delete object the magic happens: it is 100% correct. Copilot was able to suggest based on my previous typing or the current context. Cool!

Next, let’s move to a React component. This was already in place, but I needed to fill propTypes. To explain simply, the image displays the top of the file and the typing area: 

A VSCode animation of GitHub Copilot suggesting React propTypes with shaped type
A VSCode animation of GitHub Copilot suggesting React propTypes

The types are correct. Copilot probably got them from their use in the file. But, I’d like to be more specific and shape the arrays.

A VSCode animation of GitHub Copilot suggesting React propTypes with shaped type
A VSCode animation of GitHub Copilot suggesting React propTypes with shaped types

Copilot got the context from a related file (CountryTimezoneField.js tab, where the variables were declared) and made perfect suggestions, including a shaped type inside a shaped type. The context is wider and more accurate when related files are opened in other tabs.

This is a tedious task. Usually, we split the screen vertically to have both files side by side and copy/paste or type each attribute. But not anymore it seems.

Here’s a final example with more of the context power:

A VSCode animation of GitHub Copilot suggesting MaterialUI Grid and React conditional display pattern.

There’s no need to know the library, but the MaterialUI Grid pattern can be inferred from the context. In the sequence, Copilot suggests an unused imported component. This isn’t correct as the file isn’t open to get the variables, so I ignored it. But once I began typing showProgress, Copilot suggested the expected pattern. Hit Tab and Tab again, and it’s done.

AI GitHub Copilot vs Search Engine Results

As developers, we do a lot of searching to find solutions, get language or API references, and learn from recipes or libraries/frameworks/tools documentation. This is time consuming and, in some cases, inevitable because we need to learn new concepts, understand how a tool works, or read a manual, for example.

So how does Copilot compare to using search engines? Let’s find out by testing how search engines answer the previous “convert hex color to rgb” function:

Conducting a browser search is a lot of tedious steps for developers. They have to:

Some search engines display the “best match result” in a sidebar, generally from a popular question-and-answer forum. In contrast, with GitHub Copilot there’s no need to switch from the IDE to the browser,  alternatives can be checked with a simple shortcut or in a separate tab, and the context is considered in suggestions.

Verdict: There’s a huge time saving in these cases. That’s another point for GitHub Copilot!

Main Concerns About GitHub Copilot

AI works using trained data. As such, any AI needs a large amount of data to be effective, and OpenAI Codex is trained over GitHub public repositories. This brings not only ethical but also legal questions. 

For instance, Free Software Foundation (FSF) claims that GitHub Copilot is “unacceptable and unjust” because it requires paying for the software and is a Service as a Software substitute. The FSF’s white papers raise questions around GitHub Copilot, like:

The FSF white papers dive deeper into these questions and consider other valid concerns about AI tools like Copilot. It’s worth a read.

Although, as developers, we should consider that we research and borrow solutions all the time to complete daily tasks. Are developers always careful about code licensing? Is it fair use to grab an hex2rgb() function from a famous or obscure forum anywhere in the world? Are we putting our jobs in danger by doing that? Many of the FSF questions apply to a non-AI world as well.

Equally as important is the question of privacy of code. Will code that we write be used to train the AI model? Will we see the money we spent on solutions being shared with a competitor? According to GitHub’s privacy statement, the answer is no. However, it’s up to you to trust this statement or not.

Another valid concern is how the tool affects students. CS Teachers should take note if the class grades suddenly rise. Emery Berger, a professor of computer science at the University of Massachusetts Amherst, warns teachers that GitHub Copilot may be perfect for cheating CompSci programming exercises, but fighting against it might be useless. Unlike copy/pasting programming, GitHub Copilot “makes new solutions that are superficially different enough that they plausibly could have come from a student”, he says. 

Alternatives to GitHub Copilot

Are there any worthy alternatives to GitHub Copilot? 

The short answer: at the time of this writing, the only alternatives are Tabnine and Amazon CodeWhisperer (the latter just launched a preview).

The long answer: OpenAI Codex started as an open-source project that would allow, in theory, many tools to use its base to create various alternatives. But this is not a cheap enterprise! They recently moved to a more traditional closed business model and received a strong investment from Microsoft, which owns GitHub. 

Some open-source projects started using the remaining open-source training model, but most are stalled. Unless they were to receive funding, it would be pretty hard to achieve the same level of efficiency GitHub Copilot offers now. The latest version of its model is now available for licensing, preferably by Microsoft. As other companies can license OpenAI Codex, it’s possible we will have more alternatives in the future.

Tabnine aims to provide a similar product and already has a customer base (they claim to have more than 1 million developers using their product). They stand for using only open-source code with permissive licenses for their Open Source Trained AI model (MIT, Apache 2.0, BSD-2-Clause, BSD-3-Clause) from GitHub repositories or customer’s repositories/local code. You can also download the model to your machine and get even faster responses. Their Pro subscription comes in at $12 US per month. 

In June 2022, Amazon launched a preview of CodeWhisperer, another tool similar to GitHub Copilot. This seems to be a legitimate competitor, but there’s not enough information to compare them so far.

It’s likely that we’ll have more competitives to evaluate in the future. For now, we’ll say it’s “to be continued…”

Addendum: FauxPilot is an open source project that aims to run a model locally, using a Nvidia GPU to process the data, and VSCode GitHub Copilot plugin itself.  

Key Takeaways from My Week with GitHub Copilot

GitHub Copilot certainly has its strengths and weaknesses. Here’s a roundup of key takeaways from my trial period with the tool:

Fewer Browser Searches

With GitHub Copilot, I don’t need to switch to the browser to search as often. Common functional solutions are available after a few words + Tab, and established patterns used in the most used frameworks/libraries are there, too. It’s a great time-saver.

Context Matters

My expectation of how Copilot should work was wrong. At first, I thought it was an “ask me something and I return some options” kind of flow. But even if I don’t ask anything, I still get impressive suggestions in the context of my work.

“Context” is key here. The AI considers factors like the frameworks or libraries in use, the variables in the current file, and related files and scope in order to produce better suggestions. Patterns are also considered in part of the answer as well. When you embrace the technology and understand its strengths and weaknesses, you start to get more out of it without changing your flow.

It’s Best for Experienced Developers

Although AI assistance might look like a good learning tool, it’s not. Code suggestions are just that: suggestions. The developer needs to be able to evaluate these suggestions and choose to accept, skip, or ignore them. In order to use the tool properly, it’s necessary to know the context, logic, and expected output to make a smart (and fast) decision. Otherwise, the suggestion can create errors.

GitHub Copilot actually confirms this insight. The GitHub Copilot FAQ entry Fairness and broader impact states that “inexperienced developers may struggle to use GitHub Copilot to effectively generate code, and their lack of experience might inhibit their capability to effectively review and edit suggestions made by GitHub Copilot.”

Verdict: It’s Worth the Money

From my time using the tool, I concluded that yes, GitHub Copilot is worth the cost, and here’s why: 

Given the rates most software developers are making these days, I think that if this tool saves you even one minute per day, the $10 monthly fee easily pays for itself. In reality, I think you save a lot more time than that, and what you get out of the tool increases with use (you get used to learning how to get better answers). Sounds like a good deal to me.

Final conclusions

Developers aren’t encyclopedias. Even recent solutions we’ve worked on aren’t always readily available in our brains. It takes a lot of searching and research for small solutions, like a for/foreach pattern ten-line function created five years ago for a dead project. The majority of problems that have to be solved every day have already been solved many times over and are available as question-and-answer forums or public code from open source projects. 

An AI-powered tool like GitHub Copilot can speed up daily work and bring so much knowledge to your fingertips in your IDE, but it won’t do the job for you. At the end of the day, developers are still responsible for delivering the work even if they use AI assistance. With that said, advanced tools like this won’t make you less of a developer, but they will improve your productivity.

For decades, companies have been trying to automate coding. And, although many common programming activities were automated, abstracted or simplified, real people still need to do the work. Maybe in the future we’ll need another kind of programming, but at least for now, there still needs to be someone’s actual brain in action to solve humanity’s problems.

Final tips for using GitHub Copilot:

And remember: GitHub Copilot is AI. Ultimately, you’re in charge and responsible for the end work. 

Originally published on Sep 23, 2022Last updated on Mar 9, 2023

Looking to hire?

Join our newsletter

Join thousands of subscribers already getting our original articles about software design and development. You will not receive any spam, just great content once a month.


Read Next

Browse Our Blog