GitHub Copilot is the original AI-powered coding assistant and has a lot of publicity due to GitHub’s phenomenal distribution channel to developers (via GitHub itself!). That is why every time we get on a call with a potential enterprise customer, we expect to get questions on GitHub Copilot for Business and how we differentiate Codeium for Enterprises. While we love talking about the differentiators (just see our content here, here, and here as examples), we have noticed that there are also a number of misconceptions on GitHub Copilot for Business that we end up spending a lot of time clearing up. In this post, we explicitly discuss what an enterprise does and does not get from GitHub Copilot for Business, using a combination of material from GitHub, external discourse, and our own investigation on these tools.

Security & Legal

Let us start with the basics that any enterprise would consider as table stakes before even assessing quality and functionality. Security refers to the safety of your code that is used by the tool to function while legal more refers to the safety of the code that is generated that you are bringing into your codebase.

Do: Zero Day Retention on Code Snippets, Encryption in Transit

GitHub Copilot for Business is a SaaS tool, which means that relevant snippets of your code are sent externally to GitHub/OpenAI servers to perform each model inference. They do guarantee zero day retention on the code that is sent over, which means they promise to delete the code and therefore are unable to use your code for other purposes, such as training models that can be used by others. And as far as we can tell, your code is encrypted in transit to their servers. This is similar to all other enterprise SaaS AI-powered code assistants, such as Codeium Teams. Of course, an even stricter security guarantee would come from a self-hosted solution where there is no IP going out, which is a key differentiator for offerings such as Codeium for Enterprises.

Don’t: Third Party Audits

While GitHub does claim zero day retention and other security features, GitHub Copilot has NOT been audited by a third party to verify and test these claims. So while GitHub’s core product is SOC2 compliant (among others), GitHub Copilot explicitly is not, and there haven’t been any external penetration tests conducted. This is very different from tools such as Codeium Teams or Tabnine, which are SOC2 compliant. GitHub Copilot will NOT have this compliance for a while according to their own trust center FAQ:

Audits and Certifications: GitHub Copilot is not currently included in GitHub’s existing audits and certifications, including SOC 2, ISO 27001, and FedRAMP Tailored. Compliance at GitHub begins with good security, so our first focus is fully onboarding Copilot to GitHub security programs and tooling. GitHub is engaging with a third-party audit firm to perform a gap assessment of Copilot as part of readiness activities for SOC 2 Type 1 (security criteria) and ISO 27001, with a goal of having the full audits completed by May 2024.

External Penetration Test: GitHub has not yet performed an external penetration test of Copilot. GitHub plans to include Copilot for Business penetration testing in the next cycle, which is in the 2nd half of 2023, with the report being issued in early 2024

Don’t: Safety from Non-Permissive Licenses

GitHub Copilot trains on non-permissively licensed public code, including GPL, LGPL, and AGPL licensed code. We have done an in-depth analysis on how this means that GitHub Copilot can pretty easily regurgitate such code, putting your firm at legal risk for violating copyright law. GitHub Copilot has built filters and recently even an attribution tool to counter such concerns, but as our analysis showed, these post-generation recognition tools do not actually work to catch violations (while simultaneously massively degrading the performance of the tool). The reality is doing anything post-generation is quite difficult, and it is obvious that GitHub Copilot’s naive exact string matching approach will not catch very common edits such as minor variable renaming or argument reordering. These minor differences will not be considered as differing sufficiently from the source material in any legal case. This is quite different from tools such as Codeium and Tabnine, who have proactively removed non-permissively licensed code from the training data, making it incredibly hard to generate anything remotely close to non-permissively licensed code.

Don’t: Indemnity in Case of License Violation

This one builds off the previous point. We commonly hear that companies are aware of the non-permissive license issue and the reality that the filters don’t really work, but believe that GitHub will provide complete indemnity in the case of a legal dispute over use of non-permissively licensed code as long as you have the filters turned on. After all, their terms and conditions state pretty explicitly:

If your Agreement provides for the defense of third party claims, that provision will apply to your use of GitHub Copilot. Notwithstanding any other language in your Agreement, any GitHub defense obligations related to your use of GitHub Copilot do not apply if (i) the claim is based on Code that differs from a Suggestion provided by GitHub Copilot, or (ii) you have not set the Duplicate Detection filtering feature available in GitHub Copilot to its “Block” setting.

However, the devil is in the details. This same condition states that "any GitHub defense obligations related to your use of GitHub Copilot do not apply if (i) the claim is based on Code that differs from a Suggestion provided by GitHub Copilot." This means that even if all filtering features are enabled, the defense obligations do not apply if the Customer makes any changes to the Suggestions. There could be any reason for editing a suggestion, from fixing a minor bug to aesthetics. The underlying GitHub customer agreement also limits defense to GitHub products “not combined with anything else.” In case of a lawsuit, GitHub would absolutely argue that the incorporation of a Suggestion into a customer’s code would be both a “change” and a combination. Finally, the Copilot terms explicitly say: “You retain all responsibility for your code, including Suggestions you include in Your Code.” So taken together, what GitHub is really saying (albeit in a slightly tricky legal manner) is that "if you enable all filtering features, we’ll defend you against a claim that our unmodified suggestions, used in a vacuum and not incorporated into your own code base, infringe licenses.” This is obviously impractical, and Microsoft very much knows this - given the far reaching implications of such legal cases (if they do arise), even a company like Microsoft/GitHub will not want to be held liable for suits brought up against individual companies.

Pretty much all tools have this same conditional indemnity, so you only truly get protection if the tool does not train on non-permissively licensed code to begin with.

Product Quality

We often find a lot of confusion on the actual internals of the GitHub Copilot product that determine how useful it would be to a developer. These are not enterprise specific, but obviously an important consideration for enterprises trying to choose a tool.

Do: Leading Autocomplete code model, Chat soon

In terms of model size, number of tokens trained on, and active capabilities such as fill in the middle, GitHub Copilot’s autocomplete model has a lot of very positive attributes that most competitors’ autocomplete models cannot compare to. In addition, over four months since announcing chat as part of Copilot X, GitHub Copilot is finally rolling out their in-IDE chat interface to Enterprise customers. While this is months later than other tools such as Codeium, we expect such a tool to help developers in more exploratory tasks, which will complement the acceleration provided by the autocomplete system.

Don’t: Large Context Length

While a lot of aspects of GitHub Copilot’s autocomplete model are great, the one (albeit important) aspect where it lacks is the context length, i.e. the maximum amount of code can be passed into the autocomplete model to generate a suggestion. While extremely large context lengths may not actually be beneficial, GitHub Copilot’s context length is only 2048 tokens, or 150-200 lines of code. That is a very limited amount of context that can be passed in for an inference, which is why Copilot often starts degrading in larger files and codebases where the relevant snippets might be farther apart. A tool like Codeium has a context length of 4096 tokens, double that of Copilot’s.

Do: Good Latency

We have found many companies overlook this basic aspect, getting caught up in suggestion quality that they forget for a tool to be useful, it needs to be fast, especially for autocomplete, where if it takes even over a second to make a suggestion, it is useless. This latency is where tools such as Sourcegraph Cody, which focuses on codebase awareness, completely break down, but GitHub Copilot does not have that problem! Like Codeium, GitHub Copilot has invested time in a powerful in-house inference engine to make suggestions come within a latency constraint that makes sense for developers.

Don’t: Updated Training Data

GitHub Copilot is using a model that is now over two years old, and the model itself has even been deprecated as an OpenAI API. The biggest concern here is that this means that GitHub Copilot’s training data does not contain the latest and greatest best practices, libraries/packages, and more. For example, in this post, we realized that GitHub Copilot probably has no insight into the very popular, but recent, Langchain open-source repository at all. In addition, the training data contains a lot of open source code that has since been found to contain security vulnerabilities, placing more risk on utilizing the generated data. To compare, at Codeium we retrain our model every one to two months.

Don’t: Codebase Awareness

This ties a bit into the earlier point on context length, but besides not being able to put much context into an inference, GitHub Copilot also does not have a very powerful engine to decide what parts of the overall codebase are the most relevant to shove into that limited context. More advanced parts of their context retrieval engine may also require your code to be on the GitHub SaaS product (for core source code management). This context awareness is something we dived deeply into in one of our blog posts. Also, without advanced codebase understanding, GitHub Copilot will not be able to offer AI powered code search and navigation capabilities, which tools such as Codeium and Sourcegraph Cody already provide.

Don’t: Fast Velocity and Future Potential

There should be no argument that GitHub Copilot is a trendsetter in the AI-powered software tooling space. However, the reality is that the product itself has been developed jointly by two separate entities - OpenAI for the model layer and GitHub for the product layer. Unfortunately, these two entities have different incentives and goals, and they don’t always prioritize making GitHub Copilot better. This is why the model has rarely been updated over the last few years or why Copilot is now lagging behind tools like Codeium, which started later, in new functionalities such as Chat and Search. In a fast moving space such as generative AI, it is the rate of development that will determine the long-term best products, not the capabilities at any individual near-term snapshot.

Enterprise-Specific Capabilities

There are also a bunch of things that an enterprise often expects from third-party dev tooling, from analytics to support to integrations with existing systems, so it is important to walk through what you get from GitHub Copilot along these axes.

Don’t: Real-time Analytics

Analytics are incredibly important for any tooling so that an enterprise customer can be confident that they are getting the expected value out of a material financial commitment. GitHub Copilot does collect usage statistics (unclear whether usage statistics are utilized in improving the generic system, such as in preference modeling, but that’s another topic). However, your access to these statistics is limited by requiring a request to your account executive on GitHub’s side. If you are a large enough customer, they will give you a screenshot of an internal dashboard for your team’s usage.

While puzzling, GitHub Copilot has decided so far to not provide a real-time view into the analytics to administrators, which would reduce the current burden and activation energy to go out to the account executive every time the customer wanted to see statistics. Tools such as Codeium provide real-time analytics dashboards with breakdowns across user, time, languages, and more, so that administrators can take action immediately to improve usage of the tool. For example, we often find that different developers need different levels of education to make the best use of this new AI-powered tooling, and so an administrator being able to see a developer not using the product in a few days is a good signal to go and engage that developer.

Do: Self-Serve

You can get started with GitHub Copilot for Business with a few clicks of a button and a credit card, which is very simple. Of course this also means you might not have a dedicated account executive to ask for analytics, but it is a SaaS tool, and SaaS tools are easy to spin up.

Don’t: Training and Support

From a support point of view, GitHub Copilot for Business at this moment is relatively barebones out of the box. One would expect training, live or pre-recorded, and responsive customer support in case of issues, but neither are guaranteed with GitHub Copilot. While larger organizations may receive such support, generally companies are left to rely on fragmented online discourse to figure out how to best use the tool.

Don’t: Agnostic Integrations with Your Existing Systems

It is probably obvious, but GitHub Copilot for Business will be biased towards working well with GitHub, the source code management (SCM) tool. For example, getting access to the new release of GitHub Copilot Chat requires you to enable the settings in your GitHub policies tab, which exists only on GitHub. This of course is classic vendor lock-in, and so if you are using other SCMs such as Bitbucket or Gitlab, or just generally like optionality, this is something to keep in mind. There is also limited support, if any, for integrations with other existing systems, such as your Identity & Access Management (IAM) system, to support SSO onto the tool, which is provided by self-hosted solutions such as Codeium for Enterprises.

Final Thoughts

GitHub Copilot is a great tool, one that has paved the way for all AI-powered tooling for software developers. However, using it in an enterprise setting does come with a lot of caveats and gaps, from security and legal, current and future quality limitations, and simple enterprise tooling expectations. As long as you are aware of what you are considering purchasing for your company, you can make an informed decision on what solution will be best for you.