Tabnine AI coding assistant flexes its models

Tabnine, the “OG” AI-powered coding assistant, offers several powerful features that improve code generation and code generation, including context-aware suggestions, a chat window with a strong array of selectable AI models, and personalization of its models. The Tabnine Protected model supports about 15 popular programming languages at the “excellent” or “good” level, and another 65 or so languages and frameworks at varying levels of support. Tabnine Protected 2 (the latest update, which dropped as I was writing this review) supports more than 600 programming languages and frameworks. Tabnine expects its prompts to be in English, although other languages may work.

The use cases for Tabnine cover the full software development life cycle (SDLC), but currently lack any support for the command-line interface (CLI). Tabnine answers common developer questions and requests, such as “Where in our code base do we …,” “Write a unit test for this code,” “Generate documentation for this function,” and “Explain what this code does.” Its capabilities include generating code from plain language, onboarding developers to new code bases, autonomous generation of tests and documentation, code refactoring, and AI-generated fixes.

Tabnine competes directly with GitHub Copilot, JetBrains AI Assistant, Sourcegraph Cody, and Amazon Q Developer, and indirectly competes with a number of large language models (LLMs) and small language models (SLMs) that know about code, such as Code Llama, StarCoder, Bard/Gemini Pro, OpenAI Codex, and Mistral Codestral. Because Tabnine currently allows you to choose among seven good AI models for its chat window, it’s hard to take its indirect competition very seriously.

Deploying and installing Tabnine

You can either use Tabnine’s normal SaaS services, or deploy it yourself. If you choose the latter, you can deploy it in a VPC, or on-premises. Tabnine’s deployment partners are currently AWS, DigitalOcean, Google, NVidia, Oracle, and VMware. If you deploy privately, you can still optionally pull updates from a Tabnine update server. You have the option to be fully air-gapped if you wish.

Client installation depends on which IDE you choose. No matter which IDE you’ll use, you need to sign up with Tabnine or join a team following the instructions you receive from your administrator. Then you’ll have to install the plugin for your Visual Studio Code, Visual Studio, JetBrains, or Eclipse IDE.

That will give you two interacting facilities: inline code completions in your editor, and Tabnine chat in an IDE window. If you need low-level help, up to and including implementing whole functions based on inline comments, use code completions. If you need high-level help, for example high-level design, general questions about code, or to understand instructions, use the chat facility. To correct errors, first highlight the problem in the code and then invoke the chat for a fix, either using context-menu commands in the code pane, slash commands in the chat, or natural-language requests in the chat.

These facilities are competitive with their analogs in GitHub Copilot and Amazon Q Developer. One difference for the higher subscription categories in Tabnine is that you can choose the model you want to use for each request to the chat.

Protected and private models

Tabnine’s proprietary models—Tabnine Universal for code completions and Tabnine Protect for chat—are “private and protected.” According to Tabnine, these models balance high performance with guaranteed privacy and maximum protection from legal liability due to misuse of licensed code. Because these models have been trained exclusively on permissively licensed code, they guarantee protection against legal liabilities. “With this approach, there is no requirement for post-generation filtering for potential IP violations,” the company says.

For enterprise customers, Tabnine offers a model that runs in complete isolation inside the customer’s private installation, and provides indemnification against intellectual property infringement. And for SaaS users, Tabnine promises zero data retention. The company says that any information sent to its inference servers is “encrypted in transit, runs only in memory, and is deleted after delivering the response.”

Tabnine Protected is one of eight models currently available for Tabnine Chat. The other seven models offer similar performance but have varying levels of privacy and lack IP violation protection. The Tabnine + Mistral model offers zero data retention but no IP protection; Claude 3.5 Sonnet, Codestral, Command R, GPT-3.5 Turbo, GPT-4 Turbo, and GPT-4o all lack data privacy guarantees and IP violation protection, as they send code and requests back to their own servers (a possible data privacy risk), and were trained on code with unspecified licenses (a possible IP violation risk).

In practice, some companies, particularly those involved in government work, finance, health care, and other regulated industries, care deeply about data privacy and IP violation protection, and some don’t care at all. If you don’t care about these, you may find that some of the other models may give you slightly better suggestions than Tabnine Protected, at least sometimes. Frankly, however, the ability to consider context is at least as important as the model, which is something that the developers of GitHub Copilot found out when they started allowing their model to consider all the code available in the workspace and not just the current active file.

Context, connection, coaching, and customization

That brings us to the four levels of context considered by Tabnine. The initial context (besides the current request) is the currently selected code, the current open files, and all the code in the current workspace, which is supplied to the model using both vector and semantic retrieval-augmented generation (RAG). The next level of context is global RAG, which you can set up by establishing a connection to your repos to access your entire code base, your requirements, and your documentation.

The third level of context, coaching, is achieved by providing your coding rules, a “golden” repo of code to prioritize, and expert solutions. These are added to Tabnine’s own rule set and guide its code review agent, code generation, and chat answers. The fourth level, customization, fine-tunes Tabnine’s AI models using your high-quality code base.

Testing and evaluating Tabnine

I tested Tabnine using an Enterprise-level subscription supplied by the company. (I wanted to test with a free basic subscription, but the company recommended against it.) I started by using Tabnine to understand a Java project supplied by the company, then went on to test code completion, code generation, test generation, code explanation, and code fixes in Python and C++. Tabnine updated and improved its Tabnine Protected model and its test generation while I was testing the product, but it’s not clear exactly when they did so.

At one point I attempted to generate a screen-scraping program in Python with each of Tabnine’s models. In the course of doing that, I spotted code that looked awfully familiar, saw code references to previous versions, and finally one model (GPT-4 Turbo) told me outright that “The code provided in the open file already contains a Python function scrape_website_for_heads that scrapes a website for headings (h1 to h6) and returns them as a list. Additionally, it includes a test function test_scrape_infoworld that tests this scraping function against https://www.infoworld.com/ by checking if certain expected headings are present.”

It went on to explain the code concisely, and show me the pattern for using my function to scrape other websites. Ten marks to GPT-4 for that. Also ten marks to Tabnine for providing lots of context to whatever model you use. Claude 3.5 Sonnet also recognized that I had code open that did what I had asked it for, but it decided to offer me a bunch of improvements to the code, including better error handling and a more flexible test. By the way, Tabnine maintains a full chat history.

I suppose that if I really wanted to test every model de novo I could open a new project for each one so that they couldn’t see any of my previous tests. It wasn’t that important to me. Once I finished laughing at myself, I moved on to my C++ code generation test, a quicksort function for arrays of strings, with tests and timings. I did this entirely with the Tabnine Protected model.

In response to my first inline comment,

// Function to sort an array of strings in alphabetical order using the quicksort algorithm

Tabnine correctly generated a quicksort function followed by a partition function. That didn’t compile, of course, but when I asked for a fix, Tabnine correctly generated a partition function prototype to place above the quicksort function.

When I asked it to generate tests for the quicksort function, it suggested five sets of five unit tests each; I chose the tests that used 1,000 elements and inserted them into my code. I forgot to set my insertion point correctly and made a mess, but a few undo commands got me back to the point where I could insert the tests correctly.

Then I asked Tabnine Chat to add timing for the tests. It suggested bracketing my tests with

auto start = std::chrono::high_resolution_clock::now();

and

auto end = std::chrono::high_resolution_clock::now(); auto duration = std::chrono::duration_cast(end - start); std::cout