How to boost Python program performance with Zig

Python might not be the fastest of languages, but it has many advantages. Users enjoy Python’s convenience and development speed, as well as its broad ecosystem of libraries. Something else people appreciate about Python is how it bridges language ecosystems. Anything compiled to C, or that exposes a C-like interface, can be used with Python. That includes Zig—a language whose popularity is on the rise due to its native speed and relative safety.

You can use Zig to write libraries that work with Python, or to create full-blown Python extensions. In this article, we’ll quickly walk through the mechanics of both—but note they are very different processes. Writing a C-interface library in Zig is easy enough, although the interfaces are generally limited to C types. Writing a Python extension in Zig is a bit harder. It requires using CPython’s structures and interfacing with the CPython runtime itself.

Write a C-interface library in Zig

If all you want to do is write a Zig library that exposes C-style public interfaces and use that with Python, the process is incredibly simple. Here’s a simple Zig program that exposes a method to multiply two 64-bit integers:

const std = @import('std');

export fn mult(a: i64, b: i64) i64 {
return a * b;
}

To compile this program, you’d use zig build-lib calc.zig -dynamic, which generates a linkable library. To call it from Python, all you need is ctypes:

import ctypes
calc = ctypes.CDLL('./calc.so')
# ^ or calc.dll in Microsoft Windows
print (calc.mult(512, 513))

If you’ve already worked with ctypes or other libraries for interfacing C with Python, the Python side of the equation should be familiar.

What to watch out for

There is one disadvantage to using the pure C interface with Zig libraries: If you want to pass around anything other than primitive C values, you need to do more work. It’s not hard to pass pointers back and forth to C structures, for instance. But it’s much harder to pass a Python object to a Zig module and work with it natively.

There are reasons to avoid this in the first place, though. Anything involving a rich Python object typically means dealing with the CPython runtime. You’ll not only use CPython headers to describe those rich objects, but you’ll contend with the possible performance issues of using them.

The Python project Cython has similar limitations. Transforming Python code into C with Cython doesn’t much speed things up unless your transformed code uses C types rather than Python objects.

You’ll get the best performance with Zig by using only C types for the interfaces, and native Zig constructions (not CPython objects) for your Zig code.

Writing native Python extension modules with Zig

So far we’ve only talked about using Zig to write modules that interface with Python through C types—in other words, modules that work with any language as long as they expose a C calling interface.

Far more complex—and more powerful—is writing Zig modules that work as native Python extension modules. These can be imported and used directly in Python as if they were Python code. Writing modules like this “by hand” is complex because it requires some knowledge of how to work with Python objects internally.

The first step, adding a reference to the CPython headers from its source, isn’t hard; you’d just use something like:

const python = @cImport({
@cInclude('Python.h');
});

But everything after that, like how to manage Python objects and pass them around, is more complex. (See this repository for some ideas, although it’s somewhat dated.)

In the Python world, there are code-generation tools that simplify writing Python extension modules, so you work as little as possible with Python’s internals. In the Zig world, there’s a project called Ziggy Pydust that serves a similar purpose. We can use it to write idiomatic Zig code that works as a Python extension module. All the interfacing with the CPython runtime is handled more or less automatically.

Here’s an example of a Pydust project from Pydust’s docs:

const py = @import('pydust');

pub fn hello()!py.PyString {
return try py.PyString.create('Hello!');
}

comptime {
py.rootmodule(@This());
}

Import the compiled module into Python, run its hello method, and you’ll get back a Python string with the text Hello!. Functions, classes and class instances, exceptions, the Python buffer protocol (for zero-copy memory access), and suspending the GIL are all available through Pydust.

Pydust currently has several limitations. The biggest and most problematic is that it currently only supports up to Zig version 0.11. The development of Pydust to support future versions of Zig also appears to be stalled. An open issue on GitHub tracks efforts to add support for the latest versions of Zig. Another possible limitation, depending on how you develop with Python, is Pydust’s dependency on the Poetry project management system. If you don’t already use Poetry, you may have to retool your workflow to set things up properly.

Conclusion

Python has been around for decades, with many firmly established behaviors that aren’t going anywhere. Zig is a young language, and developing rapidly, with plenty of room for change. The two complement each other more than they compete, and we’ve only started seeing all the ways they can work together.