PythoC: A new way to generate C code from Python

Python and C share more than it might seem. The reference version of the Python interpreter is written in C, and many of the third-party libraries written for Python wrap C code. It’s also possible to generate C code from Python.

Generating C code with Python has typically involved libraries like Cython, which use type-annotated Python code to generate C extension modules for Python.

A new project, PythoC, takes a different approach. It uses type-hinted Python to programmatically generate C code—but chiefly for standalone use, and with many more compile-time code generation features than Cython has.

PythoC’s makers use the phrase “C level runtime, Python powered compile time” to describe their approach. The project is still in its early phases, but there are enough working features to make it worth a look.

A basic PythoC program

Here’s a simple program adapted from PythoC’s examples:

from pythoc import compile, i32

@compile
def add(x: i32, y: i32) -> i32:
return x + y

if __name__ == '__main__':
print(add(10, 20))

To indicate which functions in a module to compile to C, you use PythoC’s @compile decorator, supplying type hints for the result and each parameter. Note that you also need to import PythoC’s own i32 hint, instead of using Python’s native int. This means you’re using machine-native integers and not Python’s arbitrary-size ints.

When you run this program, you’ll get 30 as the output, after a delay. The C code is compiled on the fly each time you execute the program, hence the delay. PythoC doesn’t yet have a mechanism for re-using compiled code when it’s called from Python, the way Cython does.

At first this seems like a pretty big limitation. But it’s actually the point: You can use PythoC as a code generation system for C programs that run independently, rather than C modules imported into Python.

Generating standalone C programs

Here’s a new verson of the same program, with different behaviors.

from pythoc import compile, i32, ptr, i8
from pythoc.libc.stdio import printf

@compile
def add(x: i32, y: i32) -> i32:
return x + y

@compile
def main(argc: i32, argv: ptr[ptr[i8]]) -> i32:
printf('%un', add(10, 20))

if __name__ == '__main__':
from pythoc import compile_to_executable
compile_to_executable()

The first thing you’ll probably notice is the block at the bottom. The compile_to_executable() function is exactly what it sounds like. Call it, and the current module is compiled to an executable of the same name, with all the @compile-decorated functions included.

Another difference is that the main() function now has the same signature as the main() function in a C program. This means the compiled executable will automatically use that as its entry point.

Finally, when you run this program, the generated executable (which shows up in a build subdirectory) doesn’t run automatically; you have to run it yourself. The aim here is to build a standalone C program, indistinguishable from one you wrote by hand in C, but using Python’s syntax.

PythoC’s emulation of C features

With only a few exceptions, PythoC can generate code that fully utilizes C’s feature set and runtime.

You’ve already seen how to use type annotations to indicate primitive data types. You can likewise use the ptr[T] annotation to describe a pointer (also shown above), and use array[T, N] for N-dimensional arrays of type T. You can make structs, unions, and enums by decorating Python classes, and all the usual operators and control-flow operations (except for goto) will work. For switch/case, just use match/case, although fall-through cases aren’t available.

Something else that’s missing is variable-length arrays. In C, this feature is only supported in C11 and beyond, and support for it in compilers is optional, so it’s not surprising PythoC doesn’t support it yet.

Compile-time code generation

It’s possible to use Cython for compile-time code generation, which means you can produce different kinds of C code, or even fall back to Python code, depending on what happens at compile time. But PythoC’s compile-time code generation has abilities Cython lacks.

Here’s an example from PythoC’s documentation:

from pythoc import compile, struct, i32, f64

def make_point(T):
@struct(suffix=T)
class Point:
x: T
y: T

@compile(suffix=T)
def add_points(p1: Point, p2: Point) -> Point:
result: Point = Point()
result.x = p1.x + p2.x
result.y = p1.y + p2.y
return result

return Point, add_points

Point_i32, add_i32 = make_point(i32)
Point_f64, add_f64 = make_point(f64)

The make_point(T) function takes in some type annotation (i32, f64), and generates at compile time type-specialized versions of the Point class and add_points functions. The suffix parameter for @compile means “alter the name of the generated object so that the type is used in the name”—so, for example, Point becomes Point_i32 and Point_i64, which in C is one way to distinguish between multiple versions of the same function with a different type signature. It’s also possible to use this in conjunction with runtime dispatch to provide polymorphism.

Memory safety features

The bugs that can spring from C’s manual memory management are gloomily familiar to anyone who uses the language. Cython has memory safety features to address this, but PythoC offers unique type-based features in this vein.

One is a feature called linear types. The linear import lets you generate a “proof,” usually to accompany a memory allocation, that has to be “consumed” when the same memory is deallocated. If you don’t have a matching consume(prf) for each prf=linear(), the PythoC compiler will generate a compile-time error. The documentation for this, linked above, shows how to create simple lmalloc()/lfree() functions to allocate and free memory safely. Nothing says you must us linear types over manually using malloc()/free(), but they can automate much manual checking and centralize it at compile time rather than runtime.

Another type-based safety feature is refinement types. The idea here is that you can define a function to perform a certain kind of check—e.g., for a null pointer—with a boolean result. You can then use the refine() function to pass a value to that function and get back a type specific to that function, refined[func]. This allows the compiler to ensure that type has to be handled in some manner before being returned, and allows common checks (again for things like a non-null pointer) to be handled in a single place in your code. Cython’s type system is mostly for emulating C’s behaviors directly, and so doesn’t include anything like this.

Possible future directions for PythoC

PythoC is still quite new, so its future development is relatively open ended. One possibility is that it could integrate more closely with Python at runtime. For instance, a @cached decorator could compile modules once, ahead of time, and then re-use the compiled modules when they’re called from within Python, instead of being recompiled at each run. Of course, this would also require integration with Python’s existing module build system. While that level of integration might not be part of the project’s aim, it would make PythoC more immediately useful to those integrating C and Python.