Python CFFI

August 2018, PyGraz, Lukas Prokop

CFFI

A foreign function interface (FFI) is a mechanism by which a program written in one programming language can call routines or make use of services written in another.
—Wikipedia: Foreign function interface

Foreign Function Interface for Python calling C code
—PyPI: cffi

Maintainers: Armin Rigo and Maciej Fijalkowski
builtin support from PyPy! Based on LuaJIT design.

CFFI: Goals

The goal is to call C code from Python without learning a 3rd language: existing alternatives require users to learn domain specific language (Cython, SWIG) or API (ctypes). The CFFI design requires users to know only C and Python, minimizing the extra bits of API that need to be learned.

Slide by Armin Rigo: “CFFI - Call C from Python” (SPS16)

Armin Rigo: “CFFI - Call C from Python” (SPS16 / EuroPython 2016) [CFFI Intro]

Jean Baptiste Aviat: “Writing a C Python Extension in 2017” (PyCon 2017)

Ideas

CPython: implemented in C, so quite easy?!

#include <Python.h>
Just call Python functions you need - done?!
We need to compile it
So create a library and load it on runtime
Windows: LoadLibrary, Linux: dlopen
Windows: GetProcAddress, Linux: dlsym

Example:

void* handle = dlopen("./my_lib.so", RTLD_LAZY);
*(void**)(&my_fn) = dlsym(handle, "my_fn");

Remark: name mangling

int foobar(int a) { return a + 1; }

⇒ in Assembler: .type _Z6foobari,@function

extern "C" int foobar(int a);
int foobar(int a) { return a + 1; }

⇒ in Assembler: .type foobar,@function

Conclusion: Use extern keyword to avoid name mangling. C/Pascal has no function overloading. Some compilers do it anyways → use extern.

#[no_mangle]
pub extern "C" fn ...

The difficulties

What?	C	Python
runtime	compiled (def&decl at compile time)	interpreted (everything at runtime)
memory management	various	ref counting (mark-and-sweep, generational GC)
memory safety	unsafe/efficient	safe/inefficient
data structures	primitive	fancy

Type system is irrelevant. In a library, there is no type system (only bytes, pointers and identifiers).

The difficulties

If I give Python an integer, is it still the same value? (2-complement?)
If I create a Python object in C[++], will Python call its destructor?
Will python find the library, I am linking in my C code?
Can I run setup/teardown C code, before the Python runtime starts/terminates?

Hints

Many datatypes are standardized (IEEE 754 for floating point numbers)
Memory allocation / resource management is application-specific in C++: RAII, etc.
Library linking happens at compile time
Python runtime needs to be initialized in native extensions

Approaches

Considered:

CPython native extension (since ≤2000)
SWIG [Simplified Wrapper and Interface Generator] (since 1996) (C/C++ → Lua, Perl, PHP, Python, R, Ruby, Tcl, C#, Java, JavaScript, Go, Modula-3, OCaml, Octave, Scilab, Scheme)
ctypes (since 2006)
Cython (since 2007), forked from Pyrex (2002-2010)
cffi (since 2012)

Example: native extension

Approach:

Use API provided by #include <Python.h> for everything!
Declare as Extension in setup.py
distutils will generate shared library
Shared library will be loaded at runtime

Example: native extension

via mdobson's python-spam-module
[via python2 docu]

from distutils.core import setup, Extension

spam_module = Extension('spam',
    sources = ['spammodule.c'])

setup(name='Spam',
      version='1.0',
      description='Sample module.',
      ext_modules = [spam_module])

Example: native extension

via mdobson's python-spam-module

#include <Python.h>

static PyObject * SpamError;

static PyObject *
spam_system(PyObject *self, PyObject * args)
{
  const char *command;
  int sts;

  if(!PyArg_ParseTuple(args, "s", &command))
    return NULL;

  sts = system(command);
  if ( sts < 0 ) {
    PyErr_SetString(SpamError, "System command failed");
    return NULL;
  }
  return PyLong_FromLong(sts);
}

Example: native extension

via mdobson's python-spam-module

PyMODINIT_FUNC
initspam(void)
{
  PyObject *m;
  static PyMethodDef SpamMethods[] = {
    {"system", spam_system, METH_VARARGS, "Execute a shell command."},
    {NULL, NULL, 0, NULL}
  };
  m = Py_InitModule("spam", SpamMethods);
  if ( m == NULL )
    return;

  SpamError = PyErr_NewException("spam.error", NULL, NULL);
  Py_INCREF(SpamError);
  PyModule_AddObject(m, "error", SpamError);
}

int
main(int argc, char *argv[])
{
  Py_SetProgramName(argv[0]);
  Py_Initialize();
  initspam();
}

Example: native extension

Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import spam
>>> dir(spam)
['__doc__', '__file__', '__name__', '__package__', 'error', 'system']
>>> spam.system.__doc__
'Execute a shell command.'
>>> spam.system('ls -la')
total 8
drwxrwxr-x 1 meisterluk meisterluk   58 Jul 26 00:30 .
drwxr-xr-x 1 meisterluk meisterluk 2162 Jul 26 00:31 ..
drwxrwxr-x 1 meisterluk meisterluk  164 Jul 26 00:31 build
-rw-rw-r-- 1 meisterluk meisterluk  213 Jul 26 00:24 setup.py
-rw-rw-r-- 1 meisterluk meisterluk  834 Jul 26 00:24 spammodule.c
drwxrwxr-x 1 meisterluk meisterluk   72 Jul 26 00:30 venv
0L

Example: SWIG

example.c implementation file:

/* File : example.c */ 
#include <time.h>
double My_variable = 3.0;
int fact(int n) {
   if (n <= 1) return 1;
   else return n*fact(n-1);
}
int my_mod(int x, int y) {
   return (x%y);
}
char *get_time() {
   time_t ltime;
   time(&ltime);
   return ctime(&ltime);
}

Example: SWIG

example.i interface file:

/* example.i */
%module example
%{
// Put header files here or function declarations like below
extern double My_variable;
extern int fact(int n);
extern int my_mod(int x, int y);
extern char *get_time();
%}
extern double My_variable;
extern int fact(int n);
extern int my_mod(int x, int y);
extern char *get_time();

Example: SWIG

Build library:

$ swig -python example.i
$ gcc -fPIC -c example.c example_wrap.c \
      -I/usr/include/python3.6
$ ld -shared example.o example_wrap.o -o _example.so

Usage:

>>> import example
>>> example.fact(5)
120
>>> example.my_mod(7, 3)
1
>>> example.get_time()
'Tue Aug  7 00:12:31 2018\n'
>>>

Example: SWIG

$ wc -l example.py
113 example.py
$ wc -l example_wrap.c
4043 example_wrap.c

example.py is generated:

# This file was automatically generated by SWIG (http://www.swig.org).
# Version 3.0.12
#
# Do not make changes to this file unless you know what you are doing--modify
# the SWIG interface file instead.

from sys import version_info as _swig_python_version_info
if _swig_python_version_info >= (2, 7, 0):
    def swig_import_helper():
        import importlib
        pkg = __name__.rpartition('.')[0]
        mname = '.'.join((pkg, '_example')).lstrip('.')
        try:
            return importlib.import_module(mname)
        except ImportError:
            return importlib.import_module('_example')
    _example = swig_import_helper()
    del swig_import_helper
elif _swig_python_version_info >= (2, 6, 0):
    def swig_import_helper():
        from os.path import dirname
        import imp
        fp = None
        try:
            fp, pathname, description = imp.find_module('_example', [dirname(__file__)])
        except ImportError:
            import _example
            return _example
        try:
            _mod = imp.load_module('_example', fp, pathname, description)
        finally:
            if fp is not None:
                fp.close()
        return _mod
    _example = swig_import_helper()
    del swig_import_helper
else:
    import _example
del _swig_python_version_info

try:
    _swig_property = property
except NameError:
    pass  # Python < 2.2 doesn't have 'property'.

try:
    import builtins as __builtin__
except ImportError:
    import __builtin__

def _swig_setattr_nondynamic(self, class_type, name, value, static=1):
    if (name == "thisown"):
        return self.this.own(value)
    if (name == "this"):
        if type(value).__name__ == 'SwigPyObject':
            self.__dict__[name] = value
            return
    method = class_type.__swig_setmethods__.get(name, None)
    if method:
        return method(self, value)
    if (not static):
        if _newclass:
            object.__setattr__(self, name, value)
        else:
            self.__dict__[name] = value
    else:
        raise AttributeError("You cannot add attributes to %s" % self)


def _swig_setattr(self, class_type, name, value):
    return _swig_setattr_nondynamic(self, class_type, name, value, 0)


def _swig_getattr(self, class_type, name):
    if (name == "thisown"):
        return self.this.own()
    method = class_type.__swig_getmethods__.get(name, None)
    if method:
        return method(self)
    raise AttributeError("'%s' object has no attribute '%s'" % (class_type.__name__, name))


def _swig_repr(self):
    try:
        strthis = "proxy of " + self.this.__repr__()
    except __builtin__.Exception:
        strthis = ""
    return "<%s.%s; %s >" % (self.__class__.__module__, self.__class__.__name__, strthis,)

try:
    _object = object
    _newclass = 1
except __builtin__.Exception:
    class _object:
        pass
    _newclass = 0


def fact(n):
    return _example.fact(n)
fact = _example.fact

def my_mod(x, y):
    return _example.my_mod(x, y)
my_mod = _example.my_mod

def get_time():
    return _example.get_time()
get_time = _example.get_time
# This file is compatible with both classic and new-style classes.

cvar = _example.cvar

Example: SWIG

SWIG is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages. SWIG is used with different types of target languages including common scripting languages such as Javascript, Perl, PHP, Python, Tcl and Ruby. [...] SWIG is typically used to parse C/C++ interfaces and generate the 'glue code' required for the above target languages to call into the C/C++ code. [...] SWIG is free software and the code that SWIG generates is compatible with both commercial and non-commercial projects.

—swig.org

Example: ctypes

Approach:

Use datatypes provided by ctypes (e.g. c_bool)
Generate shared library explicitly
Call Python functions explicitly to read shared library

Example: ctypes

via Chinmay Kanchi on Stackoverflow

#include <stdio.h>
void myprint() { printf("hello world\n"); }

$ gcc -shared -Wl,-soname,testlib -o testlib.so -fPIC testlib.c
$ file testlib.so
testlib.so: ELF 64-bit LSB shared object, x86-64,
version 1 (SYSV), dynamically linked,
BuildID[sha1]=b1b8a2cdf92fafcc5561f55fb94ca6a27b40270d,
not stripped

Example: ctypes

import ctypes

testlib = ctypes.CDLL('/full/path/testlib.so')
testlib.myprint()

$ python2 testlibwrapper.py
hello world

$ python3 testlibwrapper.py
hello world

Example: Cython

helloworld.pyx:

print("Hello World")

setup.py:

from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules = cythonize("helloworld.pyx"))

Build python library:

$ python setup.py build_ext --inplace
$ wc -l helloworld.c
2513 helloworld.c

Example: Cython

/* Generated by Cython 0.28.5 */

/* BEGIN: Cython Metadata
{
    "distutils": {
        "name": "helloworld",
        "sources": [
            "helloworld.pyx"
        ]
    },
    "module_name": "helloworld"
}
END: Cython Metadata */

#define PY_SSIZE_T_CLEAN
#include "Python.h"
#ifndef Py_PYTHON_H
    #error Python headers needed to compile C extensions, please install development version of Python.
#elif PY_VERSION_HEX < 0x02060000 || (0x03000000 <= PY_VERSION_HEX && PY_VERSION_HEX < 0x03030000)
    #error Cython requires Python 2.6+ or Python 3.3+.
#else
#define CYTHON_ABI "0_28_5"
#define CYTHON_FUTURE_DIVISION 0
#include <stddef.h>
#ifndef offsetof
  #define offsetof(type, member) ( (size_t) & ((type*)0) -> member )
#endif
#if !defined(WIN32) && !defined(MS_WINDOWS)
  #ifndef __stdcall
    #define __stdcall
  #endif
  #ifndef __cdecl
    #define __cdecl
  #endif
  #ifndef __fastcall
    #define __fastcall
  #endif
#endif
#ifndef DL_IMPORT
  #define DL_IMPORT(t) t
#endif
#ifndef DL_EXPORT
  #define DL_EXPORT(t) t
#endif
#define __PYX_COMMA ,
#ifndef HAVE_LONG_LONG
  #if PY_VERSION_HEX >= 0x02070000
    #define HAVE_LONG_LONG
  #endif
#endif
#ifndef PY_LONG_LONG
  #define PY_LONG_LONG LONG_LONG
#endif
#ifndef Py_HUGE_VAL
  #define Py_HUGE_VAL HUGE_VAL
#endif
#ifdef PYPY_VERSION
  #define CYTHON_COMPILING_IN_PYPY 1
  #define CYTHON_COMPILING_IN_PYSTON 0
  #define CYTHON_COMPILING_IN_CPYTHON 0
  #undef CYTHON_USE_TYPE_SLOTS
  #define CYTHON_USE_TYPE_SLOTS 0
  #undef CYTHON_USE_PYTYPE_LOOKUP
  #define CYTHON_USE_PYTYPE_LOOKUP 0
  #if PY_VERSION_HEX < 0x03050000
    #undef CYTHON_USE_ASYNC_SLOTS
    #define CYTHON_USE_ASYNC_SLOTS 0
  #elif !defined(CYTHON_USE_ASYNC_SLOTS)
    #define CYTHON_USE_ASYNC_SLOTS 1
  #endif
  #undef CYTHON_USE_PYLIST_INTERNALS
  #define CYTHON_USE_PYLIST_INTERNALS 0
  #undef CYTHON_USE_UNICODE_INTERNALS
  #define CYTHON_USE_UNICODE_INTERNALS 0
  #undef CYTHON_USE_UNICODE_WRITER
  #define CYTHON_USE_UNICODE_WRITER 0
  #undef CYTHON_USE_PYLONG_INTERNALS
  #define CYTHON_USE_PYLONG_INTERNALS 0
  #undef CYTHON_AVOID_BORROWED_REFS
  #define CYTHON_AVOID_BORROWED_REFS 1
  #undef CYTHON_ASSUME_SAFE_MACROS
  #define CYTHON_ASSUME_SAFE_MACROS 0
  #undef CYTHON_UNPACK_METHODS
  #define CYTHON_UNPACK_METHODS 0
  #undef CYTHON_FAST_THREAD_STATE
  #define CYTHON_FAST_THREAD_STATE 0
  #undef CYTHON_FAST_PYCALL
  #define CYTHON_FAST_PYCALL 0
  #undef CYTHON_PEP489_MULTI_PHASE_INIT
  #define CYTHON_PEP489_MULTI_PHASE_INIT 0
  #undef CYTHON_USE_TP_FINALIZE
  #define CYTHON_USE_TP_FINALIZE 0
[...]

#if PY_MAJOR_VERSION >= 3
  #define __Pyx_PyString_Format(a, b)  PyUnicode_Format(a, b)
#else
  #define __Pyx_PyString_Format(a, b)  PyString_Format(a, b)
#endif
#if PY_MAJOR_VERSION < 3 && !defined(PyObject_ASCII)
  #define PyObject_ASCII(o)            PyObject_Repr(o)
#endif
#if PY_MAJOR_VERSION >= 3
  #define PyBaseString_Type            PyUnicode_Type
  #define PyStringObject               PyUnicodeObject
  #define PyString_Type                PyUnicode_Type
  #define PyString_Check               PyUnicode_Check
  #define PyString_CheckExact          PyUnicode_CheckExact
  #define PyObject_Unicode             PyObject_Str
#endif
[...]
static int __Pyx_sys_getdefaultencoding_not_ascii;
static int __Pyx_init_sys_getdefaultencoding_params(void) {
    PyObject* sys;
    PyObject* default_encoding = NULL;
    PyObject* ascii_chars_u = NULL;
    PyObject* ascii_chars_b = NULL;
    const char* default_encoding_c;
    sys = PyImport_ImportModule("sys");
    if (!sys) goto bad;
    default_encoding = PyObject_CallMethod(sys, (char*) "getdefaultencoding", NULL);
    Py_DECREF(sys);
    if (!default_encoding) goto bad;
    default_encoding_c = PyBytes_AsString(default_encoding);
    if (!default_encoding_c) goto bad;
    if (strcmp(default_encoding_c, "ascii") == 0) {
        __Pyx_sys_getdefaultencoding_not_ascii = 0;
    } else {
        char ascii_chars[128];
        int c;
        for (c = 0; c < 128; c++) {
            ascii_chars[c] = c;
        }
        __Pyx_sys_getdefaultencoding_not_ascii = 1;
        ascii_chars_u = PyUnicode_DecodeASCII(ascii_chars, 128, NULL);
        if (!ascii_chars_u) goto bad;
        ascii_chars_b = PyUnicode_AsEncodedString(ascii_chars_u, default_encoding_c, NULL);
        if (!ascii_chars_b || !PyBytes_Check(ascii_chars_b) || memcmp(ascii_chars, PyBytes_AS_STRING(ascii_chars_b), 128) != 0) {
            PyErr_Format(
                PyExc_ValueError,
                "This module compiled with c_string_encoding=ascii, but default encoding '%.200s' is not a superset of ascii.",
                default_encoding_c);
            goto bad;
        }
        Py_DECREF(ascii_chars_u);
        Py_DECREF(ascii_chars_b);
    }
    Py_DECREF(default_encoding);
    return 0;
bad:
    Py_XDECREF(default_encoding);
    Py_XDECREF(ascii_chars_u);
    Py_XDECREF(ascii_chars_b);
    return -1;
}
[...]
static PyObject *__pyx_m = NULL;
static PyObject *__pyx_d;
static PyObject *__pyx_b;
static PyObject *__pyx_cython_runtime = NULL;
static PyObject *__pyx_empty_tuple;
static PyObject *__pyx_empty_bytes;
static PyObject *__pyx_empty_unicode;
static int __pyx_lineno;
static int __pyx_clineno = 0;
static const char * __pyx_cfilenm= __FILE__;
static const char *__pyx_filename;
static const char *__pyx_f[] = {
  "helloworld.pyx",
};
[...]
/* Module declarations from 'helloworld' */
#define __Pyx_MODULE_NAME "helloworld"
extern int __pyx_module_is_main_helloworld;
int __pyx_module_is_main_helloworld = 0;

/* Implementation of 'helloworld' */
static const char __pyx_k_end[] = "end";
static const char __pyx_k_file[] = "file";
static const char __pyx_k_main[] = "__main__";
static const char __pyx_k_test[] = "__test__";
static const char __pyx_k_print[] = "print";
static const char __pyx_k_Hello_World[] = "Hello World";
static const char __pyx_k_cline_in_traceback[] = "cline_in_traceback";
static PyObject *__pyx_kp_s_Hello_World;
static PyObject *__pyx_n_s_cline_in_traceback;
static PyObject *__pyx_n_s_end;
static PyObject *__pyx_n_s_file;
static PyObject *__pyx_n_s_main;
static PyObject *__pyx_n_s_print;
static PyObject *__pyx_n_s_test;
/* Late includes */

static PyMethodDef __pyx_methods[] = {
  {0, 0, 0, 0}
};
[...]
static __Pyx_StringTabEntry __pyx_string_tab[] = {
  {&__pyx_kp_s_Hello_World, __pyx_k_Hello_World, sizeof(__pyx_k_Hello_World), 0, 0, 1, 0},
  {&__pyx_n_s_cline_in_traceback, __pyx_k_cline_in_traceback, sizeof(__pyx_k_cline_in_traceback), 0, 0, 1, 1},
  {&__pyx_n_s_end, __pyx_k_end, sizeof(__pyx_k_end), 0, 0, 1, 1},
  {&__pyx_n_s_file, __pyx_k_file, sizeof(__pyx_k_file), 0, 0, 1, 1},
  {&__pyx_n_s_main, __pyx_k_main, sizeof(__pyx_k_main), 0, 0, 1, 1},
  {&__pyx_n_s_print, __pyx_k_print, sizeof(__pyx_k_print), 0, 0, 1, 1},
  {&__pyx_n_s_test, __pyx_k_test, sizeof(__pyx_k_test), 0, 0, 1, 1},
  {0, 0, 0, 0, 0, 0, 0}
};
static int __Pyx_InitCachedBuiltins(void) {
  return 0;
}

static int __Pyx_InitCachedConstants(void) {
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("__Pyx_InitCachedConstants", 0);
  __Pyx_RefNannyFinishContext();
  return 0;
}

static int __Pyx_InitGlobals(void) {
  if (__Pyx_InitStrings(__pyx_string_tab) < 0) __PYX_ERR(0, 1, __pyx_L1_error);
  return 0;
  __pyx_L1_error:;
  return -1;
}
[...]
#if PY_MAJOR_VERSION < 3
__Pyx_PyMODINIT_FUNC inithelloworld(void) CYTHON_SMALL_CODE; /*proto*/
__Pyx_PyMODINIT_FUNC inithelloworld(void)
#else
__Pyx_PyMODINIT_FUNC PyInit_helloworld(void) CYTHON_SMALL_CODE; /*proto*/
__Pyx_PyMODINIT_FUNC PyInit_helloworld(void)
#if CYTHON_PEP489_MULTI_PHASE_INIT
{
  return PyModuleDef_Init(&__pyx_moduledef);
}
[...]
  /* "helloworld.pyx":1
 * print("Hello World")             # <<<<<<<<<<<<<<
 */
  if (__Pyx_PrintOne(0, __pyx_kp_s_Hello_World) < 0) __PYX_ERR(0, 1, __pyx_L1_error)
  __pyx_t_1 = __Pyx_PyDict_NewPresized(0); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 1, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if (PyDict_SetItem(__pyx_d, __pyx_n_s_test, __pyx_t_1) < 0) __PYX_ERR(0, 1, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;

  /*--- Wrapped vars code ---*/

  goto __pyx_L0;
  __pyx_L1_error:;
  __Pyx_XDECREF(__pyx_t_1);
  if (__pyx_m) {
    if (__pyx_d) {
      __Pyx_AddTraceback("init helloworld", 0, __pyx_lineno, __pyx_filename);
    }
    Py_DECREF(__pyx_m); __pyx_m = 0;
  } else if (!PyErr_Occurred()) {
    PyErr_SetString(PyExc_ImportError, "init helloworld");
  }
  __pyx_L0:;
  __Pyx_RefNannyFinishContext();
  #if CYTHON_PEP489_MULTI_PHASE_INIT
  return (__pyx_m != NULL) ? 0 : -1;
  #elif PY_MAJOR_VERSION >= 3
  return __pyx_m;
  #else
  return;
  #endif
}
[...]

Example: Cython

def primes(int nb_primes):
    cdef int n, i, len_p
    cdef int p[1000]
    if nb_primes > 1000:
        nb_primes = 1000
    len_p = 0  # The current number of elements in p.
    n = 2
    while len_p < nb_primes:
        # Is n prime?
        for i in p[:len_p]:
            if n % i == 0:
                break
        # If no break occurred in the loop, we have a prime.
        else:
            p[len_p] = n
            len_p += 1
        n += 1
    # Let's return the result in a python list:
    result_as_list  = [prime for prime in p[:len_p]]
    return result_as_list

Example: Cython

import random
from libc.stdlib cimport malloc, free
def random_noise(int number=1):
    cdef int i
    # allocate number * sizeof(double) bytes of memory
    cdef double *my_array = <double *> malloc(number * sizeof(double))
    if not my_array:
        raise MemoryError()
    try:
        ran = random.normalvariate
        for i in range(number):
            my_array[i] = ran(0, 1)
        # ... do some more heavy C calculations here ...
        return [x for x in my_array[:number]]
    finally:
        # return the previously allocated memory to the system
        free(my_array)

Example: Cython

from cpython cimport array
import array

cdef array.array a = array.array('i', [1, 2, 3])
cdef int[:] ca = a

cdef int overhead(object a):
    cdef int[:] ca = a
    return ca[0]

cdef int no_overhead(int[:] ca):
    return ca[0]

print(overhead(a))  # create new memory view, overhead
print(no_overhead(ca))  # memory view ca exists, no overhead

Example: CFFI

via github/tmcclintock/cffi_example

// The .c file with the functions themselves.
#include "mysrc.h"
#include <stdio.h>

void mybasic(void) {
  printf("mybasic() called\n");
  return;
}
int myprint(char* message) {
  return printf("%s\n",message);
}
double myadd(double x, double y) {
  return x + y;
}

Example: CFFI

via github/tmcclintock/cffi_example

// A header file with function signatures for my functions.
void mybasic(void);
int myprint(char*);
double myadd(double, double);

Example: CFFI

via github/tmcclintock/cffi_example/mycffi/__init__.py

# The init file where everything is linked together.
import os, cffi, glob

# Figure out the directories to everything.
mycffi_dir = os.path.dirname(__file__)
include_dir = os.path.join(mycffi_dir, 'include')
# This will be the name of the library file that gets compiled.
lib_file = os.path.join(mycffi_dir, '_mycffi.so')

# Use cffi.FFI().cdef() to be able to call the functions
# specified in the header files. This is necessary.
_ffi = cffi.FFI()
for file_name in glob.glob(os.path.join(include_dir, '*.h')):
    _ffi.cdef(open(file_name).read())
_lib = _ffi.dlopen(lib_file)

# Import everything from test. This isn't necessary,
# since we could call mycffi._lib.myfunc() directly,
# but I wrote nice wrappers in the .test module.
# This is also how you would get documentation to appear,
# since it won't be generated directly from the headers.
from .test import *

Example: CFFI

via github/tmcclintock/cffi_example/mycffi/test.py

# Interfaces to the functions.
# Alternatively one can call mycffi._lib.myfunc() directly.
def mybasic():
    import mycffi
    mycffi._lib.mybasic()

def myprint(message):
    import mycffi
    mycffi._lib.myprint(message)
    return

def myadd(x, y):
    """Adds two numbers together.
    x: first number
    y: second number
    returns the sum
    """
    import mycffi
    return mycffi._lib.myadd(x, y)

Example: CFFI

via github/tmcclintock/cffi_example/setup.py

from __future__ import print_function
import sys, os, glob
import setuptools
from setuptools import setup, Extension

# Create the symlink.
try:
    os.symlink('../include/', 'mycffi/include')
except:
    OSError
    
# Specify the sources
sources = glob.glob(os.path.join('src','*.c'))
print('sources = ',sources)
# and the header files.
headers = glob.glob(os.path.join('include','*.h'))
print('headers = ',headers)
# Create the extension.
ext=Extension("mycffi._mycffi", sources, depends=headers, include_dirs=['include'])

# Create the distribution.
dist = setup(name="mycffi", author="Tom McClintock",
             description="A usage example for cffi.",
             license="MIT License",
             url="https://github.com/tmcclintock/cffi_example",
             packages=['mycffi'],
             package_data={'mycffi': headers},
             ext_modules=[ext])

# setup.py doesn't put the .so file in the mycffi directory, 
# so this bit makes it possible to
# import mycffi from the root directory.  
# Not really advisable, but everyone does it at some
# point, so might as well facilitate it.
build_lib = glob.glob(os.path.join('build','*','mycffi','_mycffi*.so'))
if len(build_lib) >= 1:
    lib = os.path.join('mycffi', '_mycffi.so')
    if os.path.lexists(lib): os.unlink(lib)
os.link(build_lib[0], lib)

Example: CFFI

Python 3.6.5 (default, Apr  1 2018, 05:46:30) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits"
  or "license" for more information.
>>> import mycffi
>>> mycffi.mybasic()
mybasic() called
>>> mycffi.myadd(4, 9)
13.0
>>> mycffi.myprint(b"Hello World")
Hello World

Example: CFFI

4j + 3j + 2

>>> 4j + 3j + 2
(2+7j)

>>> mycffi.myadd(4, 13j)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/meisterluk/py-cffi-example/mycffi/test.py", line 21, in myadd
    return mycffi._lib.myadd(x, y)
TypeError: can't convert complex to float
>>> mycffi.myprint(4)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/meisterluk/py-cffi-example/mycffi/test.py", line 9, in myprint
    mycffi._lib.myprint(message)
TypeError: initializer for ctype 'char *' must be a cdata pointer, not int
>>> mycffi.myprint("Hello World")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/meisterluk/py-cffi-example/mycffi/test.py", line 9, in myprint
    mycffi._lib.myprint(message)
TypeError: initializer for ctype 'char *' must be a bytes or list or tuple, not str
>>>

Another CFFI example

via CFFI documentation

>>> from cffi import FFI
>>> ffi = FFI()
>>> ffi.cdef("""
...     int printf(const char *format, ...); // copy-pasted from the man page
... """)
>>> C = ffi.dlopen(None)             # loads the entire C namespace
>>> arg = ffi.new("char[]", "world") # equivalent to C code: char arg[] = "world";
>>> C.printf("hi there, %s.\n", arg) # call printf
hi there, world.
17       # this is the return value
>>>

Another CFFI example

This example does not call any C compiler. It works in the so-called ABI mode, which means that it will crash if you call some function or access some fields of a structure that was slightly misdeclared in the cdef().

—CFFI documentation

Yet another CCFI example

via CFFI documentation

# file "example_build.py"

# Note: we instantiate the same 'cffi.FFI' class as in the previous
# example, but call the result 'ffibuilder' now instead of 'ffi';
# this is to avoid confusion with the other 'ffi' object you get below

from cffi import FFI
ffibuilder = FFI()

ffibuilder.set_source("_example",
   r""" // passed to the real C compiler,
        // contains implementation of things declared in cdef()
        #include <sys/types.h>
        #include <pwd.h>

        struct passwd *get_pw_for_root(void) {
            return getpwuid(0);
        }
    """,
    libraries=[])   # or a list of libraries to link with
    # (more arguments like setup.py's Extension class:
    # include_dirs=[..], extra_objects=[..], and so on)

ffibuilder.cdef("""
    // declarations that are shared between Python and C
    struct passwd {
        char *pw_name;
        ...;     // literally dot-dot-dot
    };
    struct passwd *getpwuid(int uid);     // defined in <pwd.h>
    struct passwd *get_pw_for_root(void); // defined in set_source()
""")

if __name__ == "__main__":
    ffibuilder.compile(verbose=True)

Yet another CFFI example

You need to run the example_build.py script once to generate “source code” into the file _example.c and compile this to a regular C extension module. (CFFI selects either Python or C for the module to generate based on whether the second argument to set_source() is None or not.)

You need a C compiler for this single step. It produces a file called e.g. _example.so or _example.pyd. If needed, it can be distributed in precompiled form like any other extension module.

—CFFI documentation

Yet another CFFI example

via CFFI documentation

from setuptools import setup
setup(
    ...
    setup_requires=["cffi>=1.0.0"],
    cffi_modules=["example_build.py:ffibuilder"],
    install_requires=["cffi>=1.0.0"],
)

Yet another CCFI example

via CFFI documentation

from _example import ffi, lib
p = lib.getpwuid(0)
assert ffi.string(p.pw_name) == b'root'
p = lib.get_pw_for_root()
assert ffi.string(p.pw_name) == b'root'

Conclusions

As Vladislav pointed out, they are different things. Thus they solve different problems. I would choose cffi when I just need to call a few C libraries. When I need performance or to fluidly interact with C code, I would choose Cython. By giving hints to the language it can take optimal paths for the limited scope you are describing. Just a few hints can give you tremendous performance gains.

To be honest, I would probably just use Cython for both, but that is preference.

—Quora: “When would you choose cffi compared to Cython?”

Thanks!

Q/A?

😀

http://lukas-prokop.at/talks/pygraz-cffi/