Setup python environment (Ubuntu 20.04)

Get pip and virtualenv installed:

sudo apt-get install python3-pip
sudo apt-get update
sudo apt-get install python3-venv

Create a new environment and activate it:

python3 -m venv tf
source tf/bin/activate

Install the tensorflow library and, if needed, opencv:

pip install tensorflow
pip install opencv-contrib-python

Tensorflow lite

If there is no need to train models, a TensorFlow lite environment can be used by itself. One needs to select the correct version from this list and install it. For example on Ubuntu 20.04 64bit:

pip3 install https://dl.google.com/coral/python/tflite_runtime-2.1.0.post1-cp37-cp37m-linux_x86_64.whl

The tensorflow interpreter can now be used as follows:

import tflite_runtime.interpreter as tflite
interpreter = tflite.Interpreter(model_path=args.model_file)

Coral USB accelerator

First, install the coral dependencies:

sudo apt-get install curl

echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

sudo apt-get update

Finally, install the coral library. There are two versions available, the standard and the max version, which is faster but the USB device gets hot fast. NB: only one version can be installed in the system. Installing a different one will override the previous.

# standard:
sudo apt-get install libedgetpu1-std

# max:
sudo apt-get install libedgetpu1-max

Compile model for edge hardware

To achieve the best performance from the coral USB accelerator we need to compile our tensorflow model. To do so we need to install the edgetpu-compiler following the instruction of this link. Notice that we can not use the compiler directly on an ARM64 device such as the raspberry pi, but we need a more powerful desktop environment. Once the compiler is installed we can convert our model as follows:

edgetpu_compiler my_model.tflite

The output will be my_model_edge.tflite.

Notice that the compiler requires a quantized tflite model as input. The quantize model can be created during training or by transforming a trained model. All the instructions can be found here. The post-training quantization procedure is the following:

import tensorflow as tf

def representative_dataset():
    for data in tf.data.Dataset.from_tensor_slices((rep_images)).batch(1).take(100):
        yield [data]
        
saved_model_dir = 'my_tensorflow_full_model.model'
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
tflite_quant_model = converter.convert()

with tf.io.gfile.GFile('quantized_model.tflite', 'wb') as f:
    f.write(tflite_quant_model)

where my_tensorflow_full_model.model is the path to the original tensorflow model and rep_images is a list containing representative samples the model can expect as input (for example out test data).