Mastering MCAP - Working with OAK-D MCAP Data in Python
Background
Embarking on a new problem space is often a journey of frustration and enlightenment, a relentless cycle of micro-victories punctuated by setbacks, until a breakthrough finally arrives. Recently, my team and I found ourselves thrust into this all-too-familiar cycle once again.
We had taken on a computer vision project that demanded the use of an on-edge camera device to perform real-time event detection powered by artificial intelligence. Little did we know, we were about to engage in a battle against incomplete documentation, outdated tools, and a seemingly endless string of failures — all in a quest to learn how to work with the camera’s data in a Python environment, both in real-time and batch processing modes.
Our Mission: prove that it was possible to load MCAP data from an OAK-D camera into a Python environment and manipulate the video frames just like ordinary images.
OAK-D Camera
OAK-D by Luxonus is a vision camera with stereo depth and high-resolution colour. It is supported by handy Python packages that make it easy to record footage and deploy AI algorithms.
What is MCAP?
MCAP is an open source container file format for multi-modal log data. In other words, it is a way to store data from multiple sources (like cameras and sensors) together, with timestamps. It is typically used in robotics or circumstances where on edge compute is required (like our project!).
Devilish Data
Understanding the structure of the data within the MCAP files output by the camera was our first critical task. Given the flexibility of the MCAP format, the underlying packaged data can vary drastically, making this step crucial.
Let’s get our Python environment foundations in place, starting with a few essential dependencies.
! pip install mcap
! pip install imagecodecs
! pip install cv_bridge
! sudo apt-get install ros-sensor-msgs
! pip install std_msgs
! pip install Pillow
Ready to dive into MCAP files? Buckle up! This code snippet is your launchpad.
We’ll be taking a peek at what the code outputs, but first things first: let’s talk about the data we’re working with. This code focuses on the “color/compressed” channel, which essentially acts as your image frame, just like the red, green, and blue (RGB) data that forms a regular picture. We’re leaving the depth and stereo channels from the camera out of the picture for now, but don’t worry — we can explore those in future adventures!
from mcap.reader import make_reader
with open("recording.mcap", "rb") as f:
reader = make_reader(f)
for schema, channel, message in reader.iter_messages():
if(channel.topic=='color/compressed'):
print(channel)
print(schema)
print(message.data)
The Python code outputs all the metadata we have about the channel, message schema and the data content of the message. The latter of which is critical for our problem, this contains a compressed byte stream that represents our image frame!
This is an excellent first step, but how do we convert this byte stream data into something we can work with using common Python image processing packages?
The Bumpy Road to Success
Our mission wasn’t a smooth journey. It was riddled with unexpected twists and turns, including several failed attempts and dead ends. But instead of dwelling on those setbacks, let’s cut to the chase: what made things so challenging, and how did we overcome those obstacles?
The Painful Root Cause
Using the metadata previewed prior, we knew the video frames were in a JPEG format. Byte streams typically have hints about the underlying image format (IE JPEG images will have the hexadecimal header sequence ff\xd8\xff present at the start of the stream).
b'\x00\x00\x00\x00\x06\x07\x00\x00!)\x05\x00\x00\x00\x00\x00\x04\x00\x00\x00jpeg\xed\xf2\t\x00\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00
For the eagle eyed amongst you, you will notice the JPEG header sequence is present…BUT…its not present at the start of the byte stream. This observation unlocked our solution.
Our Novel Solution
This unexpected (and easy to miss!) nuance triggered an experiment. If we write a Python script to trim the byte stream to start at the JPEG, would it allow the byte to be decoded into a conventional image format (e.g Numpy ndarray) and banish the dreaded error messages we were battling with?
from PIL import Image
import numpy as np
import cv2
# Detect the JPEG header and edit the byte sequence!
start_index = data.find(b'\xff\xd8\xff')
header = image[start_index:start_index+3]
remaining_bytes = image[start_index+3:]
new_bytestream = b'\xff\xd8\xff' + remaining_bytes
# Attempt to decode the modified byte stream!
img = cv2.imdecode(np.frombuffer(new_bytestream, np.uint8), cv2.IMREAD_COLOR)
# Plot the image!
img = Image.fromarray(img.astype(np.uint8))
from matplotlib import pyplot as plt
plt.imshow(img, interpolation='nearest')
plt.show()
… Ta-da! 🎉
Appendix – A Common Issue
While we performed over a dozen experiments to decompress and load the data correctly, the issue we expect many to encounter can be seen with the code below.
import numpy as np
import cv2
img = cv2.imdecode(np.frombuffer(data, np.uint8), cv2.IMREAD_COLOR)
img
The problem with this approach, is that the img will be of NoneType. If you have this problem, our solution will be the breakthrough you are seeking!
