I would just link it with OpenCV (make it easy to use both together / OpenCV's data structs) than re-invent the wheel. It's already a super well-known library with tons of programmers and schools supporting it.
WARNING: IIRC, old OpenCV is C (which Python has bindings for), but the new OpenCV is C++ and heavily uses templates to make everything prettier so there's much fewer binding support for say, Python. They said the older OpenCV (2?) still works fine, but the new one was refactored to model "where the industry is moving" in terms of hardware.