This project started from a practical problem: building an MCMT (Multi-Camera Multi-Object Tracking) system using dual cameras on a physical access control device.
VIDEO
Reference
My first approach was straightforward:
Use two cameras
Apply triangulation
Estimate the 3D position of objects in space
To achieve this, we must map each image pixel to a 3D ray in the real world.
Use this image for example best described as:
Diagonal fisheye lens , orUltra wide-angle lens (>180° FOV)
These lenses require nonlinear projection models .
For fisheye and ultra wide-angle lenses, we need a more general model.
This is a widely used generic model for fisheye cameras.
P = [ x y z ] f = focal length d = x 2 + y 2 θ = arctan 2 ( d , z ) (angle from optical axis) φ = arctan 2 ( y , x ) r = { f tan ( θ ) (perspective) 2 f tan ( θ / 2 ) (stereographic) f θ (equidistant) 2 f sin ( θ / 2 ) (equisolid angle) f sin ( θ ) (orthographic) f ∑ i = 1 5 k i θ 2 i − 1 (polynomial general form) p = [ r cos ( φ ) r sin ( φ ) ] [ u v ] = [ f x p x + c x f y p y + c y ] \begin{aligned}
P &= \begin{bmatrix} x \\ y \\ z \end{bmatrix} \\
f &= \text{focal length} \\
d &= \sqrt{x^2 + y^2} \\
\theta &= \arctan2(d, z) \quad \text{(angle from optical axis)} \\
\varphi &= \arctan2(y, x) \\
\\
r &=
\begin{cases}
f \tan(\theta) & \text{(perspective)} \\
2f \tan(\theta/2) & \text{(stereographic)} \\
f \theta & \text{(equidistant)} \\
2f \sin(\theta/2) & \text{(equisolid angle)} \\
f \sin(\theta) & \text{(orthographic)} \\
f \sum_{i=1}^{5} k_i \theta^{2i-1} & \text{(polynomial general form)}
\end{cases}
\\
\\
p &=
\begin{bmatrix}
r \cos(\varphi) \\
r \sin(\varphi)
\end{bmatrix}
\\
\\
\begin{bmatrix}
u \\
v
\end{bmatrix}
&=
\begin{bmatrix}
f_x p_x + c_x \\
f_y p_y + c_y
\end{bmatrix}
\end{aligned} P f d θ φ r p [ u v ] = x y z = focal length = x 2 + y 2 = arctan 2 ( d , z ) (angle from optical axis) = arctan 2 ( y , x ) = ⎩ ⎨ ⎧ f tan ( θ ) 2 f tan ( θ /2 ) f θ 2 f sin ( θ /2 ) f sin ( θ ) f ∑ i = 1 5 k i θ 2 i − 1 (perspective) (stereographic) (equidistant) (equisolid angle) (orthographic) (polynomial general form) = [ r cos ( φ ) r sin ( φ ) ] = [ f x p x + c x f y p y + c y ] This model supports multiple projection types and is flexible enough to fit real fisheye lenses.
To perform triangulation, we need to reverse the projection:
p x = u − c x f x , p y = v − c y f y r = p x 2 + p y 2 φ = arctan 2 ( p y , p x ) θ = 2 arcsin ( r 2 f ) (example: equisolid projection) r a y = [ sin ( θ ) cos ( φ ) sin ( θ ) sin ( φ ) cos ( θ ) ] \begin{aligned}
p_x &= \frac{u - c_x}{f_x}, \quad
p_y = \frac{v - c_y}{f_y} \\
r &= \sqrt{p_x^2 + p_y^2} \\
\varphi &= \arctan2(p_y, p_x) \\
\theta &= 2 \arcsin\left(\frac{r}{2f}\right)
\quad \text{(example: equisolid projection)} \\
\mathbf{ray} &=
\begin{bmatrix}
\sin(\theta)\cos(\varphi) \\
\sin(\theta)\sin(\varphi) \\
\cos(\theta)
\end{bmatrix}
\end{aligned} p x r φ θ ray = f x u − c x , p y = f y v − c y = p x 2 + p y 2 = arctan 2 ( p y , p x ) = 2 arcsin ( 2 f r ) (example: equisolid projection) = sin ( θ ) cos ( φ ) sin ( θ ) sin ( φ ) cos ( θ ) This gives us a unit direction vector in 3D space corresponding to a pixel.
