Geometric Wide-Angle Camera Calibration: A Review and Comparative Study

Abstract

Wide-angle cameras are widely used in photogrammetry and autonomous systems which rely on the accurate metric measurements derived from images. To find the geometric relationship between incoming rays and image pixels, geometric camera calibration (GCC) has been actively developed. Aiming to provide practical calibration guidelines, this work surveys the existing GCC tools and evaluates the representative ones for wide-angle cameras. The survey covers the camera models, calibration targets, and algorithms used in these tools, highlighting their properties and the trends in GCC development. The evaluation compares six target-based GCC tools, namely BabelCalib, Basalt, Camodocal, Kalibr, the MATLAB calibrator, and the OpenCV-based ROS calibrator, with simulated and real data for wide-angle cameras described by four parametric projection models. These tests reveal the strengths and weaknesses of these camera models, as well as the repeatability of these GCC tools. In view of the survey and evaluation, future research directions of wide-angle GCC are also discussed.

1. Introduction

Cameras have been integral in various applications requiring metric measurements, such as remote sensing [1,2], cultural heritage [3], and robotics [4]. The image formation process involves two key components: the geometric aspect, which defines the relationship between world points and their corresponding image pixels, and the photometric aspect, which deals with the relationship between illumination and pixel colors. The calibration of the two components leads to geometric camera calibration (GCC) and color correction, respectively. For most of these applications, color correction is generally more linear and focused on appearance [5,6], whereas GCC is of greater importance due to its highly nonlinear nature. The GCC information includes the extrinsic component that transforms points from the object space to the camera coordinate system and the intrinsic component that projects rays in the camera frame to the imaging sensor. For remote-sensing applications, the extrinsic and intrinsic components are often unified into a generalized model for computation convenience [7]. For close-range applications, the two components are usually treated separately and GCC mainly refers to the intrinsic calibration.

Based on the angle of view (AOV), cameras can be roughly grouped [8,9] into conventional cameras, wide-angle cameras, fisheye cameras, and omnidirectional cameras (≥${180}^{\circ }$). The omnidirectional cameras include fisheye cameras with an AOV $\ge {180}^{\circ }$, and catadioptric cameras comprising of lenses and mirrors (“cata” for mirror reflection and “dioptric” for lens refraction). There are also camera rigs consisting of multiple cameras which achieve a great AOV by stitching images. Based on whether all incoming rays pass through a single point, cameras can be divided into central cameras of a single effective viewpoint, i.e., the optical center, and non-central cameras. Central cameras include the conventional cameras, fisheye cameras with an AOV ≤ 195° [10], and many catadioptric cameras built by combining a pinhole camera and hyperbolic, parabolic, or elliptical mirrors [11]. Analogous to [12], we define the term wide-angle cameras to include the wide-angle cameras with AOV < ${120}^{\circ }$, fisheye cameras, and the central catadioptric cameras. These are the prevalent cameras used in applications requiring metric information.

The GCC for these wide-angle cameras has been studied since the 1960s [13] and is still being actively researched today. This is reflected in the evolution of camera models from the classic parametric models to the generic models of thousands of parameters and deep neural networks. The parametric models are realized with a few specific parameters, e.g., focal length and radial distortion. Since each parameter affects the imaging of all pixels, they are also known as global models [11]. A parametric model is usually designed for a specific type of camera. These models are well supported by the existing calibration tools, and structure from motion (SfM) packages. The generic model has dense parameters and each parameter subset describes the imaging process in a local area, e.g., B-spline models [14] and per-pixel models [15]. A generic model is supposed to work with a wide range of cameras, but obviously requires more observations for calibration. Recently, the deep neural network has been widely used in image rectification, as surveyed in [12,16]. A neural network trained with lots of distorted and undistorted image pairs, captured by a variety of cameras, can regress a parametric camera model or estimate the distortion vector at every pixel for a new image or video clip. These deep camera calibration studies usually place attention on distortion removal. This paper focuses more on the repeatable parametric and generic models which are dominantly used in photogrammetry and robotics for accurate metric measurements.

Numerous tools have been developed for carrying out GCC, each with a unique set of features. They are often available as proprietary programs, such as the camera calibrator in MATLAB [17] and Agisoft Metashape [18], or open-source programs, such as Kalibr [19]. As for similarities, existing tools usually support global camera models and calibration with a specific planar target. Notably, many tools are based on the same underlying packages, e.g., OpenCV [20]; thus, they tend to have similar limitations. Moreover, many programs developed independently are very close in functionality, implying a possible duplicate effort. As for practical differences, these tools usually support different sets of camera models and calibration targets.

The diverse landscape of camera models and calibration tools on one hand offers ready-to-use solutions in a variety of situations, but on the other hand, it gets overwhelming for practitioners to choose the proper calibration tool. To address this difficulty, quite a few comparative studies have been conducted. For instance, three calibration algorithms were compared in [21] for cameras with large focal lengths. Digital displays and printed targets were compared in [22] for close-range cameras. Hughes et al. [23] reviewed distortion models for fisheye cameras with AOV < 180°. Both [24] and [25] surveyed calibration methods for omnidirectional cameras.

Considering the prevalence of wide-angle cameras, there is a lack of a qualitative overview and quantitative comparison of existing GCC tools which elucidate choosing the proper camera model and calibration tool. To fill this gap, we extensively review existing GCC tools for wide-angle cameras from several practical aspects and benchmark several popular tools with simulated and real data.

The contributions of this work are summarized as follows: First, this review categorizes camera models, calibration targets, and calibration algorithms as used in GCC tools for wide-angle cameras, providing a concise reference for these aspects. We then qualitatively reveal the strengths and similarities of these calibration tools, hopefully preventing repetitive development efforts in the future. Second, an evaluation of six representative calibration tools is conducted for in-house cameras with varying AOV by simulation and real-data tests to show their accuracy and repeatability. The evaluation clearly shows strengths and weaknesses of four popular global geometric camera models and indicates which calibration tool to use for wide-angle camera applications. Third, based on the review and evaluation, we highlight future research directions for wide-angle GCC.

The following text is organized as shown in Figure 1. Next, Section 2 briefly reviews related work on comparative studies of GCC. For the available camera calibration tools, Section 3 considers the camera models, the calibration targets, and the calibration algorithms. The GCC tools are reviewed in Section 4. Section 5 presents experiments of six calibration tools with a range of cameras and several popular global camera models. Finally, the conclusions and future research trends are given in Section 6.

2. Related Work

As shown in Figure 2, the geometric camera calibration process typically consists of four steps: data collection, target extraction, calibration optimization, and result verification. The target extraction is applicable when an artificial target is used, otherwise, it will be replaced by feature extraction and association. For the usual geometric wide-angle camera calibration, the process is characterized by three fundamental components: the camera model, the calibration target, and the calibration algorithm. As a result, this section briefly reviews comparative studies and surveys about wide-angle GCC from these aspects, with an emphasis on approaches based on geometric optimization. Deep learning-based camera calibration approaches are surveyed in [12,16].

2.1. Camera Models

Comparative studies about camera models are usually conducted in papers proposing new or enhanced models. For fisheye cameras, in [26], the double sphere (DS) model was proposed and compared with several global models including the Kannala–Brandt (KB) model [9], the extended unified camera model (EUCM) [27], the field of view (FOV) model [28], validating that its accuracy rivaled that of the KB model with eight parameters. In [29], a per-pixel generic model was shown to be more accurate than a pinhole camera model with radial distortion. The generic B-spline model [30] was enhanced in [14] with a denser grid of control points for the cubic B-spline surface, and it was shown that well-calibrated generic models led to more accurate results than global models in photogrammetric applications. The authors of [11,31] extensively reviewed existing camera models and established a taxonomy based on several criteria. In this paper, we survey the camera models commonly found in GCC tools and provide their exact formulations for reference (Section 3.1).

2.2. Calibration Targets

Though self-calibration has gained momentum over the years, e.g., [32,33,34], to ensure high accuracy, GCC can be performed with a set of points of known positions, such as a calibration field used in remote sensing, and calibration targets in close-range applications. The diversity of calibration targets made a comparative analysis of these targets necessary. Regarding control point detection in camera calibration, circle grids and checkerboards were studied in [35] and it was found that circles suffered from perspective and distortion biases, whereas corner points of checkerboards were invariant to the distortion bias. Schmalz et al. [22] systematically compared the active targets with digital displays to the printed checkerboard for GCC with several combinations of displays, cameras, and lenses. They found that calibration with the active target had much lower reprojection errors, but required compensation for the refraction of the glass plate and multiple images per pose, and hence a tripod or the like. In an underwater environment, fiducial markers including the ARToolKit [36], the AprilTag [37], and the Aruco [38] were compared in [39], where the AprilTag showed better detection performance but required higher computation. In environments with occlusions and rotations, three markers, the ARTag [40], the AprilTag [37], and the CALTag [41], were compared in [42] and the CALTag emprically achieved the best recognition rate. For pose tracking in surgery, Kunz et al. [43] compared the Aruco and AprilTag markers and found that both could achieve sub-millimeter accuracy at distances up to 1 m. For the localization of unmanned aerial systems, four fiducial markers, the ARTag, the AprilTag, the Aruco, and the STag [44], were compared in [45] in terms of detection rate and localization accuracy. The AprilTag, the STag, and the Aruco were shown to have close performance, whereas the Aruco was the most efficient in computation. For drone landing, several variants of the AprilTag and the circular WhyCode [46] were compared in [47] on an embedded system and the suitable variants were determined. For camera pose estimation, [48] proposed several multiplanar fiducial objects with custom-shaped markers, which showed favorable detectability to traditional fiducial objects with square markers. Unlike the above comparative studies about targets, our paper briefly surveys the calibration targets (Section 3.2) supported by the available GCC tools.

2.3. Calibration Algorithms

The algorithms for GCC are vast, ranging from target-based to self-calibration, from offline to interactive calibration, from sequence-based to single-image calibration. Quite a few papers have reviewed the GCC methods in view of different applications. For close-range photogrammetry, an overview of the developments of camera calibration methods up to 1995 was provided in [49]. Several calibration techniques up to 1992 for conventional cameras with a pinhole model were reviewed and evaluated in [50]. For close-range applications, several target-based and self-calibration methods were compared in [51] with a 3D target and a checkerboard, showing that the self-calibration methods based on bundle adjustment often achieved good calibration for consumer-grade cameras. For time-of-flight range cameras, three intrinsic calibration methods were compared in [52] for calibrating camera lens parameters and range error parameters by using a multi-resolution planar target. For cameras of large focal lengths (≥35 mm), Hieronymus [21] compared three calibration methods, one with a test field of a known geometric pattern, and two methods with devices for generating laser beams. He found that these methods achieved comparably high accuracy for the pinhole model with radial and tangential distortion. For cameras with lenses of focal lengths ≥ 50 mm in particle tracking velocimetry, Joshi et al. [53] studied the accuracy of three camera calibration methods, the direct linear transform (DLT) that ignores the distortion [54], a linear least squares method with the rational polynomial coefficient (RPC) model [55] but only using the numerator terms, and Tsai’s method, which determines the intrinsic and extrinsic parameters in two steps [56]. They found that the errors of the Tsai’s method were fluctuant due to the unstable nonlinear optimization. For omnidirectional cameras, four calibration methods were compared in [24] and shown to achieved similar calibration accuracy. For infrared cameras, Usamentiaga et al. [57] compared three calibration methods, a DLT method, an iterative method, and a complete method that considered lens distortion, and unsurprisingly, the last method resulted in best distance measurements. For roadside cameras, GCC methods based on vanishing points were compared in [58], assuming no lens distortion. For X-ray cameras ignoring radial distortion, the DLT method [59], Tsai’s method [56], and Zhang’s method [60] were compared in [61], and the DLT showed superiority in accuracy and operation simplicity. For a camera–projector pair, Tiscareno et al. [62] calibrated the camera with the DLT method, Tsai’s method, and Zhang’s method, and calibrated the projector with the DLT, through simulation. They found that Zhang’s method gave smaller reprojection errors than the others for camera calibration. For zoom-lens cameras with varying focal lengths, the calibration methods were reviewed in [63]. For wide-angle cameras, Yin et al. [64] compared a distortion estimation method using the image’s central data, Zhang’s method [60], and the distortion calibration in [65], and verified the advantage of lens distortion estimation isolated from overall camera calibration. Different from the preceding surveys and comparisons focusing on calibration methods, this paper reviews and compares GCC tools for wide-angle cameras of fixed intrinsic parameters.

3. Geometric Camera Calibration Components

This section reviews geometric camera models, targets, and algorithms as available in existing calibration tools, especially for wide-angle cameras.

Before elaborating on GCC, some definitions are clarified here. The focal length is defined to be the distance between the camera’s optical center and the sensor, as in [31]. The distance’s official name is principal distance, but the term focal length is widely used instead. Since the optical center is defined only for central cameras, the focal length is not defined for non-central cameras. Accordingly, the focal length can take a range of values, including the one when the camera is focused at infinity. We define the optical axis as the line passing through the optical center and orthogonal to the sensor chip. Its foot on the sensor chip is the principal point. For ease of use, with pinhole cameras, the sensor is often inverted and placed in front of the optical center, forming the image plane [59]. For a catadioptric camera, the mirror axis refers to the symmetry axis of the mirror. We define the AOV of a lens to be the maximum angle formed by rays coming into the lens. Likewise, the AOV of a camera is defined as the maximum angle formed by rays corresponding to the sensor’s exposed pixels, along the sensor’s horizontal, vertical, or diagonal axis, leading to HAOV, VAOV, or DAOV, respectively. Thus, the AOV of a camera depends on both the lens and the sensor.

3.1. Camera Models

The following describes the variety of camera models used in close-range applications, which have been adopted as the GCC tools surveyed in this paper. The camera models used for remote sensing, such as the RPC model [55] and the detector directional model [66], are referred to in [67]. We begin with global parametric models for central cameras, which dominate the GCC tools, and end with local generic models. These global models are typically defined in a (forward) projection manner, where image points are formulated and given world points or rays, although the same formulae may be used the other way round to obtain a ray given an image point, i.e., backward projection/back-projection/unprojection, for instance, (4) and (8). For local models, however, the backward projection is usually used to express the camera model, as the forward projection can be very complex [14]. For the below camera models listed in Figure 3, we describe either the forward or the backward model unless both are closed-form, with the understanding that going the other way often requires iterative optimization.

A set of symbols is defined in order here. We denote a point in the camera frame by ${\mathbf{x}}_c=[X_c,Y_c,Z_c]$ with Euclidean coordinates $X_c$, $Y_c$, and $Z_c$. The measured image point is denoted by ${\mathbf{u}}_m=[u_m,v_m]$ with pixel coordinates $u_m$ and $v_m$. The world-to-image forward projection is denoted by $\mathit{\pi }({\mathbf{x}}_c,\mathbf{i}):{\mathbb{R}}^3\to {\mathbb{R}}^2$, where $\mathbf{i}$ is the set of intrinsic parameters. Its inverse, the image-to-world inverse projection model is ${\mathit{\pi }}^{-1}({\mathbf{u}}_m,\mathbf{i}):{\mathbb{R}}^2\to {\mathbb{S}}^2$, where ${\mathbb{S}}^2$ is the set of 3D unit vectors. We denote by $\theta$ the incidence angle between an incoming ray and the optical axis. We use the subscripts ‘m’, ‘d’, ‘n’, and ‘c’ to indicate measurement, distortion, normalization, and the camera coordinate frame.

3.1.1. Global Models for Wide-Angle Cameras

Conventional and wide-angle cameras usually have little distortion and satisfy the pinhole model well. The set of parameters in the pinhole projection without distortion are $\mathbf{i}=[f_x,f_y,c_x,c_y]$, including the focal length and the principal point along the image plane’s two axes in units of pixels. The distortion-free pinhole model is given by

(1)${\mathbf{u}}_m=\mathit{\pi }({\mathbf{x}}_c,\mathbf{i})=\left[\begin{array}{c}{f}_x{X}_c/Z_c+c_x\\ f_y{Y}_c/Z_c+c_y\end{array}\right],$

with the closed-form inverse model,

(2)${\mathit{\pi }}^{-1}({\mathbf{u}}_m,\mathbf{i})={\displaystyle \frac{1}{\sqrt{x_n^2+y_n^2+1}}}\left[\begin{array}{c}{x}_n\\ y_n\\ 1\end{array}\right],$

where $x_n=(u_m-c_x)/f_x$ and $y_n=(v_m-c_y)/f_y$.

To account for lens distortion, a variety of distortion models for pinhole cameras have been proposed. A popular one is the radial–tangential polynomial model, i.e., the plumb bob model or the Brown–Conrady model [68]. Its intrinsic parameters, $\mathbf{i}=[f_x,f_y,c_x,c_y,k_1,k_2,p_1,p_2]$, include the pinhole projection parameters, the radial distortion parameters $k_j,j=1,2,\cdots ,p$ (the maximum index p is usually truncated to two in practice), and the tangential/decentering distortion parameters $p_1,p_2$. The pinhole radial–tangential model [20] is given by

(3)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_n\\ y_n\end{array}\right]& =\left[\begin{array}{c}{X}_c/Z_c\\ Y_c/Z_c\end{array}\right],r_n^2=x_n^2+y_n^2,\hfill \end{array}$

(4)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_d\\ y_d\end{array}\right]& =\left[\begin{array}{c}{x}_n(1+\sum _{j=1}^p{k}_j{r}_n^{2j})+{\delta }_{ud}\\ y_n(1+\sum _{j=1}^p{k}_j{r}_n^{2j})+{\delta }_{vd}\end{array}\right],\hfill \end{array}$

(5)$\begin{array}{cc}\hfill \left[\begin{array}{c}{\delta }_{ud}\\ {\delta }_{vd}\end{array}\right]& =\left[\begin{array}{c}2p_1{x}_n{y}_n+p_2(r_n^2+2x_n^2)\\ p_1(r_n^2+2y_n^2)+2p_2{x}_n{y}_n\end{array}\right],\hfill \end{array}$

(6)$\begin{array}{cc}\hfill \mathit{\pi }({\mathbf{x}}_c,\mathbf{i})& =\left[\begin{array}{c}{f}_x{x}_d+c_x\\ f_y{y}_d+c_y\end{array}\right].\hfill \end{array}$

This model usually suits lenses with an AOV < 120° well [26].

The inverse of (4) has no closed-form solution, and usually requires an iterative procedure. Notably, Drap and Lefèvre [69] propose an exact formula involving a power series to invert (4). Alternatively, the pinhole radial–tangential model can also be defined in a backward manner, i.e.,

(7)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_d\\ y_d\end{array}\right]& =\left[\begin{array}{c}(u_m-c_x)/f_x\\ (v_m-c_y)/f_y\end{array}\right],r_d^2=x_d^2+y_d^2,\hfill \end{array}$

(8)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_n\\ y_n\end{array}\right]& =\left[\begin{array}{c}{x}_d(1+\sum _{j=1}^p{k}_j{r}_n^{2j})+{\delta }_{ud}\\ y_d(1+\sum _{j=1}^p{k}_j{r}_n^{2j})+{\delta }_{vd}\end{array}\right],\hfill \end{array}$

(9)$\begin{array}{cc}\hfill \left[\begin{array}{c}{\delta }_{ud}\\ {\delta }_{vd}\end{array}\right]& =\left[\begin{array}{c}2p_1{x}_d{y}_d+p_2(r_d^2+2x_d^2)\\ p_1(r_d^2+2y_d^2)+2p_2{x}_d{y}_d\end{array}\right],\hfill \end{array}$

(10)$\begin{array}{cc}\hfill {\mathit{\pi }}^{-1}({\mathbf{u}}_m,\mathbf{i})& ={\displaystyle \frac{1}{\sqrt{x_n^2+y_n^2+1}}}\left[\begin{array}{c}{x}_n\\ y_n\\ 1\end{array}\right].\hfill \end{array}$

Obviously, for the same camera, the parameters of the backward model differ from those of the forward model. This backward model has been used in, e.g., the PhotoModeler [70].

The forward pinhole radial–tangential model in (4) can be simplified to the division model proposed by [71], which is a radial-symmetric model with the set of intrinsic parameters $\mathbf{i}=[f_x,f_y,c_x,c_y,k_1]$,

(11)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_d\\ y_d\end{array}\right]& =\left[\begin{array}{c}(u_m-c_x)/f_x\\ (v_m-c_y)/f_y\end{array}\right],r_d=\sqrt{x_d^2+y_d^2},\hfill \end{array}$

(12)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_n\\ y_n\end{array}\right]& =\left[\begin{array}{c}{x}_d/(1+k_1{r}_d^2)\\ y_d/(1+k_1{r}_d^2)\end{array}\right],\hfill \end{array}$

(13)$\begin{array}{cc}\hfill {\mathit{\pi }}^{-1}({\mathbf{u}}_m,\mathbf{i})& ={\displaystyle \frac{1}{\sqrt{x_n^2+y_n^2+1}}}\left[\begin{array}{c}{x}_n\\ y_n\\ 1\end{array}\right].\hfill \end{array}$

A backward rational model is proposed in [72],

(14)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_n\\ y_n\end{array}\right]& =\left[\begin{array}{c}{x}_d\\ y_d\end{array}\right]{\displaystyle \frac{1+{\sum }_{j=1}^p{k}_j^1{r}_d^{2j}}{1+{\sum }_{j=1}^q{k}_j^2{r}_d^{2j}}},\hfill \end{array}$

(15)$\begin{array}{cc}\hfill {\mathit{\pi }}^{-1}({\mathbf{u}}_m,\mathbf{i})& ={\displaystyle \frac{1}{\sqrt{x_n^2+y_n^2+1}}}\left[\begin{array}{c}{x}_n\\ y_n\\ 1\end{array}\right],\hfill \end{array}$

with the intrinsic parameters $\mathbf{i}=[f_x,f_y,c_x,c_y]\cup {\mathbf{k}}^1\cup {\mathbf{k}}^2$, where ${\mathbf{k}}^1=[k_j^1,j=1,2,\cdots ,p]$ and ${\mathbf{k}}^2=[k_j^2,j=1,2,\cdots ,q]$. The rational model in OpenCV [20] supports $p\le 3$ and $q\le 3$.

Furthermore, the thin prism effect is considered in [73] along with radial and tangential distortion, where the model is defined as

(16)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_n\\ y_n\end{array}\right]& =\left[\begin{array}{c}{X}_c/Z_c\\ Y_c/Z_c\end{array}\right],r_n^2=x_n^2+y_n^2,\hfill \end{array}$

(17)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_d\\ y_d\end{array}\right]& =\left[\begin{array}{c}{x}_n(1+k_1{r}_n^2)+{\delta }_{ud}+{\delta }_{up}\\ y_n(1+k_1{r}_n^2)+{\delta }_{vd}+{\delta }_{vp}\end{array}\right],\hfill \end{array}$

(18)$\begin{array}{cc}\hfill \left[\begin{array}{c}{\delta }_{up}\\ {\delta }_{vp}\end{array}\right]& =\left[\begin{array}{c}{s}_1{r}_n^2\\ s_2{r}_n^2\end{array}\right],\hfill \end{array}$

(19)$\begin{array}{cc}\hfill \mathit{\pi }({\mathbf{x}}_c,\mathbf{i})& =\left[\begin{array}{c}{f}_x{x}_d+c_x\\ f_y{y}_d+c_y\end{array}\right],\hfill \end{array}$

where the tangential distortion $[{\delta }_{ud},{\delta }_{vd}]$ is given in (5). Overall, the intrinsic parameter set is $\mathbf{i}=[f_x,f_y,c_x,c_y,k_1,p_1,p_2,s_1,s_2]$. The OpenCV considers more terms for the thin prism effect by ${\delta }_{up}=s_1{r}_n^2+s_2{r}_n^4$ and ${\delta }_{vp}=s_3{r}_n^2+s_4{r}_n^4$.

3.1.2. Global Fisheye Camera Models

Fisheye cameras typically have an AOV $\ge {100}^{\circ }$, and can reach 280° (https://www.back-bone.ca/product/entaniya-280/, accessed on 24 September 2024). They are quite common but show great distortion; thus, quite a few global models have been proposed as listed in [74,75]. The most popular ones are probably the KB model [9] and the FOV model [28].

The full KB model proposed in [9] has 23 parameters, where four describe the affine transform (6), five describe an equidistant radial-symmetric distortion, and the other 14 describe the asymmetric distortion. The commonly used KB-8 model is radially symmetric and has 8 intrinsic parameters, $\mathbf{i}=[f_x,f_y,c_x,c_y,k_1,k_2,k_3,k_4]$. It is defined by

(20)$\begin{array}{cc}\hfill \mathit{\pi }({\mathbf{x}}_c,\mathbf{i})& =\left[\begin{array}{c}{f}_{x}d\left(\theta \right)X_c/r_c+c_x\\ f_{y}d\left(\theta \right)Y_c/r_c+c_y\end{array}\right],\hfill \end{array}$

(21)$\begin{array}{cc}\hfill r_c& =\sqrt{X_c^2+Y_c^2}=Z_{c}tan\left(\theta \right),\hfill \end{array}$

(22)$\begin{array}{cc}\hfill d\left(\theta \right)& =\theta +k_1{\theta }^3+k_2{\theta }^5+k_3{\theta }^7+k_4{\theta }^9,\hfill \end{array}$

Unlike the KB-9 in [9], the KB-8 model sets the coefficient of the term $\theta$ in $d\left(\theta \right)$ to be 1. The KB-8 model can handle an AOV $\ge {180}^{\circ }$, but when it is formulated as an equidistant distortion on top of a pinhole projection, as in Kalibr [19] and OpenCV, the projection will fail for points of $Z_c\le 0$ [75].

The Scaramuzza model [10] for central catadioptric cameras and fisheye cameras up to a 195° AOV resembles the inverse of the KB-8 model. It is defined in a backward manner for a measured image point $[u_m,v_m]$ as

(23)$\begin{array}{cc}\hfill \left[\begin{array}{c}{u}_m\\ v_m\end{array}\right]& =\left[\begin{array}{cc}c& d\\ e& 1\end{array}\right]\left[\begin{array}{c}{u}_h\\ v_h\end{array}\right]+\left[\begin{array}{c}{c}_x\\ c_y\end{array}\right],\hfill \end{array}$

(24)$\begin{array}{cc}\hfill {\mathit{\pi }}^{-1}({\mathbf{u}}_m,\mathbf{i})& ={\displaystyle \frac{1}{\sqrt{u_h^2+v_h^2+w_h^2\left({\rho }_h\right)}}}\left[\begin{array}{c}{u}_h\\ v_h\\ w_h\left({\rho }_h\right)\end{array}\right],\hfill \end{array}$

(25)$\begin{array}{cc}\hfill {\rho }_h& =\sqrt{u_h^2+v_h^2},\hfill \end{array}$

(26)$\begin{array}{cc}\hfill w_h\left({\rho }_h\right)& =a_0+a_2{\rho }_h^2+a_3{\rho }_h^3+a_4{\rho }_h^4,\hfill \end{array}$

where $u_h$, $v_h$ are the ideal coordinates of the image point on a hypothetical plane orthogonal to the mirror axis. The parameter vector for the model is $\mathbf{i}=[a_0,a_2,a_3,a_4,c_x,c_y,c,d,e]$. Since c in the 2 × 2 stretch matrix is about one, $a_0$ is similar in role to $f_x$ or $f_y$ in (20). This model is available in the MATLAB camera calibrator [17]. For projecting a world point to the image, a polynomial approximation of the involved forward projection is adopted in [10] to reduce the computation.

The FOV model [28] has one distortion parameter and a closed-form inversion. It has been popular for fisheye lenses in consumer products, e.g., Tango phones. With intrinsic parameters $\mathbf{i}=[f_x,f_y,c_x,c_y,\omega ]$, its definition is given by

(27)$\begin{array}{cc}\hfill \mathit{\pi }({\mathbf{x}}_c,\mathbf{i})& =\left[\begin{array}{c}{f}_x{X}_c{\displaystyle \frac{r_d}{r_u}}+c_x\\ f_y{Y}_c{\displaystyle \frac{r_d}{r_u}}+c_y\end{array}\right],\hfill \end{array}$

(28)$\begin{array}{cc}\hfill r_u& =\sqrt{X_c^2+Y_c^2},\hfill \end{array}$

(29)$\begin{array}{cc}\hfill r_d& ={\displaystyle \frac{1}{\omega }}\arctan 2(2r_{u}tan{\displaystyle \frac{\omega }{2}},Z_c).\hfill \end{array}$

For the backward projection of an image point, the FOV model has a closed-form solution given by

(30)$\begin{array}{cc}\hfill {\mathit{\pi }}^{-1}({\mathbf{u}}_m,\mathbf{i})& ={\left[\begin{array}{ccc}{\displaystyle \frac{x_{d}sin\left(r_d\omega \right)}{2r_{d}tan\frac{\omega }{2}}}& {\displaystyle \frac{y_{d}sin\left(r_d\omega \right)}{2r_{d}tan\frac{\omega }{2}}}& cos\left(r_d\omega \right)\end{array}\right]}^{\mathsf{T}}\hfill \end{array}$

(31)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_d\\ y_d\end{array}\right]& =\left[\begin{array}{c}(u_m-c_x)/f_x\\ (v_m-c_y)/f_y\end{array}\right],\hfill \end{array}$

(32)$\begin{array}{cc}\hfill r_d& =\sqrt{x_d^2+y_d^2}.\hfill \end{array}$

Despite having only one distortion parameter, the FOV model often requires as much computation as the KB-8 model for forward and backward projections due to the trigonometric functions.

The DS model [26] fits large AOV lenses well, has a closed-form inversion, and does not involve trigonometric functions, thus making it very efficient. This model contains six parameters, $\mathbf{i}=[f_x,f_y,c_x,c_y,\xi ,\alpha ]$. In forward projection, a world point is projected consecutively onto two unit spheres of a center offset $\xi$, and lastly projected onto the image plane using a pinhole model. The projection model is defined by

(33)$\begin{array}{cc}\hfill \mathit{\pi }({\mathbf{x}}_c,\mathbf{i})& =\left[\begin{array}{c}{f}_x{\displaystyle \frac{X_c}{\alpha d_2+(1-\alpha )(\xi d_1+Z_c)}}+c_x\\ f_y{\displaystyle \frac{Y_c}{\alpha d_2+(1-\alpha )(\xi d_1+Z_c)}}+c_y\end{array}\right],\hfill \end{array}$

(34)$\begin{array}{cc}\hfill d_1& =\sqrt{X_c^2+Y_c^2+Z_c^2},\hfill \end{array}$

(35)$\begin{array}{cc}\hfill d_2& =\sqrt{X_c^2+Y_c^2+{(\xi d_1+Z_c)}^2}.\hfill \end{array}$

Its closed-form unprojection is given by

(36)$\begin{array}{cc}\hfill {\mathit{\pi }}^{-1}({\mathbf{u}}_m,\mathbf{i})& ={\displaystyle \frac{z_d\xi +\sqrt{z_d^2+(1-{\xi }^2)r_d^2}}{z_d^2+r_d^2}}\left[\begin{array}{c}{x}_d\\ y_d\\ z_d\end{array}\right]-\left[\begin{array}{c}0\\ 0\\ \xi \end{array}\right],\hfill \end{array}$

(37)$\begin{array}{cc}\hfill z_d& ={\displaystyle \frac{1-{\alpha }^2{r}_d^2}{\alpha \sqrt{1-(2\alpha -1)r_d^2}+1-\alpha }},\hfill \end{array}$

(38)$\begin{array}{cc}\hfill r_d^2& =x_d^2+y_d^2.\hfill \end{array}$

This model has been implemented in Basalt [26] and Kalibr.

3.1.3. Global Omnidirectional Camera Models

An omnidirectional camera has an HAOV $\ge {180}^{\circ }$ and a DAOV up to ${360}^{\circ }$. Several models have been developed for such cameras.

The unified camera model (UCM) in [76] can deal with both fisheye cameras and central catadioptric cameras, defined by

(39)$\begin{array}{cc}\hfill \mathit{\pi }({\mathbf{x}}_c,\mathbf{i})& ={\left[\begin{array}{cc}{\gamma }_x{\displaystyle \frac{X_c}{\xi \rho +Z_c}}+c_x& {\gamma }_y{\displaystyle \frac{Y_c}{\xi \rho +Z_c}}+c_y\end{array}\right]}^{\mathsf{T}},\hfill \end{array}$

(40)$\begin{array}{cc}\hfill \rho & =\sqrt{X_c^2+Y_c^2+Z_c^2},\hfill \end{array}$

with the intrinsic parameters $\mathbf{i}=[{\gamma }_x,{\gamma }_y,c_x,c_y,\xi ]$. When $\xi$ = 0, the above model degenerates to a pinhole model.

The unified model is formulated equivalently in [26] for better numeric stability. The formulation is given by

(41)$\mathit{\pi }({\mathbf{x}}_c,\mathbf{i})=\left[\begin{array}{c}{f}_x{\displaystyle \frac{X_c}{\alpha \rho +(1-\alpha )Z_c}}+c_x\\ f_y{\displaystyle \frac{Y_c}{\alpha \rho +(1-\alpha )Z_c}}+c_y\end{array}\right],$

with intrinsic parameters $\mathbf{i}=[f_x,f_y,c_x,c_y,\alpha ]$, where

(42)$\alpha =\xi /(1+\xi ),f_x={\gamma }_x/(1+\xi ),f_y={\gamma }_y/(1+\xi ).$

The unprojection function for the UCM is given by

(43)$\begin{array}{cc}\hfill {\mathit{\pi }}^{-1}(\mathbf{u},\mathbf{i})& ={\displaystyle \frac{\xi +\sqrt{1+(1-{\xi }^2)r_d^2}}{1+r_d^2}}\left[\begin{array}{c}{x}_d\\ y_d\\ 1\end{array}\right]-\left[\begin{array}{c}0\\ 0\\ \xi \end{array}\right],\hfill \end{array}$

(44)$\begin{array}{cc}\hfill x_d& ={\displaystyle \frac{u_m-c_x}{f_x(1+\xi )}},y_d={\displaystyle \frac{v_m-c_y}{f_y(1+\xi )}},\hfill \end{array}$

(45)$\begin{array}{cc}\hfill r_d^2& =x_d^2+y_d^2,\xi ={\displaystyle \frac{\alpha }{1-\alpha }}.\hfill \end{array}$

For better accuracy with the UCM, Mei and Rives [77] also consider the lens distortion, the misalignment and the sensor skew. The Mei model is defined by

(46)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_n\\ y_n\end{array}\right]& =\left[\begin{array}{c}{X}_c/(Z_c+\xi \rho )\\ Y_c/(Z_c+\xi \rho )\end{array}\right],r_n=\sqrt{x_n^2+y_n^2},\hfill \end{array}$

(47)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_d\\ y_d\end{array}\right]& =\left[\begin{array}{c}{x}_{n}d\left(r_n\right)+2p_1{x}_n{y}_n+p_2(r_n^2+2x_n^2)\\ y_{n}d\left(r_n\right)+p_1(r_n^2+2y_n^2)+2p_2{x}_n{y}_n\end{array}\right],\hfill \end{array}$

(48)$\begin{array}{cc}\hfill d\left(r_n\right)& =1+k_1{r}_n^2+k_2{r}_n^4+k_3{r}_n^6,\hfill \end{array}$

(49)$\begin{array}{cc}\hfill \mathit{\pi }({\mathbf{x}}_c,\mathbf{i})& =\left[\begin{array}{c}{\gamma }_x(x_d+s{y}_d)+c_x\\ {\gamma }_y{y}_d+c_y\end{array}\right],\hfill \end{array}$

with the intrinsic parameters $\mathbf{i}=[{\gamma }_x,{\gamma }_y,c_x,c_y,\xi ,k_1,k_2,k_3,p_1,p_2,s]$, where $k_1$, $k_2$, and $k_3$ are for radial distortion, $p_1$ and $p_2$ for misalignment, and s for skew. This model is adopted in [78] and Camodocal [79]. As pointed out in [27], $k_1$ of the Mei model is redundant with $\xi$.

The extended unified camera model (EUCM) [27] enhances the UCM by a parameter $\beta$ to deal with the radial distortion. Its projection model is given by

(50)$\begin{array}{cc}\hfill \left[\begin{array}{c}{u}_m\\ v_m\end{array}\right]& =\left[\begin{array}{c}{f}_x{\displaystyle \frac{X_c}{\alpha \rho +(1-\alpha )Z_c}}+c_x\\ f_y{\displaystyle \frac{Y_c}{\alpha \rho +(1-\alpha )Z_c}}+c_y\end{array}\right],\hfill \end{array}$

(51)$\begin{array}{cc}\hfill \rho & =\sqrt{\beta (X_c^2+Y_c^2)+Z_c^2},\hfill \end{array}$

with parameters $\mathbf{i}=[f_x,f_y,c_x,c_y,\alpha ,\beta ]$, where $\alpha \in [0,1]$, $\beta >0$, and $\alpha \rho +(1-\alpha )Z_c>0$. The unprojection function for the EUCM is given by

(52)$\begin{array}{cc}\hfill {\mathit{\pi }}^{-1}(\mathbf{u},\mathbf{i})& ={\displaystyle \frac{1}{\sqrt{x_d^2+y_d^2+z_d^2}}}\left[\begin{array}{c}{x}_d\\ y_d\\ z_d\end{array}\right],\hfill \end{array}$

(53)$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_d\\ y_d\end{array}\right]& =\left[\begin{array}{c}(u_m-c_x)/f_x\\ (v_m-c_y)/f_y\end{array}\right],r_d^2=x_d^2+y_d^2,\hfill \end{array}$

(54)$\begin{array}{cc}\hfill z_d& ={\displaystyle \frac{1-\beta {\alpha }^2{r}_d^2}{\alpha \sqrt{1-(2\alpha -1)\beta r_d^2}+1-\alpha }}.\hfill \end{array}$

3.1.4. Local Generic Camera Models

The preceding global camera models are available in a variety of GCC tools, possibly due to their simplicity, but they are mostly for a specific type of cameras. To work with a wide range of cameras, generic models with thousands of parameters have been proposed, such as [30,80]. They are still behind the global models in terms of availability among GCC tools and support by downstream applications.

We briefly describe two generic models implemented in [14], a per-pixel model and a B-spline model. The per-pixel model of [15] associates a ray direction to every pixel for a central camera and a ray direction and a 3D point on the ray to every pixel for a non-central camera. Furthermore, interpolation between pixels is used to achieve continuous projection. A B-spline model adopted in [14] associates ray parameters to a sparse set of grid points instead of all pixels. These grid points control the cubic B-spline surface, which represents the back projection function. Notably, this B-spline model is initialized using the relative camera poses, computed with the method [15] developed for the per-pixel model. In general, a per-pixel model demands more data, computation, and storage compared to a B-spline-based model.

3.2. Calibration Targets

GCC usually depends on passive or active man-made objects, e.g., ground control points in remote sensing or planar targets in close-range calibrations. Recent self/auto-calibration methods, e.g., [81,82], use opportunistic environmental features, whereas infrastructure-based methods [79] use a prior landmark map of the environment. Since artificial targets are still commonly used for better accuracy control, this section surveys the targets supported by GCC tools, as listed in Figure 4.

There are a few 3D targets, such as cubes [83] and icosahedrons [84], each of which is usually a composite of multiple planar targets. The accuracy requirements of length and orthogonality complicate their manufacturing and hamper their accessibility. The majority of calibration targets are planar, including surveyed markers on flat walls, and a variety of coded patterns either displayed on digital screens [22,85,86] or printed out. The targets based on digital displays usually have accurate size and good flatness and can deal with defocusing [86,87], but such a target usually requires capturing multiple pattern images at each pose and compensating the refraction of the display’s glass plate.

So far, the printed boards are the most common targets and are widely supported by GCC tools. They include the checkerboard, the AprilGrid [19], the circle grid, the Charuco [38] board, and the recent deltille board [84], etc., as shown in Figure 5. Their properties are briefly described below. There are also numerous customized calibration targets tailored for specific algorithms, e.g., the random pattern aggregated from noise at multiple scales in [78], the pattern in [14] with dense corners for generic models, the Ecocheck board [88], the PhotoModeler circle board [70]. A custom board can often be created by combining markers to disambiguate orientations, e.g., the AprilTag, and corners invariant to perspective and lens distortion, e.g., formed from repeating squares. Lists of fiducial markers resilient to rotation can be found in [38,42].

3.2.1. Checkerboard

The checkerboard is probably the most common calibration target. It is also known as chessboard. We prefer the name checkerboard which is more general than chessboard. Many checkerboard detection improvements have been proposed, such as [79,89]. The checkerboard requires that the corners inside the board are fully visible in an image so that their coordinates can be uniquely determined. Though this weakness is reported to be remedied by a few recent methods [84,90,91,92], most current tools have not kept up. To ensure that the pattern does not look the same after a 180° rotation, a checkerboard with odd rows and even columns or even rows and odd columns is usually used.

3.2.2. Circle Grid

A circle grid [93] usually consists of an array of circles, symmetrically or asymmetrically distributed (see Figure 5). The circle centers are target points for calibration, and can be detected from images based on area, circularity, convexity, inertia (https://learnopencv.com/blob-detection-using-opencv-python-c/, accessed on 24 September 2024), etc. The circle grid has several downsides: first, all circles should be visible in each image; second, the detected circle centers suffer from the eccentricity error due to the perspective effect and lens distortion [35]. The eccentricity error is worth attention especially for lenses of large distortion. Moreover, the symmetric circle grid also has 180° ambiguity, and thus the asymmetric circle grid is generally preferred.

3.2.3. Charuco

The Charuco board [83] combines the checkerboard and the Aruco tags [38] to deal with inaccurate corner positions and occlusions. As shown in Figure 5d, the white squares of checkerboards are occupied by uniquely identifiable Aruco tags.

3.2.4. AprilGrid

The AprilGrid is an array of AprilTag markers [37] connected by smaller black squares, as shown in Figure 5e, developed in the Kalibr package [19]. It is resilient to occlusion due to the AprilTag markers, and has accurate positions of corners which are surrounded by two black squares.

3.2.5. Deltille Grid

The Deltille grid is a pattern of adjacent regular triangles filled with alternating colors, as shown in Figure 5f. It is the only other possible tiling with alternating colors besides the checkerboard tiling. Its benefits compared to checkerboards are higher corner density and more accurate corner positions (triple intersections). The wide use of Deltille grids is mainly hindered by the effort to adapt the interfaces of existing calibration tools.

Overall, for these planar targets, achieving accurate calibration relies heavily on high-quality printing and ensuring that the surface remains flat and non-deformable. During the target extraction phase, employing subpixel detection methods is also preferred. Given the importance of extraction accuracy and the availability of extraction algorithms, we recommend using checkerboards, Charuco boards, or AprilGrids for geometric calibration.

3.3. Calibration Algorithms

This section gives a high-level overview of the calibration algorithms as implemented in GCC tools. According to the used solver, GCC algorithms can be grouped into traditional geometric and learning-based ones. Generally speaking, geometric approaches are explainable and accurate, whereas the learning-based approaches are intended to be more robust and flexible, e.g., using neural radiance fields [94], transformers [95], and genetic algorithms [96].

A calibration algorithm minimizes an objective function to derive the optimal set of camera parameters. The objective function quantifies the discrepancy between actual measurements and the predictions made by the camera model. The discrepancy can be defined on the imaging plane, in the photometric space, or in the feature space, leading to reprojection residuals, photometric residuals, and feature residuals, respectively.

Most GCC tools rely on reprojection residuals. For a point ${\mathbf{P}}_i\in {\mathcal{R}}^3$ observed in image k, the reprojection residual ${\mathbf{r}}_{ik}$ is the difference between the projected point in the image ${\mathbf{p}}_{ik}$ and its measurement ${\mathbf{m}}_{ik}$,

(55)${\mathbf{p}}_{ik}=\mathit{\pi }({\mathbf{R}}_k{\mathbf{P}}_i+{\mathbf{t}}_k,\theta ),{\mathbf{r}}_{ik}={\mathbf{p}}_{ik}-{\mathbf{m}}_{ik}\in {\mathcal{R}}^2$

where $\mathit{\theta }$ represents the camera intrinsic parameters, and ${\mathbf{R}}_k$ and ${\mathbf{t}}_k$ are the orientation and translation of the world frame relative to the camera.

Photometric residuals and feature residuals are also employed in camera calibration, e.g., [97,98]. Let us define a D-dimensional feature map, which could be the RGB image ($D=3$) or the multiple layers resulting from a convolutional neural network. The photometric and feature residuals can then be unified and defined between two feature maps ${\mathbf{F}}_q$ and ${\mathbf{F}}_k$ as

(56)${\mathbf{r}}_{ik}={\mathbf{F}}_q\left[{\mathbf{p}}_{iq}\right]-{\mathbf{F}}_k\left[{\mathbf{p}}_{ik}\right]\in {\mathcal{R}}^D,$

where $\left[·\right]$ denotes lookup in the feature maps. With these residuals, the objective function is expressed as

(57)$L(\mathit{\theta },\{{\mathbf{R}}_i,{\mathbf{t}}_i\})=\sum _{i,k}\rho \left({∥{\mathbf{r}}_{ik}∥}^2\right)$

where $\rho$ is a robust loss function.

According to the type of calibration targets, GCC algorithms can be grouped into those based on artificial targets, those based on mapped natural scenes, and self-calibration algorithms without targets. Calibration with an artificial target is pretty standard and widely supported in GCC packages. It is typically offline, and usually involves two phases, linear initialization and iterative nonlinear refinement. The instances of linear initialization are DLT, [15,99]. Iterative refinement is exemplified by [19,56,60,100,101]. We refer to [50] for an overview of artificial-target-based methods.

Calibration with objects of known geometry includes line-based undistortion methods [102,103,104,105], infrastructure-based calibration methods [79,106]. The goal of line-based methods is usually image rectification given a single image. The infrastructure-based methods require an accurate 3D reconstruction of the site for calibration, and rough values for intrinsic parameters and are suitable for camera systems with motion constraints.

Broadly speaking, camera self-calibration by using observations of opportunistic landmarks includes recursive refinement methods, methods that recover only camera intrinsic parameters, and methods that recover structure, motion and camera intrinsic parameters [32]. Methods in the first group recursively refine calibration parameters and have to start from coarse parameter values, e.g., [81,107]. The second group includes methods like [32,108] and is reviewed in [109]. Methods in the last group usually rely on bundle adjustment; thus, they typically have the best accuracy among self-calibration methods and are commonly supported in SfM packages, e.g., colmap [110].

4. GCC Tools

This section reviews tools developed for wide-angle camera calibration. These tools mainly implement algorithms using artificial targets or target-free bundle adjustment. Several learning-based GCC tools are also cited as examples from this active research field. Since our focus is on intrinsic calibration, tools solely for extrinsic calibration are left out, e.g., [106,111,112]. An extensive list of GCC tools to our knowledge is given in Table 1. For brevity, the table only lists a few photogrammetric software tools which unanimously allow self-calibration. This table can serve as a reference in choosing a proper GCC tool and hopefully can help prevent the duplication of development efforts.

We assess a GCC tool based on characteristics which are grouped into accessibility and quality evaluation. For accessibility, these characteristics include supported camera models and targets, stereo/multiple camera support, the user interface, source availability, and the coding language. Usually, a graphical user interface (GUI) is more accessible than a command line interface to an average user. When a tool is open-source or modular, it is easy to extend it to other camera models and calibration targets. The coding language usually implies the execution efficiency and the community support.

From quality evaluation, we look at the outlier strategy and the availability of covariance output. The outlier strategy dictates how to handle outliers in detected corners which may deviate from their true positions by a few pixels. For the quality check, all calibration tools output some metric based on reprojection errors, such as the mean reprojection error and the root mean square (RMS) reprojection error. However, these metrics are highly dependent on the used image corners, and thus are inadequate to compare results from different methods [119]. The covariance output is an quality indicator in addition to these metrics, and directly links to the correlation analysis [125]. Next, we describe several popular calibration tools in terms of these characteristics.

4.1. BabelCalib

The monocular camera calibrator, BabelCalib (https://github.com/ylochman/babelcalib, accessed on 24 September 2024), employs a back-projection model as a proxy for a variety of radial-symmetric forward camera models, including the pinhole radial distortion model (4), DS (33), EUCM (50), FOV (27), KB-8 (20), and UCM (39). In practice, the back-projection model, a two-parameter division model of even degrees (12), can be obtained by linear solvers, and then the desired camera models can be regressed from the division model. BabelCalib is agnostic to the calibration targets, supports calibration with multiple targets, and handles outliers with the Huber loss.

4.2. Basalt

The Basalt package (https://gitlab.com/VladyslavUsenko/basalt, accessed on 24 September 2024) [26] can carry out monocular camera calibration, supporting camera models including DS, EUCM, FOV, KB-8, and UCM. Its default calibration target is the AprilGrid. A Levenberg–Marquardt algorithm is implemented in Basalt for robust calibration with the Huber loss. With the neat use of C++ templates, it is a lean and fast tool.

4.3. calio.io

As shown in Figure 6a, the commercial calibration tool by calio.io comes with an intuitive GUI, supports a variety of camera models, including the pinhole rational radial–tangential model with the thin prism effect (19), the division model (12), DS, KB-8, FOV, EUCM, and a B-spline camera model, and supports many calibration targets, including the checkerboard and the Charuco board. Moreover, it allows for calibrating multiple cameras with multiple targets, optimizing the target points to deal with board deformation, and deals with outliers with the Huber loss.

4.4. Camodocal

The Camodocal package (https://github.com/hengli/camodocal, accessed on 24 September 2024) supports monocular and stereo GCC with models including the pinhole radial–tangential model (4), KB-8, and Mei (49). By default, it supports the checkerboard, but it is relatively easy to extend to other targets. It uses the Cauchy loss to deal with outliers.

4.5. Kalibr

Kalibr (https://github.com/ethz-asl/kalibr, accessed on 24 September 2024) is a popular GCC tool that can select informative images for calibration [19]. It supports projection models, including pinhole projection (1), UCM, EUCM, and DS, and distortion models including radial–tangential distortion, equidistant distortion, and FOV. As mentioned for (20), the KB-8 model in Kalibr discards points of non-positive depth $Z_c$. The supported targets include checkerboards and AprilGrids. Outliers are handled by removing corners of reprojection errors exceeding a certain threshold. This tool has been extended to deal with the rolling shutter effect [126], and to better detect corners in images of high distortion lenses [119].

4.6. MATLAB Camera Calibrator

The MATLAB camera calibrator [17] supports both monocular and stereo camera calibration with both the pinhole radial–tangential model (4) and the Scaramuzza model (24). It can be seen as a superset of [127] and [10]. The supported targets by default are checkerboards, circle grids, and AprilTag grids. With its modular design, it is easy to use other calibration targets, e.g., the AprilGrid. As shown in Figure 6b, the MATLAB calibrator has an easy-to-follow GUI and many visualization functions.

4.7. ROS Camera Calibrator

The OpenCV library provides functions for calibrating monocular and stereo cameras with the pinhole rational radial–tangential model with the thin prism effect (19), the KB-8 model for fisheye cameras, and the Mei model for omnidirectional cameras. The omnidirectional module in OpenCV also supports a multi-camera setup and can be seen as a reimplementation of the MATLAB tool in [78]. The current KB-8’s realization in OpenCV does not support points of non-positive depth. The calibration functions in OpenCV do not have outlier handling schemes, but its omnidirectional module removes images of large total reprojection errors in calibration.

Several programs have been developed on top of OpenCV, such as the ROS camera calibrator [123] and the MRPT camera calibrator [122]. The ROS camera calibrator is a thin wrap of OpenCV calibration functions which can run in both interactive and batch mode, and supports checkerboards, circle grids, and Charuco boards. The GUI of the ROS camera Calibrator, as shown in Figure 6c, is simple and clear, making it suitable for beginners. Besides wrapping the OpenCV functions, the MRPT camera calibrator extends the checkerboard detection to support multiple checkerboards.

4.8. Self-Calibration Tools with SfM

Self-calibration is usually based on a SfM pipeline which is realized in commercial software or open source programs. For space, we limit the discussion to several representatives of the two groups. Professional photogrammetric packages usually support self-calibration, for instance, the Metashape by Agisoft [18], the calibrator in PhotoModeler [70], and the Pix4D mapper [82]. The Metashape realizes both checkerboard-based calibration and self-calibration using natural landmarks within its SfM pipeline as shown in Figure 6d. Both methods support the pinhole radial–tangential model and a customized fisheye model that is made of the equidistant projection and the radial–tangential distortion. The calibration tool in PhotoModeler adopts the inverse pinhole radial–tangential model (8), and supports target-based calibration with either multiple boards each of five RAD (Ringed Automatically Detected) tags or a single board of a circle grid with four non-ringed coded tags. When the scene to be reconstructed is much larger than the printed targets, a self-calibration of the camera in the field may be conducted with PhotoModeler. The Pix4D mapper can also estimate the camera’s intrinsic parameters with a collection of images of natural scenes. It supports the pinhole radial–tangential model and an adapted Scaramuzza model.

The open-source SfM packages also widely support camera self-calibration, such as the popular colmap, and the recent self-calibration package based on the Neural Radiance Field, SCNeRF [94]. Based on geometric bundle adjustment, colmap [110] supports camera models including the pinhole radial–tangential model with the thin prism distortion, KB-8, and FOV. The learning-based SCNeRF considers both geometric and photometric consistency in constructing the implicit scene geometry and estimating the camera parameters.

5. Evaluation of Target-Based GCC Tools

This section evaluates six popular target-based GCC tools on simulated and real wide-angle camera data acquired by cameras of varying AOVs, to show their extensibility and repeatability.

5.1. Data Acquisition

The real data were captured by an UI-3251LE-M-GL camera of a 1/1.8” sensor from IDS Imaging, fitted with six fixed-focus lenses listed in Table 2, leading to varying camera DAOVs from 90° to 194° as shown by the sample images in Figure 7. Notably, in focal length, the 90° lens resembles lenses on smartphones, whose actual focal lengths are about 4 mm. Also, empirically, the calibrated focal lengths are close to the physical focal lengths (from datasheets) in Table 2 in pixels. The camera can capture grayscale images at 25 frames/second and a resolution of 1600 × 1200 in global shutter mode. Prior to data capture, the exposure time was set to 5 ms to reduce motion blur. For each lens, the camera was gently moved in front of an AprilGrid, passing through a variety of poses. We chose the AprilGrid since it is accurate [39], widely used, and resilient to occlusions, among the reviewed calibration targets. Three sequences of a minute each were recorded for each lens. From each sequence, three subsequences of 200 frames each were uniformly drawn without replacement. This resulted in $54=6\times 3\times 3$ calibration sequences for six lenses.

We evaluated six GCC tools on Ubuntu 20.04, including BabelCalib [113], Basalt [26], Camodocal [79], TartanCalib [119] (Kalibr with enhanced corner detection), the Matlab calibrator [17], and the ROS calibrator [123] based on OpenCV. which were chosen for their widespread use and easy extension to an alternative type of target and data input. Within these tools, we evaluated several camera models, the pinhole model with the radial–tangential distortion for wide-angle cameras, KB-8 for fisheye cameras, and Mei / EUCM for omnidirectional cameras, which were chosen mainly for their wide support by GCC tools and downstream applications. The test plan is shown in Table 3, which lists GCC tools and camera models for processing particular data. In general, the pinhole model with distortion was used for cameras with a DAOV <120°, KB-8 for cameras with a DAOV ≥ 100°, and Mei / EUCM for cameras with a DAOV ≥ 120°.

The simulation data were generated from the real data with the workflow shown in Figure 8 (bottom). We first processed the real data by the TartanCalib with proper models according to the test plan. Thus, we obtained the frames of detected corners and their poses, and the estimated calibration parameters, from TartanCalib. As an exception, for simulating observations of the KB-8 model on MTV185 sequences, we first processed them by TartanCalib with the Mei model to obtain the frame poses, and then estimated the KB-8 parameters by Camodocal on the used corners by TartanCalib. In any case, these frame poses and camera parameters were then used to simulate the corners in images by projecting the target landmarks and adding a Gaussian noise of 0.7 px at both x- and y-axes. These camera parameters served as the reference in evaluation.

5.2. Data Processing

For either real or simulated data, the evaluation pipeline is shown in Figure 8 (top). To facilitate a better comparison, all tools except for the ROS calibrator used the same corners. Specifically, we first ran TartanCalib on a (real or simulated) sequence, and saved the frames with detected corners and marked the frames used by TartanCalib. The TartanCalib was chosen to extract corners from AprilGrid images since it could identify sufficient corners under large distortion [119]. All the frames of the corners were provided to the ROS calibrator. But only the frames of corners used by TartanCalib were supplied to the four methods, BabelCalib, Basalt, Camodocal, and the MATLAB calibrator. For these tools, we wrote the necessary data-loading functions and adapted the calibration initialization with the AprilGrid if needed. Note that TartanCalib/Kalibr always failed for the MTV185 sequences; we gave these four tools the corners of TartanCalib with the Mei model for these sequences.

Figure 8

(Top) The calibration workflow for either real data or simulated data. (Bottom) The workflow to simulate image points from real data.

[Figure omitted. See PDF]

There were several other reasons behind feeding the four tools by TartanCalib. First, empirically, Kalibr usually chose ≤ 40 informative frames for calibration. This coincided with the assertion that global camera models were usually well-constrained, with 40 frames in [29]. Second, BabelCalib often failed to find a solution with too many frames (e.g., ≥100), especially for the pinhole model with radial distortion. Third, the MATLAB calibrator took up to an hour to solve for the Scaramuzza model with 100 frames.

For the ROS calibrator, we ran it five times, each with a sample of 40 randomly chosen frames without replacement, and kept the run of the minimum RMS reprojection error as the final result. The exclusive treatment of the ROS calibrator was because the OpenCV calibration functions hardly dealt with outliers and often gave poor results on the corners used by TartanCalib.

Apart from the above, we ran these six calibration tools with their default parameter settings.

A test run was considered failed if no solution was found or the recovered focal lengths deviated from the nominal values (for real data) or the reference (in simulation) by ≥100 px. Failures in the real and simulated tests are marked in Table 3, for BabelCalib, Basalt, Kalibr, and the ROS calibrator. BabelCalib failed once, when it converged to a wrong focal length. Basalt failed due to either converging to a wrong focal length or being unable to converge in 100 iterations. Kalibr’s failures were due to the poorly implemented equidistant model for large FOV cameras. The ROS calibrator was affected by outliers and largely unsuccessful on MTV185 sequences for the poorly implemented pinhole equidistant model. Oddly, it always aborted with ill-conditioned matrices on sequences of 103° and 127° DAOV cameras, perhaps unable to initialize in such cases.

Next, we evaluated the GCC tools by looking at the consistency of estimated camera parameters and the RMS reprojection errors for both simulated and real data. The RMS reprojection errors are computed by these tools on all inlier observations. The RMS values should be viewed tentatively when comparing across tools since the inlier sets may vary slightly even for the same data.

5.3. Simulation Results

The simulated data were processed as described above. The data from cameras with S04525, E1M3518, and BM4218 lenses were processed by five tools with the pinhole radial–tangential model, except Basalt, which did not support the model. The camera parameter errors and the RMS reprojection errors are shown in Figure 9, where the failed tests were excluded when drawing the box plots. The units are specified in parentheses for all box-plot figures. These tools generally gave very similar results, close to the reference values. The focal lengths and principal points were usually within (−2, 2) px of the true values. Since BabelCalib did not consider the tangential distortion, its estimates had larger errors than other methods, especially for BM4218 sequences of 103° DAOV. The RMS reprojection errors slightly above 0.9 resulted from the Gaussian noise of $\sigma =\sqrt{2}\times 0.7=0.99$. The ROS calibrator based on OpenCV had slightly larger error dispersions, likely due to corners of large reprojection residuals. For a BM4218 sequence, the MATLAB calibrator converged to a focal length off by 37 px for no apparent reason.

For cameras with a DAOV $\ge {100}^{\circ }$, the KB-8 model was solved for by using five tools, except for the MATLAB calibrator, which does not support KB-8. The parameter errors and RMS errors are shown in Figure 10. The ROS calibrator results for the BM4218, BM4018, and MTV185 lenses, and the MATLAB results for the MTV185 lens, were excluded for the consistent failures explained in Section 5.2. Among these tools, we can see that the Basalt and the OpenCV-based ROS calibrator sometimes converged to focal lengths of large errors >5 px. Other tools consistently estimated the focal lengths and principal points within (−2, 2) px as well as the distortion parameters.

For sequences with the lenses BM4018, BT2120, and MTV185, three tools including Kalibr, Camodocal, and the ROS/OpenCV calibrator were used to solve for the Mei parameters. For comparison, we also solved for the EUCM model by BabelCalib and Basalt. The parameter errors and reprojection errors are shown in Figure 11, where we used $(f_x,f_y)$ instead of $({\gamma }_x,{\gamma }_y)$, as the latter has large variance caused by $\xi$. For both the BM4018 and BT2120 sequences, the three methods with the Mei model gave similar results. Overall, Kalibr gave the best estimates, notably on the MTV185 sequences. The ROS calibrator tended to have larger variances in focal lengths and principal points but their errors were within (−2, 2) px. For the MTV185 sequences, the Camodocal and OpenCV results showed about 3 px errors in focal lengths and about 0.7 errors in $\xi$, but with reasonable RMS errors. We attribute this to two reasons. First, the UCM model is numerically unstable. Second, $k_1$ in the Mei model is redundant. As for the EUCM models, the BabelCalib achieved smaller dispersions in $f_x$ and $f_y$ than Basalt, although the data were simulated with the Mei model.

5.4. Real Data Results

We processed the real data according to Table 3 and analyzed the estimated parameters and RMS reprojection errors. For clarity, the nominal focal lengths from Table 2 and the coordinates (800, 600) are subtracted from their estimates in plots.

The sequences of S04525, E1M3518, and BM4218 lenses were processed by five tools, except for Basalt. The calibration parameters and the RMS reprojection errors are shown in Figure 12, where the failed cases are not included in the box plots. These five tools had fairly similar results. The difference in principal points for BabelCalib was caused by its model, which ignored tangential distortion. The large dispersion in projection parameters of BM4218 sequences was likely because the pinhole radial–tangential model was somewhat improper for the camera with this lens, as implied in Figure 13.

For the KB-8 model, the sequences of BM4218, BM4018, BT2120, and MTV185 lenses were processed by five tools except the MATLAB calibrator. As shown in Figure 13, these tools achieved very similar calibration results in general. Basalt failed frequently for the BM4218 and BT2120 sequences, leading to apparent small parameter dispersions. Comparing the dispersion of focal lengths for BM4218 in Figure 12 and Figure 13, we think that the KB-8 model is more suitable than the pinhole radial–tangential model for the BM4218 sequences. The OpenCV-based ROS calibrator aborted on the BM4218 and BM4018 sequences, perhaps due to failed initialization. Both Kalibr and the ROS calibrator did not handle MTV185 sequences of a ${194}^{\circ }$ DAOV. For the BT2120 data, we can see that the results by OpenCV were affected by outliers, leading to large RMS reprojection errors and parameters slightly deviated from other methods.

For the Mei model, we processed the BM4018, BT2120, and MTV185 sequences using tools including Kalibr, Camodocal, and the ROS calibrator. For comparison to the EUCM model, these sequences were also processed by BabelCalib and Basalt. The calibration parameters and the RMS reprojection errors are shown in Figure 14. With real data, these tools obtained similar values and dispersions for focal lengths and principal points, more consistent than the simulation shown in Figure 11, where the data were simulated with the Mei model. The distortion parameters of the Mei model showed large variance despite reasonable RMS reprojection errors, due to its parameter redundancy. Otherwise, the EUCM model resulted in consistent values for $[\alpha ,\beta ]$.

6. Conclusions and Research Trends

For the prevalent wide-angle cameras, we survey the recent GCC tools from the perspectives of camera models, calibration targets, and algorithms, providing an overview of the benefits and limitations of these tools. We also evaluated six well-known calibration tools, including BabelCalib, Basalt, Camodocal, Kalibr, the MATLAB calibrator, and the OpenCV-based ROS calibrator, to study their consistency and repeatability on simulated and real data.

From the review and experiments, we summarize several findings.

1. Outlier handling is crucial for optimization-based camera calibration tools. These outliers are usually detected on corners a few pixels away from their actual image locations, and often occur in somewhat blurry images. Luckily, most GCC tools can deal with outliers.

2. The GCC tools, Camodocal, Kalibr, and the MATLAB calibrator, support the pinhole radial–tangential model well. BabelCalib and Camodocal support the KB-8 model well, and TartanCalib supports the KB-8 model for a camera with a DAOV $<{180}^{\circ }$ well. Camodocal, TartanCalib, and OpenCV support the Mei model well, but the model suffers from parameter instability and redundancy.

   Moreover, the pinhole radial–tangential model is suitable for cameras of a DAOV $<{100}^{\circ }$, but becomes inadequate for those of a DAOV over ${100}^{\circ }$. The KB-8 model is typically preferred for cameras with a large DAOV due to its wide support and good accuracy when a global camera model is to be obtained.

3. The various failure cases revealed in our tests imply the intricacy in camera model initialization and optimization of a classic GCC tool. Aside from these failures, these GCC tools in item 2 agree well with each other when calibrating conventional, fisheye, and omnidirectional cameras with proper global camera models.

Based on this study, we point out several future research directions.

Interactive Calibration It is well known that quality data and informative data are essential for GCC. They address two potential issues, image blur that may be caused by rapid motion or out of focus, and insufficient data. One way to ensure data quality and information is interactive calibration, which provides a quality check, selects the quality data, and gives next-move suggestions in real time, whether for target-based or target-free calibration. AprilCal [128] and Calibration Wizard [129] are such interactive tools for target-based calibration.

Static Calibration Target-based calibration often involves unrepeatable onerous movements which can be obviated in at least two ways, namely calibration with a programmed robot arm and static calibration. Robot arm-based calibration has been studied in [130]. Static calibration usually relies on active targets. Such methods have been developed in [85,131] with application-specific setups. We think there is still much room to explore in static calibration.

Reconstruction with Calibration The setup of the lab calibration is usually different from the in-situ setup, e.g., in focusing distance (depth of field), exposure, capture mode (snapshot or video), aperture, and the size of the objects of interest. Some work has been carried out to mitigate the differences, e.g., out of focus, in [86,87]. An ultimate solution would be self-calibration or calibration based on prior maps. These methods depend on a reconstruction engine that supports calibration. Such an engine based on bundle adjustment is colmap [110]. New engines capable of calibration based on deep learning are on the rise, for instance, [94,132].

Author Contributions

Conceptualization, J.H. and G.J.; validation, J.H., Y.S. and B.W.; data curation, J.H. and Y.S.; writing—original draft preparation, J.H.; writing—review and editing, G.J., Y.H. and A.Y.; visualization, J.H.; funding acquisition, J.H. and D.C. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The code and raw data supporting the conclusions of this article will be made available by the authors on request.

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:

AOV	Angle of view
DAOV	Diagonal angle of view
DLT	Direct linear transform
DS	Double sphere
EUCM	Extended unified camera model
FOV	Field of view
GCC	Geometrical camera calibration
KB	Kannala–Brandt
LIESMARS	Laboratory of Information Engineering in Surveying, Mapping And Remote Sensing
RMS	Root mean square
RPC	Rational polynomial coefficient
SfM	Structure from motion
UCM	Unified camera model

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. The structure of this survey on geometric wide-angle camera calibration.

Figure 2. The typical workflow for geometric camera calibration.

Figure 3. The parametric and generic camera models reviewed in Section 3.1. For completeness, deep neural network models and the generalized camera models for remote sensing are also shown.

Figure 4. Categories of targets for geometric camera calibration.

Figure 5. The passive planar calibration targets: (a) 8 × 11 checkerboard, (b) 8 × 11 circle grid, (c) 8 × 11 asymmetric circle grid, (d) 8 × 11 Charuco, (e) 7 × 10 AprilGrid, (f) 10 × 11 Deltille.

Figure 6. Graphic user interfaces of calib.io (a), the MATLAB camera calibrator (b), the ROS camera calibrator (c), and the AgiSoft MetaShape (d) for geometric wide-angle camera calibration.

Figure 7. Sample images for S04525, E1M3518, BM4218, BM4018, BT2120, MTV185, in row-major order.

Figure 9. Error distributions of pinhole radial–tangential model parameters and the root mean square (RMS) reprojection errors, by five geometric camera calibration tools, BabelCalib, Camodocal, Kalibr/TartanCalib, the MATLAB calibrator, and the ROS/OpenCV calibrator, on the simulated data for S04525, E1M3518, and BM4218 lenses, each with nine sequences. Note that BabelCalib used a pinhole radial model, resulting in zero values in [Forumla omitted. See PDF.] and [Forumla omitted. See PDF.], which are shown relative to the true values.

Figure 10. Error distributions of KB-8 parameters and the root mean square (RMS) reprojection errors, using five geometric camera calibration tools, BabelCalib, Basalt, Camodocal, Kalibr, and the OpenCV-based ROS calibrator, on the simulated data for BM4218, BM4018, BT2120, and MTV185 lenses, each with nine sequences. The results of the ROS calibrator for the BM4218, BT4018, and MTV185 lenses, and the MATLAB calibrator for the MTV185 lens, were excluded due to persistent failures.

Figure 11. Error distributions of Mei parameters and the root mean square (RMS) reprojection errors, by geometric camera calibration tools, BabelCalib, Basalt, Camodocal, Kalibr, and the OpenCV-based ROS camera calibrator, on the simulated data for BM4018, BT2120, and MTV185 lenses, each with nine sequences. BabalCalib and Basalt used the extended unified camera model (EUCM), while the others used the Mei model. Note that the [Forumla omitted. See PDF.] of the EUCM is shown together with [Forumla omitted. See PDF.] of the Mei model relative to [Forumla omitted. See PDF.]’s true value. A parameter unavailable to the EUCM, e.g., [Forumla omitted. See PDF.], is zero by default, and shown relative to the parameter’s true value. Basalt failed all MTV185 sequences.

Figure 12. Pinhole radial–tangential model parameters and the root mean square (RMS) reprojection errors, by geometric camera calibration tools, BabelCalib, Camodocal, Kalibr, the MATLAB calibrator, and the ROS/OpenCV camera calibrator, on the real datasets captured with S04525, E1M3518, and BM4218 lenses, each with nine sequences. Note that BabelCalib adopted a pinhole radial model. The ROS calibrator failed four out of nine times for the BM4218 dataset.

Figure 13. KB-8 parameters and the root mean square (RMS) reprojection errors by geometric camera calibration tools, BabelCalib, Basalt, Camodocal, Kalibr, and the ROS/OpenCV calibrator, on the real datasets captured with BM4218, BM4018, BT2120, and MTV185 lenses, each with nine sequences. Basalt failed 7 out of 9 times for both the BM4218 and MTV185 datasets, leading to its small variances.

Figure 14. Mei parameters and the root mean square (RMS) reprojection errors by geometric camera calibration tools, BabelCalib with the extended unified camera model (EUCM), Basalt with the EUCM, Camodocal, Kalibr/TartanCalib, and the ROS/OpenCV camera calibrator, on the real data captured with BM4018, BT2120, and MTV185 lenses, each with nine sequences. Note that the EUCM model parameter [Forumla omitted. See PDF.] is shown together with the Mei parameter [Forumla omitted. See PDF.]. The unavailable parameters, e.g., [Forumla omitted. See PDF.] to the EUCM model, are zero by default. Basalt failed 8 out of the 9 MTV185 sequences.

Figure 14. Mei parameters and the root mean square (RMS) reprojection errors by geometric camera calibration tools, BabelCalib with the extended unified camera model (EUCM), Basalt with the EUCM, Camodocal, Kalibr/TartanCalib, and the ROS/OpenCV camera calibrator, on the real data captured with BM4018, BT2120, and MTV185 lenses, each with nine sequences. Note that the EUCM model parameter [Forumla omitted. See PDF.] is shown together with the Mei parameter [Forumla omitted. See PDF.]. The unavailable parameters, e.g., [Forumla omitted. See PDF.] to the EUCM model, are zero by default. Basalt failed 8 out of the 9 MTV185 sequences.

Table 1

An (non-exhaustive) overview of geometric camera calibration (GCC) tools. Supported camera models are mostly explained in Section 3.1. rad: radial, tan: tangential, GUI: graphical user interface.

GCC Tools	Supported Models	Supported Targets	Outlier Handling	Param. Std. Dev.	Language	Open Source	Other Features
AprilCal [37]	KB-8; pinhole rad tan	AprilTag grid	No	No	Java	Yes	GUI; interactive
BabelCalib [113]	division; division even; DS; EUCM; FOV; KB-8; pinhole rad; UCM	agnostic	Huber	No	MATLAB	Yes	allow multiple planar targets
Basalt [26]	DS; EUCM; FOV; KB-8; UCM	AprilGrid	Huber	No	C++	Yes	efficient; GUI; modular
BoofCV [88]	KB-23; pinhole rad tan; Mei	checkerboard circle grid Ecocheck	No	No	Java	Yes	Android portable; GUI
calib.io [114]	B-spline model; division; DS; EUCM; FOV; KB-8; pinhole rational rad tan thin prism	Charuco; checkerboard	Huber	Yes	?	No	correlation analysis; GUI; multi-camera; multiple targets
Calibu by ARPG [115]	FOV; KB-8; pinhole rational rad	circle grid	No	No	C++	Yes
camcalib [116]	pinhole rad tan; KB-8; DS	AprilGrid; Charuco	likely	No	?	No	GUI; multi-camera
Camodocal [79]	pinhole rad tan; Mei; KB-8	checkerboard	Cauchy	No	C++	Yes	Improved checkerboard detector; modular; stereo camera
Colmap [110]	pinhole rad tan thin prism, KB-8; FOV	N/A	soft L1, Cauchy	No	C++	Yes	GUI; self-calibration; well-documented
Generic camera calibration [14]	central generic; non-central generic; pinhole rational rad tan thin prism;	custom grid with an AprilTag	Huber	No	C++	Yes	accurate; board deformation
Learning CCS [117]	pinhole rad	checkerboard	trim	No	Python	Yes
libomnical [118]	a centered model; a geometric model; Mei; Scaramuzza	checkerboard	No	No	MATLAB	Yes
Kalibr [19]/ TartanCalib [119]	DS; EUCM; FOV; pinhole equidistant; pinhole rad tan; Mei	AprilGrid; checkerboard	trim	Yes	C++	Yes	multi-camera; board deformation
MATLAB camera calibrator [10]	pinhole rad tan Scaramuzza	AprilTag grid; checkerboard; circle grid	likely	Yes	MATLAB	Yes	GUI; modular; stereo camera; well-documented
MC-Calib [120]	KB-8; pinhole rad tan	Charuco	Huber	No	C++	Yes	board deformation; multi-camera
Metashape calibrator [18]	a custom fisheye model; pinhole rad tan	checkerboard	?	Yes	C++ / Python	No	correlation analysis; GUI
Metashape [18]	a custom fisheye model; pinhole rad tan	N/A	Yes	No	C++	No	GUI; self-calibration
mrcal [121]	pinhole rational rad tan thin prism; splined stereographic	checkerboard	trim	No	C++ / Pyrhotn	Yes	board deformation; uncertainty analysis
MRPT camera calib [122] / OpenCV	pinhole rad tan	checkerboard	No	No	C++	Yes	GUI; multi- checkerboard detector
Omni calibrator [77]	Mei	checkerboard	No	Yes	MATLAB	Yes	GUI; manual checkerboard detection
ROS camera calibrator [123] / OpenCV	KB-8; pinhole rational rad tan thin prism; Mei	Charuco checkerboard circle grid	No, but trim (Mei)	No	Python	Yes	efficient; GUI; modular; stereo camera; well-documented
PhotoModeler calibrator [70]	backward pinhole rad tan	customized circle grid	likely	Yes	?	No	correlation analysis; GUI
Pix4DMapper [82]	pinhole rad tan; adapted Scaramuzza	N/A	Yes	No	C++	No	GUI; self-calibration
SCNeRF [94]	KB-6	N/A	trim	No	Python	Yes	self-calibration
vidar [124]	DS; EUCM; UCM	N/A	N/A	No	Python	Yes	self-calibration

Table 2

Lenses used on the industrial camera UI3251LE-M-GL, which has a CMOS sensor of 1600 × 1200 4.5 μm square pixels. The camera’s DAOV is computed using the captured images.

Lens	Max. Sensor	f-num	DAOV (°)	Focal Length
				(mm)	(px)
FocVis S04525	1/1.8”	2.5	90	4.5	1000
Matrix Vision E1M3518	1/2.5”	1.8	94	3.5	778
Lensagon BM4218	1/3”	1.8	103	4.2	933
Lensagon BM4018S118	1/1.8”	1.8	127	4.0	889
Lensagon BT2120	1/3”	2.0	164	2.1	467
ZLKC MTV185IR12MP	1/1.8”	2.0	194	1.85	411

Table 3

Tests carried out on real and simulated data. ‘r’ and ‘s’ values denote the number of failures out of nine sequences in real data tests and simulation, respectively. rad: radial, tan: tangential, equi: equidistant.

Lens	BabelCalib	Basalt	Camodocal	Kalibr/ TartanCalib	MATLAB	ROS Calibrator
S04525	pinhole rad, s: 1	N/A	pinhole rad tan	pinhole rad tan	pinhole rad tan	pinhole rad tan
E1M3518	pinhole rad	N/A	pinhole rad tan	pinhole rad tan	pinhole rad tan	pinhole rad tan
BM4218	pinhole rad	N/A	pinhole rad tan	pinhole rad tan	pinhole rad tan	pinhole rad tan, r: 4, s: 3
BM4218	KB-8	KB-8, r: 7, s: 4	KB-8	pinhole equi	Scaramuzza	pinhole equi, r: 9, s: 9
BM4018	KB-8	KB-8, s: 7	KB-8	pinhole equi	Scaramuzza	pinhole equi, r: 9, s: 9
BT2120	KB-8	KB-8, s: 4	KB-8	pinhole equi	Scaramuzza	pinhole equi
MTV185	KB-8	KB-8, r: 7, s: 7	KB-8	pinhole equi, r: 9, s: 9	Scaramuzza	pinhole equi, r: 8, s: 9
BM4018	EUCM	EUCM, s: 2	Mei	Mei	N/A	Mei
BT2120	EUCM	EUCM, r: 2, s: 2	Mei	Mei	N/A	Mei
MTV185	EUCM	EUCM, r: 8, s: 9	Mei	Mei	N/A	Mei

References

1. Toth, C.; Jóźków, G. Remote sensing platforms and sensors: A survey. ISPRS J. Photogramm. Remote Sens.; 2016; 115, pp. 22-36. [DOI: https://dx.doi.org/10.1016/j.isprsjprs.2015.10.004]

2. Li, H.; Wang, S.; Bai, Z.; Wang, H.; Li, S.; Wen, S. Research on 3D reconstruction of binocular vision based on thermal infrared. Sensors; 2023; 23, 7372. [DOI: https://dx.doi.org/10.3390/s23177372] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/37687828]

3. Kong, X.; Hucks, R.G. Preserving our heritage: A photogrammetry-based digital twin framework for monitoring deteriorations of historic structures. Autom. Constr.; 2023; 152, 104928. [DOI: https://dx.doi.org/10.1016/j.autcon.2023.104928]

4. Miao, Y.; Tao, X.; Xu, X.; Lu, J. Joint 3-D shape estimation and landmark localization from monocular cameras of intelligent vehicles. IEEE Internet Things J.; 2019; 6, pp. 15-25. [DOI: https://dx.doi.org/10.1109/JIOT.2018.2872435]

5. Finlayson, G.; Gong, H.; Fisher, R.B. Color homography: Theory and applications. IEEE Trans. Pattern Anal. Mach. Intell.; 2019; 41, pp. 20-33. [DOI: https://dx.doi.org/10.1109/TPAMI.2017.2760833]

6. Barnard, K.; Cardei, V.; Funt, B. A comparison of computational color constancy algorithms. I: Methodology and experiments with synthesized data. IEEE Trans. Image Process.; 2002; 11, pp. 972-984. [DOI: https://dx.doi.org/10.1109/TIP.2002.802531]

7. Tao, C.V.; Hu, Y. A comprehensive study of the rational function model for photogrammetric processing. Photogramm. Eng. Remote Sens.; 2001; 67, pp. 1347-1357.

8. Kraus, K. Photogrammetry: Geometry from Images and Laser Scans; Walter De Gruyter: Berlin, Germany, 2007; Volume 1.

9. Kannala, J.; Brandt, S. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. IEEE Trans. Pattern Anal. Mach. Intell.; 2006; 28, pp. 1335-1340. [DOI: https://dx.doi.org/10.1109/TPAMI.2006.153]

10. Scaramuzza, D.; Martinelli, A.; Siegwart, R. A toolbox for easily calibrating omnidirectional cameras. Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); Beijing, China, 9–15 October 2006; pp. 5695-5701. [DOI: https://dx.doi.org/10.1109/IROS.2006.282372]

11. Sturm, P.; Ramalingam, S.; Tardif, J.P.; Gasparini, S.; Barreto, J. Camera models and fundamental concepts used in geometric computer vision. Found. Trends® Comput. Graph. Vis.; 2011; 6, pp. 1-183. [DOI: https://dx.doi.org/10.1561/0600000023]

12. Fan, J.; Zhang, J.; Maybank, S.J.; Tao, D. Wide-angle image rectification: A survey. Int. J. Comput. Vis.; 2022; 130, pp. 747-776. [DOI: https://dx.doi.org/10.1007/s11263-021-01562-9]

13. Edmundson, K.L.; Alexandrov, O.; Archinal, B.A.; Becker, K.J.; Becker, T.L.; Mapel, J.A.; Moratto, Z.M.; Nefian, A.V.; Richie, J.O.; Robinson, M.S. et al. Revisiting the Apollo photogrammetric mapping system. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.; 2018; XLII-1, pp. 133-140. [DOI: https://dx.doi.org/10.5194/isprs-archives-XLII-1-133-2018]

14. Schöps, T.; Larsson, V.; Pollefeys, M.; Sattler, T. Why having 10,000 parameters in your camera model is better than twelve. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Virtual, 13–19 June 2020; pp. 2535-2544. [DOI: https://dx.doi.org/10.48550/arXiv.1912.02908]

15. Ramalingam, S.; Sturm, P. A unifying model for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell.; 2017; 39, pp. 1309-1319. [DOI: https://dx.doi.org/10.1109/TPAMI.2016.2592904] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27448339]

16. Liao, K.; Nie, L.; Huang, S.; Lin, C.; Zhang, J.; Zhao, Y.; Gabbouj, M.; Tao, D. Deep learning for camera calibration and beyond: A survey. Technical Report. arXiv; 2023; arXiv: 2303.10559

17. Mathworks Inc. MATLAB: Computer Vision Toolbox (R2021a); Mathworks Inc.: Natick, MA, USA, 2021.

18. Agisoft LLC. Agisoft Metashape; Agisoft LLC.: St. Petersburg, Russia, 2022.

19. Maye, J.; Sommer, H.; Agamennoni, G.; Siegwart, R.; Furgale, P. Online self-calibration for robotic systems. Int. J. Robot. Res.; 2016; 35, pp. 357-380. [DOI: https://dx.doi.org/10.1177/0278364915596232]

20. Bradski, G.R.; Kaehler, A. Learning OpenCV: Computer Vision with the OpenCV Library; 1st ed. O’Reilly Media, Inc.: Sebastopol, CA, USA, 2008.

21. Hieronymus, J. Comparison of methods for geometric camera calibration. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.; 2012; XXXIX-B5, pp. 595-599. [DOI: https://dx.doi.org/10.5194/isprsarchives-XXXIX-B5-595-2012]

22. Schmalz, C.; Forster, F.; Angelopoulou, E. Camera calibration: Active versus passive targets. Opt. Eng.; 2011; 50, 113601. [DOI: https://dx.doi.org/10.1117/1.3643726]

23. Hughes, C.; Glavin, M.; Jones, E.; Denny, P. Review of geometric distortion compensation in fish-eye cameras. Proceedings of the IET Irish Signals and Systems Conference (ISSC 2008); Galway, Ireland, 18–19 June 2008; pp. 162-167. [DOI: https://dx.doi.org/10.1049/cp:20080656]

24. Puig, L.; Bermúdez, J.; Sturm, P.; Guerrero, J.J. Calibration of omnidirectional cameras in practice: A comparison of methods. Comput. Vis. Image Underst.; 2012; 116, pp. 120-137. [DOI: https://dx.doi.org/10.1016/j.cviu.2011.08.003]

25. Zhang, Y.; Zhao, L.; Hu, W. A survey of catadioptric omnidirectional camera calibration. Int. J. Inf. Technol. Comput. Sci.; 2013; 5, pp. 13-20. [DOI: https://dx.doi.org/10.5815/ijitcs.2013.03.02]

26. Usenko, V.; Demmel, N.; Cremers, D. The double sphere camera model. Proceedings of the 2018 International Conference on 3D Vision (3DV); Verona, Italy, 5–8 September 2018; pp. 552-560.

27. Khomutenko, B.; Garcia, G.; Martinet, P. An enhanced unified camera model. IEEE Robot. Autom. Lett.; 2016; 1, pp. 137-144. [DOI: https://dx.doi.org/10.1109/LRA.2015.2502921]

28. Devernay, F.; Faugeras, O. Straight lines have to be straight. Mach. Vis. Appl.; 2001; 13, pp. 14-24. [DOI: https://dx.doi.org/10.1007/PL00013269]

29. Bergamasco, F.; Albarelli, A.; Rodolà, E.; Torsello, A. Can a fully unconstrained imaging model be applied effectively to central cameras?. Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Portland, OR, USA, 23–28 June 2013; pp. 1391-1398. [DOI: https://dx.doi.org/10.1109/CVPR.2013.183]

30. Beck, J.; Stiller, C. Generalized B-spline camera model. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV); Changshu, China, 26 June–1 July 2018; pp. 2137-2142. [DOI: https://dx.doi.org/10.1109/IVS.2018.8500466]

31. Ramalingam, S. Generic imaging models: Calibration and 3d reconstruction algorithms. PhD Thesis; Institut National Polytechnique de Grenoble-INPG: Grenoble, France, 2006.

32. Kenefick, J.F.; Gyer, M.S.; Harp, B.F. Analytical self-calibration. Photogramm. Eng.; 1972; 38, pp. 1117-1126.

33. Gruen, A.; Beyer, H.A. System calibration through self-calibration. Calibration and Orientation of Cameras in Computer Vision; Huang, T.S.; Kohonen, T.; Schroeder, M.R.; Gruen, A.; Huang, T.S. Springer: Berlin/Heidelberg, Germany, 2001; Volume 34, pp. 163-193. [DOI: https://dx.doi.org/10.1007/978-3-662-04567-1_7]

34. Stamatopoulos, C.; Fraser, C.S. Automated target-free network orienation and camera calibration. ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.; 2014; II5, pp. 339-346. [DOI: https://dx.doi.org/10.5194/isprsannals-II-5-339-2014]

35. Mallon, J.; Whelan, P.F. Which pattern? Biasing aspects of planar calibration patterns and detection methods. Pattern Recognit. Lett.; 2007; 28, pp. 921-930. [DOI: https://dx.doi.org/10.1016/j.patrec.2006.12.008]

36. Kato, H.; Billinghurst, M. Marker tracking and HMD calibration for a video-based augmented reality conferencing system. Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR’99); San Francisco, CA, USA, 20–21 October 1999; pp. 85-94. [DOI: https://dx.doi.org/10.1109/IWAR.1999.803809]

37. Olson, E. AprilTag: A robust and flexible visual fiducial system. Proceedings of the 2011 IEEE International Conference on Robotics and Automation; Shanghai, China, 9–13 May 2011; pp. 3400-3407. [DOI: https://dx.doi.org/10.1109/ICRA.2011.5979561]

38. Garrido-Jurado, S.; Muñoz-Salinas, R.; Madrid-Cuevas, F.J.; Marín-Jiménez, M.J. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognit.; 2014; 47, pp. 2280-2292. [DOI: https://dx.doi.org/10.1016/j.patcog.2014.01.005]

39. dos Santos Cesar, D.B.; Gaudig, C.; Fritsche, M.; dos Reis, M.A.; Kirchner, F. An evaluation of artificial fiducial markers in underwater environments. Proceedings of the OCEANS 2015—Genova; Genova, Italy, 18–21 May 2015; pp. 1-6. [DOI: https://dx.doi.org/10.1109/OCEANS-Genova.2015.7271491]

40. Fiala, M. ARTag, a fiducial marker system using digital techniques. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05); San Diego, CA, USA, 20–26 June 2005; Volume 2, pp. 590-596. [DOI: https://dx.doi.org/10.1109/CVPR.2005.74]

41. Atcheson, B.; Heide, F.; Heidrich, W. CALTag: High precision fiducial markers for camera calibration. Vision, Modeling, and Visualization; The Eurographics Association: Eindhoven, The Netherlands, 2010; Volume 10, pp. 41-48.

42. Sagitov, A.; Shabalina, K.; Sabirova, L.; Li, H.; Magid, E. ARTag, AprilTag and CALTag fiducial marker systems: Comparison in a presence of partial marker occlusion and rotation. Proceedings of the 14th International Conference on Informatics in Control, Automation and Robotics; Madrid, Spain, 26–28 July 2017; pp. 182-191. [DOI: https://dx.doi.org/10.5220/0006478901820191]

43. Kunz, C.; Genten, V.; Meißner, P.; Hein, B. Metric-based evaluation of fiducial markers for medical procedures. Proceedings of the Medical Imaging 2019: Image-Guided Procedures, Robotic Interventions, and Modeling; San Diego, CA, USA, 17–19 February 2019; Volume 10951, pp. 690-703.

44. Benligiray, B.; Topal, C.; Akinlar, C. STag: A stable fiducial marker system. Image Vis. Comput.; 2019; 89, pp. 158-169. [DOI: https://dx.doi.org/10.1016/j.imavis.2019.06.007]

45. Kalaitzakis, M.; Carroll, S.; Ambrosi, A.; Whitehead, C.; Vitzilaios, N. Experimental comparison of fiducial markers for pose estimation. Proceedings of the 2020 International Conference on Unmanned Aircraft Systems (ICUAS); Athens, Greece, 1–4 September 2020; pp. 781-789. [DOI: https://dx.doi.org/10.1109/ICUAS48674.2020.9213977]

46. Lightbody, P.; Krajník, T.; Hanheide, M. A versatile high-performance visual fiducial marker detection system with scalable identity encoding. Proceedings of the Symposium on Applied Computing; Marrakech, Morocco, 3–7 April 2017; pp. 276-282.

47. Springer, J.; Kyas, M. Evaluation of orientation ambiguity and detection rate in April Tag and WhyCode. Proceedings of the 2022 Sixth IEEE International Conference on Robotic Computing (IRC); Naples, Italy, 5–7 December 2022; pp. 281-286. [DOI: https://dx.doi.org/10.48550/arXiv.2203.10180]

48. García-Ruiz, P.; Romero-Ramirez, F.J.; Muñoz-Salinas, R.; Marín-Jiménez, M.J.; Medina-Carnicer, R. Fiducial Objects: Custom Design and Evaluation. Sensors; 2023; 23, 9649. [DOI: https://dx.doi.org/10.3390/s23249649]

49. Clarke, T.A.; Fryer, J.G. The development of camera calibration methods and models. Photogramm. Rec.; 1998; 16, pp. 51-66. [DOI: https://dx.doi.org/10.1111/0031-868X.00113]

50. Salvi, J.; Armangué, X.; Batlle, J. A comparative review of camera calibrating methods with accuracy evaluation. Pattern Recognit.; 2002; 35, pp. 1617-1635. [DOI: https://dx.doi.org/10.1016/S0031-3203(01)00126-1]

51. Remondino, F.; Fraser, C. Digital camera calibration methods: Considerations and comparisons. Proceedings of the International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences; Dresden, Germany, 25–27 September 2006; Volume 36, pp. 266-272. [DOI: https://dx.doi.org/10.3929/ETHZ-B-000158067]

52. Lichti, D.D.; Kim, C. A comparison of three geometric self-calibration methods for range cameras. Remote Sens.; 2011; 3, pp. 1014-1028. [DOI: https://dx.doi.org/10.3390/rs3051014]

53. Joshi, B.; Ohmi, K.; Nose, K. Comparative study of camera calibration methods for 3D particle tracking velocimetry. Int. J. Innov. Comput. Inf. Control.; 2013; 9, pp. 1971-1986.

54. Abdel-Aziz, Y.; Karara, H. Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry. Photogramm. Eng. Remote Sens.; 2015; 81, pp. 103-107. [DOI: https://dx.doi.org/10.14358/PERS.81.2.103]

55. Fraser, C.S.; Dial, G.; Grodecki, J. Sensor orientation via RPCs. ISPRS J. Photogramm. Remote Sens.; 2006; 60, pp. 182-194. [DOI: https://dx.doi.org/10.1016/j.isprsjprs.2005.11.001]

56. Tsai, R.Y. A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J. Robot. Autom.; 1987; 3, pp. 323-344. [DOI: https://dx.doi.org/10.1109/JRA.1987.1087109]

57. Usamentiaga, R.; Ibarra-Castanedo, C.; Maldague, X. Comparison and evaluation of geometric calibration methods for infrared cameras to perform metric measurements on a plane. Appl. Opt.; 2018; 57, pp. D1-D10. [DOI: https://dx.doi.org/10.1364/AO.57.0000D1] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30117928]

58. Kanhere, N.K.; Birchfield, S.T. A taxonomy and analysis of camera calibration methods for traffic monitoring applications. IEEE Trans. Intell. Transp. Syst.; 2010; 11, pp. 441-452. [DOI: https://dx.doi.org/10.1109/TITS.2010.2045500]

59. Hartley, R.; Zisserman, A. Multiple View Geometry in Computer Vision; 2nd ed. Cambridge University Press: Cambridge, UK, 2003.

60. Zhang, Z. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell.; 2000; 22, pp. 1330-1334. [DOI: https://dx.doi.org/10.1109/34.888718]

61. Albiol, F.; Corbi, A.; Albiol, A. Evaluation of modern camera calibration techniques for conventional diagnostic X-ray imaging settings. Radiol. Phys. Technol.; 2017; 10, pp. 68-81. [DOI: https://dx.doi.org/10.1007/s12194-016-0369-y]

62. Tiscareño, J.; Albajez, J.A.; Santolaria, J. Analysis of different camera calibration methods on a camera-projector measuring system. Procedia Manuf.; 2019; 41, pp. 539-546. [DOI: https://dx.doi.org/10.1016/j.promfg.2019.09.041]

63. Ayaz, S.M.; Kim, M.Y.; Park, J. Survey on zoom-lens calibration methods and techniques. Mach. Vis. Appl.; 2017; 28, pp. 803-818. [DOI: https://dx.doi.org/10.1007/s00138-017-0863-1]

64. Yin, W.; Zang, X.; Wu, L.; Zhang, X.; Zhao, J. A distortion correction method based on actual camera imaging principles. Sensors; 2024; 24, 2406. [DOI: https://dx.doi.org/10.3390/s24082406]

65. Ricolfe-Viala, C.; Sánchez-Salmerón, A.J. Robust metric calibration of non-linear camera lens distortion. Pattern Recognit.; 2010; 43, pp. 1688-1699. [DOI: https://dx.doi.org/10.1016/j.patcog.2009.10.003]

66. Wang, M.; Yang, B.; Hu, F.; Zang, X. On-orbit geometric calibration model and its applications for high-resolution optical satellite imagery. Remote Sens.; 2014; 6, pp. 4391-4408. [DOI: https://dx.doi.org/10.3390/rs6054391]

67. Pi, Y.; Wang, M.; Yang, B.; Gao, Z. Robust camera distortion calibration via unified RPC model for optical remote sensing satellites. IEEE Trans. Geosci. Remote Sens.; 2022; 60, pp. 1-15. [DOI: https://dx.doi.org/10.1109/TGRS.2022.3198076]

68. Brown, D.C. Close-range camera calibration. Photogramm. Eng.; 1971; 37, pp. 855-866.

69. Drap, P.; Lefèvre, J. An exact formula for calculating inverse radial lens distortions. Sensors; 2016; 16, 807. [DOI: https://dx.doi.org/10.3390/s16060807]

70. PhotoModeler Technologies. PhotoModeler; PhotoModeler Technologies: Vancouver, BC, Canada, 2022.

71. Fitzgibbon, A.W. Simultaneous linear estimation of multiple view geometry and lens distortion. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001); Kauai, HI, USA, 8–14 December 2001; [DOI: https://dx.doi.org/10.1109/CVPR.2001.990465]

72. Li, H.; Hartley, R. A non-iterative method for correcting lens distortion from nine point correspondences. Proceedings of the OMNIVIS 2005; Beijing, China, 21 October 2005.

73. Weng, J.; Cohen, P.; Herniou, M. Camera calibration with distortion models and accuracy evaluation. IEEE Trans. Pattern Anal. Mach. Intell.; 1992; 14, pp. 965-980. [DOI: https://dx.doi.org/10.1109/34.159901]

74. Abraham, S.; Förstner, W. Fish-eye-stereo calibration and epipolar rectification. ISPRS J. Photogramm. Remote Sens.; 2005; 59, pp. 278-288. [DOI: https://dx.doi.org/10.1016/j.isprsjprs.2005.03.001]

75. Schneider, D.; Schwalbe, E.; Maas, H.G. Validation of geometric models for fisheye lenses. ISPRS J. Photogramm. Remote Sens.; 2009; 64, pp. 259-266. [DOI: https://dx.doi.org/10.1016/j.isprsjprs.2009.01.001]

76. Courbon, J.; Mezouar, Y.; Eckt, L.; Martinet, P. A generic fisheye camera model for robotic applications. Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems; San Diego, CA, USA, 29 October–2 November 2007; pp. 1683-1688. [DOI: https://dx.doi.org/10.1109/IROS.2007.4399233]

77. Mei, C.; Rives, P. Single view point omnidirectional camera calibration from planar grids. Proceedings of the 2007 IEEE International Conference on Robotics and Automation; Rome, Italy, 10–14 April 2007; pp. 3945-3950. [DOI: https://dx.doi.org/10.1109/ROBOT.2007.364084]

78. Li, B.; Heng, L.; Koser, K.; Pollefeys, M. A multiple-camera system calibration toolbox using a feature descriptor-based calibration pattern. Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems; Tokyo, Japan, 3–7 November 2013; pp. 1301-1307. [DOI: https://dx.doi.org/10.1109/IROS.2013.6696517]

79. Heng, L.; Li, B.; Pollefeys, M. CamOdoCal: Automatic intrinsic and extrinsic calibration of a rig with multiple generic cameras and odometry. Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems; Tokyo, Japan, 3–7 November 2013; pp. 1793-1800. [DOI: https://dx.doi.org/10.1109/IROS.2013.6696592]

80. Rosebrock, D.; Wahl, F.M. Generic camera calibration and modeling using spline surfaces. Proceedings of the 2012 IEEE Intelligent Vehicles Symposium; Alcala de Henares, Spain, 3–7 June 2012; pp. 51-56.

81. Keivan, N.; Sibley, G. Constant-time monocular self-calibration. Proceedings of the 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014); Bali, Indonesia, 5–10 December 2014; pp. 1590-1595. [DOI: https://dx.doi.org/10.1109/ROBIO.2014.7090561]

82. Pix4D S.A. PIX4Dmapper; Pix4D S.A.: Prilly, Switzerland, 2022.

83. An, G.H.; Lee, S.; Seo, M.W.; Yun, K.; Cheong, W.S.; Kang, S.J. Charuco board-based omnidirectional camera calibration method. Electronics; 2018; 7, 421. [DOI: https://dx.doi.org/10.3390/electronics7120421]

84. Ha, H.; Perdoch, M.; Alismail, H.; Kweon, I.S.; Sheikh, Y. Deltille grids for geometric camera calibration. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV); Venice, Italy, 22–29 October 2017; pp. 5354-5362. [DOI: https://dx.doi.org/10.1109/ICCV.2017.571]

85. Gao, W.; Lin, J.; Zhang, F.; Shen, S. A screen-based method for automated camera intrinsic calibration on production lines. Proceedings of the 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE); Vancouver, BC, Canada, 22–26 August 2019; pp. 392-398. [DOI: https://dx.doi.org/10.1109/COASE.2019.8842956]

86. Ha, H.; Bok, Y.; Joo, K.; Jung, J.; Kweon, I.S. Accurate camera calibration robust to defocus using a smartphone. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV); Santiago, Chile, 7–13 December 2015; pp. 828-836. [DOI: https://dx.doi.org/10.1109/ICCV.2015.101]

87. Bell, T.; Xu, J.; Zhang, S. Method for out-of-focus camera calibration. Appl. Opt.; 2016; 55, pp. 2346-2352. [DOI: https://dx.doi.org/10.1364/AO.55.002346]

88. Abeles, P. BoofCV, 2011–2021. Available online: https://boofcv.org/index.php (accessed on 24 September 2024).

89. Rufli, M.; Scaramuzza, D.; Siegwart, R. Automatic detection of checkerboards on blurred and distorted images. Proceedings of the 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems; Nice, France, 22–26 September 2008; pp. 3121-3126. [DOI: https://dx.doi.org/10.1109/IROS.2008.4650703]

90. Geiger, A.; Moosmann, F.; Car, Ö.; Schuster, B. Automatic camera and range sensor calibration using a single shot. Proceedings of the 2012 IEEE International Conference on Robotics and Automation (ICRA); St. Paul, MN, USA, 14–18 May 2012; pp. 3936-3943. [DOI: https://dx.doi.org/10.1109/ICRA.2012.6224570]

91. Fuersattel, P.; Dotenco, S.; Placht, S.; Balda, M.; Maier, A.; Riess, C. OCPAD—Occluded checkerboard pattern detector. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV); Lake Placid, NY, USA, 7–10 March 2016; pp. 1-9. [DOI: https://dx.doi.org/10.1109/WACV.2016.7477565]

92. Yan, Y.; Yang, P.; Yan, L.; Wan, J.; Sun, Y.; Tansey, K.; Asundi, A.K.; Zhao, H. Automatic checkerboard detection for camera calibration using self-correlation. J. Electron. Imaging; 2018; 27, 033014. [DOI: https://dx.doi.org/10.1117/1.JEI.27.3.033014]

93. Meng, X.; Hu, Z. A new easy camera calibration technique based on circular points. Pattern Recognit.; 2003; 36, pp. 1155-1164. [DOI: https://dx.doi.org/10.1016/S0031-3203(02)00225-X]

94. Jeong, Y.; Ahn, S.; Choy, C.; Anandkumar, A.; Cho, M.; Park, J. Self-calibrating neural radiance fields. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); Montreal, QC, Canada, 10–17 October 2021; pp. 5826-5834. [DOI: https://dx.doi.org/10.1109/ICCV48922.2021.00579]

95. Li, Y.; Zhao, Z.; Chen, Y.; Zhang, X.; Tian, R. Automatic Roadside Camera Calibration with Transformers. Sensors; 2023; 23, 9527. [DOI: https://dx.doi.org/10.3390/s23239527] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/38067900]

96. Li, S.; Yoon, H.S. Enhancing Camera Calibration for Traffic Surveillance with an Integrated Approach of Genetic Algorithm and Particle Swarm Optimization. Sensors; 2024; 24, 1456. [DOI: https://dx.doi.org/10.3390/s24051456] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/38474992]

97. Rehder, J.; Nikolic, J.; Schneider, T.; Siegwart, R. A direct formulation for camera calibration. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA); Singapore, 29 May–3 June 2017; pp. 6479-6486. [DOI: https://dx.doi.org/10.1109/ICRA.2017.7989765]

98. Sarlin, P.E.; Lindenberger, P.; Larsson, V.; Pollefeys, M. Pixel-perfect structure-from-motion with featuremetric refinement. IEEE Trans. Pattern Anal. Mach. Intell.; 2023; pp. 1-12. [DOI: https://dx.doi.org/10.1109/TPAMI.2023.3237269] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/37021895]

99. Larsson, V.; Sattler, T.; Kukelova, Z.; Pollefeys, M. Revisiting radial distortion absolute pose. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV); Seoul, Republic of Korea, 27 October–2 November 2019; pp. 1062-1071.

100. Unal, G.; Yezzi, A.; Soatto, S.; Slabaugh, G. A variational approach to problems in calibration of multiple cameras. IEEE Trans. Pattern Anal. Mach. Intell.; 2007; 29, pp. 1322-1338. [DOI: https://dx.doi.org/10.1109/TPAMI.2007.1035]

101. Lee, Y.J.; Yilmaz, A.; Mendoza-Schrock, O. In-flight camera platform geometric calibration of the aerial multi-head camera system. Proceedings of the IEEE 2010 National Aerospace & Electronics Conference; Dayton, OH, USA, 14–16 July 2010; pp. 136-139. [DOI: https://dx.doi.org/10.1109/NAECON.2010.5712937]

102. Brauer-Burchardt, C.; Voss, K. A new algorithm to correct fish-eye- and strong wide-angle-lens-distortion from single images. Proceedings of the 2001 International Conference on Image Processing (Cat. No.01CH37205); Thessaloniki, Greece, 7–10 October 2001; Volume 1, pp. 225-228. [DOI: https://dx.doi.org/10.1109/ICIP.2001.958994]

103. Ahmed, M.; Farag, A. Nonmetric calibration of camera lens distortion: Differential methods and robust estimation. IEEE Trans. Image Process.; 2005; 14, pp. 1215-1230. [DOI: https://dx.doi.org/10.1109/TIP.2005.846025]

104. Bukhari, F.; Dailey, M.N. Automatic radial distortion estimation from a single image. J. Math. Imaging Vis.; 2013; 45, pp. 31-45. [DOI: https://dx.doi.org/10.1007/s10851-012-0342-2]

105. Xue, Z.; Xue, N.; Xia, G.S.; Shen, W. Learning to calibrate straight lines for fisheye image rectification. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Long Beach, CA, USA, 15–20 June 2019; pp. 1643-1651. [DOI: https://dx.doi.org/10.1109/CVPR.2019.00174]

106. Lin, Y.; Larsson, V.; Geppert, M.; Kukelova, Z.; Pollefeys, M.; Sattler, T. Infrastructure-based multi-camera calibration using radial projections. Proceedings of the European Conference on Computer Vision (ECCV); Glasgow, UK, 23–28 August 2020; pp. 327-344.

107. Huai, J.; Lin, Y.; Zhuang, Y.; Toth, C.; Chen, D. Observability analysis and keyframe-based filtering for visual inertial odometry with full self-calibration. IEEE Trans. Robot.; 2022; 38, pp. 3219-3237. [DOI: https://dx.doi.org/10.1109/TRO.2022.3172474]

108. Faugeras, O.D.; Luong, Q.T.; Maybank, S.J. Camera self-calibration: Theory and experiments. Proceedings of the Computer Vision—ECCV’92; Sandini, G. Lecture Notes in Computer Science Springer: Berlin/Heidelberg, Germany, 1992; pp. 321-334.

109. Hemayed, E. A survey of camera self-calibration. Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance; Miami, FL, USA, 22 July 2003; pp. 351-357. [DOI: https://dx.doi.org/10.1109/AVSS.2003.1217942]

110. Schonberger, J.L.; Frahm, J.M. Structure-from-motion revisited. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Las Vegas, NV, USA, 27–30 June 2016; pp. 4104-4113.

111. Liu, A.; Marschner, S.; Snavely, N. Caliber: Camera localization and calibration using rigidity constraints. Int. J. Comput. Vis.; 2016; 118, pp. 1-21. [DOI: https://dx.doi.org/10.1007/s11263-015-0866-1]

112. Oliveira, M.; Castro, A.; Madeira, T.; Dias, P.; Santos, V. A general approach to the extrinsic calibration of intelligent vehicles using ROS. Proceedings of the Robot 2019: Fourth Iberian Robotics Conference; Porto, Portugal, 20–22 November 2019; Silva, M.F.; Luís Lima, J.; Reis, L.P.; Sanfeliu, A.; Tardioli, D. Advances in Intelligent Systems and Computing Springer: Berlin/Heidelberg, Germany, 2019; pp. 203-215. [DOI: https://dx.doi.org/10.1007/978-3-030-35990-4_17]

113. Lochman, Y.; Liepieshov, K.; Chen, J.; Perdoch, M.; Zach, C.; Pritts, J. BabelCalib: A universal approach to calibrating central cameras. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); Montreal, QC, Canada, 10–17 October 2021; pp. 15233-15242. [DOI: https://dx.doi.org/10.1109/ICCV48922.2021.01497]

114. calib.io. Camera calibrator, 2022. Available online: https://calib.io/products/calib (accessed on 24 September 2024).

115. University of Colorado. Calibu; University of Colorado: Boulder, CO, USA, 2022.

116. IVISO GmbH. camcalib; IVISO GmbH: Vienna, Austria, 2022.

117. Zhang, Y.; Zhao, X.; Qian, D. Learning-based distortion correction and feature detection for high precision and robust camera calibration. IEEE Robot. Autom. Lett.; 2022; 7, pp. 10470-10477. [DOI: https://dx.doi.org/10.1109/LRA.2022.3192610]

118. Schönbein, M.; Strauß, T.; Geiger, A. Calibrating and centering quasi-central catadioptric cameras. Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA); Hong Kong, China, 31 May–5 June 2014; pp. 4443-4450. [DOI: https://dx.doi.org/10.1109/ICRA.2014.6907507]

119. Duisterhof, B.P.; Hu, Y.; Teng, S.H.; Kaess, M.; Scherer, S. TartanCalib: Iterative Wide-Angle Lens Calibration Using Adaptive Subpixel Refinement of AprilTags. Available online: https://arxiv.org/abs/2210.02511 (accessed on 24 September 2024).

120. Rameau, F.; Park, J.; Bailo, O.; Kweon, I.S. MC-Calib: A generic and robust calibration toolbox for multi-camera systems. Comput. Vis. Image Underst.; 2022; 217, 103353. [DOI: https://dx.doi.org/10.1016/j.cviu.2021.103353]

121. Kogan, D. mrcal, 2022. Available online: http://mrcal.secretsauce.net (accessed on 24 September 2024).

122. University of Malaga. MRPT Camera Calib; Department of System Engineering and Automation, University of Malaga: Malaga, Spain, 2022.

123. ROS ROS Camera Calibration; 2022; Available online: https://wiki.ros.org/camera_calibration (accessed on 24 September 2024).

124. Fang, J.; Vasiljevic, I.; Guizilini, V.; Ambrus, R.; Shakhnarovich, G.; Gaidon, A.; Walter, M.R. Self-supervised camera self-calibration from video. Proceedings of the 2022 International Conference on Robotics and Automation (ICRA); Philadelphia, PA, USA, 23–27 May 2022; pp. 8468-8475. [DOI: https://dx.doi.org/10.1109/ICRA46639.2022.9811784]

125. Luhmann, T.; Robson, S.; Kyle, S.; Harley, I. Close Range Photogrammetry: Principles, Techniques and Applications; Whittles Publishing: Caithness, UK, 2006.

126. Huai, J.; Zhuang, Y.; Yuan, Q.; Lin, Y. Continuous-time spatiotemporal calibration of a rolling shutter camera-IMU system. IEEE Sens. J.; 2022; 22, pp. 7920-7930. [DOI: https://dx.doi.org/10.1109/JSEN.2022.3152572]

127. Bouguet, J. Camera Calibration Toolbox for MATLAB. 2001; Available online: http://robots.stanford.edu/cs223b04/JeanYvesCalib/ (accessed on 24 September 2024).

128. Richardson, A.; Strom, J.; Olson, E. AprilCal: Assisted and repeatable camera calibration. Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); Tokyo, Japan, 3–7 November 2013; pp. 1814-1821. [DOI: https://dx.doi.org/10.1109/IROS.2013.6696595]

129. Peng, S.; Sturm, P. Calibration wizard: A guidance system for camera calibration based on modelling geometric and corner uncertainty. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV); Seoul, Republic of Korea, 27 October–2 November 2019; pp. 1497-1505. [DOI: https://dx.doi.org/10.1109/ICCV.2019.00158]

130. Werner, T.; Harrer, D.; Henrich, D. Efficient, precise, and convenient calibration of multi-camera systems by robot automation. Proceedings of the Advances in Service and Industrial Robotics: Proceedings of the 28th International Conference on Robotics in Alpe-Adria-Danube Region (RAAD 2019); Aspragathos, N.A.; Koustoumpardis, P.N.; Moulianitis, V.C. Mechanisms and Machine Science; Springer: Kaiserslautern, Germany, 2019; pp. 669-677. [DOI: https://dx.doi.org/10.1007/978-3-030-00232-9_70]

131. Tehrani, M.A.; Beeler, T.; Grundhöfer, A. A practical method for fully automatic intrinsic camera calibration using directionally encoded light. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Honolulu, HI, USA, 21–26 July 2017; pp. 125-133. [DOI: https://dx.doi.org/10.1109/CVPR.2017.21]

132. Lindenberger, P.; Sarlin, P.E.; Larsson, V.; Pollefeys, M. Pixel-perfect structure-from-motion with featuremetric refinement. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); Montreal, QC, Canada, 11–17 October 2021; pp. 5967-5977. [DOI: https://dx.doi.org/10.1109/ICCV48922.2021.00593]