Chapter \ref{cha:features} listed basic features of the ICL packages that do not pertain to image processing in the proper meaning of the word. This chapter lists and explains most image processing functionalities and interfaces. \section{ICLUtils} [Documentation: \iclpackageref{Utils}]\\ This ICLUtils package contains (per definition) no image processing functions, rather it provides more general classes and functions (e.g. matrix and vector classes, exceptions, debugging macros etc.). \subsection{Matrix and Vector Classes} Matrix and Vector algebra is very common in a lot of applications. The ICL provides two approaches for matrix and vector classes: \subsubsection{FixedMatrix Template Class\iclclassref{FixedMatrix}{Utils}} This class uses three template parameters \icode{template} to represent matrices of fixed size. FixedVectors are implemented as shallow inheriting wrappers of FixedMatrix class instances with \icode{COLS=1} (\icode{FixedColVector}) or \icode{ROWS=1} (\icode{FixedRowVector})\footnote{This cannot be implemented using typedefs, because C++ does not support templated typedefs}. By using \emph{templated} dimension parameters, matrix dimension is fixed at compilation time, which enables the compiler to apply a lot of optimizations. Most common vector and matrix algebra functionalities \footnote{As far as I can see, actually a super-set of BLAS-level 3}, including matrix mutliplication (scalar product is just a special case here), QR-Matrix decomposition and matrix inverse and pseudo-inverse\footnote{QR-decomposition-based} implementation. \icode{FixedMatrix} instances resign smart data handling to enable the compiler to apply special optimizations on fixed arrays. In particular, temporary instances (and their data) can be allocated on the stack, rather than having to allocate data on the heap. The \icode{FixedMatrix} class provides row- and column iterators, as well as extraction functions for rows, columns or rectangular sub-matrices. Here's a concise example: \codefile{fixed-matrix-demo-1.cpp}{Example for usage of the \icode{FixedMatrix}-template class} Please note, that \icode{FixedMatrix}-pseudo inverse calculation is out-sourced to a dedicated header \iclheaderref{Utils}{FixedMatrixUtils} to reduce complexity of the FixedMatrix header file. \subsubsection{DynMatrix Template Class\iclclassref{DynMatrix}{Utils}} In contrast to the \icode{FixedMatrix} template class, the \icode{DynMatrix} template class uses a single template parameter which defines the actual datatype of the matrix elements. Matrix dimensions are dynamic at runtime, which enables us to provide e.g. a \icode{setBounds} member function. In addition, \icode{DynMatrix} instances can be assigned with \icode{DynMatrix} instances of another size. Dynamic sized data types need dynamic data handling\footnote{using \icode{new []} and \icode{delete []}}. This besides allows a more flexible mechanism: optional shallow data wrapping. \icode{DynMatrix} instances can be set up to handle their data autonomously (using \icode{new []} in the constructor and \icode{delete []} in the destructor), or to \emph{count on} external data handling. Most other functionalities such as matrix multiplication or matrix inverse are also available here. Actually, matrix inverse in the \icode{FixedMatrix} class is implemented by a shallow wrapping \icode{DynMatrix} function call.\\ In addition to the actual \icode{DynMatrix}-class, we provide another header file that is called \iclheaderref{Utils}{DynMatrixUtils}. This contains a large set of arithmetical functions in different calling manners (source-destination, source-only). Additionally \icode{add}, \icode{sub} and \icode{mult} are provided multiple times to allow e.g. multiplication of two matrices A and B as $A^{\tau}B$ without the need to transpose A explicitly before.\\[10pt] The \icode{DynMatrix}-template class is very tightly integrated with the \icode{Img}-template class. The index operator (\icode{Img::operator[](int channel)} returns an instance of \icode{DynMatrix} that wraps the images channel data shallowly. This enables the programmer switch between image- and matrix notation very easily. Besides \icode{Img} instances can be created as shallow wrappers of a set of \icode{DynMatrix}-instances (one \icode{DynMatrix}-instances corresponds to one image channel) by using an appropriate \icode{Img}-constructor, that takes up to 5 \icode{DynMatrix}-arguments.\\ This also allows the user to apply arbitrary image processing functions on \icode{DynMatrix} instances. \subsection{Time and Timing classes} ICL provides some time and timer classes to facilitate handling of time values and frame rates: \begin{itemize} \item The \icode{Time} class (\iclclassref{Utils}{Time}) is the basis for timing classes. It provides a static function \icode{now()} to obtain the current system time, as well as a variety manipulation and conversion functions. \item The \icode{Timer} (\iclclassref{Utils}{Timer}) class allows to compute time differences e.g. for benchmarking. \item The \icode{StackTimer} (\iclclassref{Utils}{StackTimer}) can be used with the macro \icode{BENCHMARK\_THIS\_FUNCTION} and \icode{BENCHMARK\_THIS\_SECTION} to profile a function. \item \icode{FPSEstimator} (\iclclassref{Utils}{FPSEstimator}) and \icode{FPSLimiter} (\iclclassref{Utils}{FPSLimiter}) can be used to compute or to limit the FPS\footnote{Frames Per Second} count of a application. \end{itemize} \subsection{Threading Classes} The \icode{Thread}-class (\iclclassref{Utils}{Thread}) and the \icode{Mutex}-class (\iclclassref{Utils}{Mutex}) are simple object-oriented wrappers for the pthread library. \subsection{ProgArg-Environment} As a lot of applications need to parse command line parameters, ICL provides a simple interface to define, explain, extract and parse command line arguments (\iclheaderref{Utils}{ProgArg}). Application arguments are always fixed and need to be accessible globally, so the program argument environment is implemented as a set of global functions each having the \icode{pa_}-prefix. \icode{pa\_init()} can be used to define the list of allowed program arguments and their specific count of sub-arguments\footnote{E.g consider a program argument \icode{-size} could be implemented with a single sub-argument: the size.}. \icode{pa\_exmplain()} explains sub-arguments. \icode{pa\_defined()} and \icode{pa\_arg()} are used to obtain information of the actual given program arguments. The \icode{pa\_subarg()} template function can be used to extract and parse given sub-arguments e.g. as \icode{int}: \displaycode{int c = pa\_subarg("-channels",0,3);} or as \icode{std::string} \displaycode{Size size = translateSize(pa\_subarg("-size",0,"VGA"));}. More information can be found in the documentation. \subsection{Utility Structures} Firstly, there are 3 main utility structures: \begin{enumerate} \item The \icode{Size}-class (\iclclassref{Core}{Size}), which in substance consists of an \icode{int}-width and -height parameter, represents e.g. image-, widget-, ROI, or screen sizes. \item The \icode{Point}-class (\iclclassref{Core}{Point} ; and it's \icode{float}-precision variation \icode{Point32f}; \iclclassref{Core}{Point32f}), should be used to defined 2D-locations e.g. pixel locations, ROI-offsets or perhaps region-centers. \item At last, the \icode{Rect}-class (\iclclassref{Core}{Rect}; also available with floating point precision; \iclclassref{Core}{Rect32f}) with essentially combines a \icode{Point} and a \icode{Size} to represent rectangular regions, bounding-boxes, ROI's etc. \end{enumerate} All these classes are derived from corresponding IPP data types (\icode{ippPoint}, \icode{ippSize} and \icode{ippRect}) in case of having IPP support, which enables the programmer to pass e.g. an \icode{icl::Point} directly to an IPP-function. \subsubsection{Lines} The \icode{Line} and \icode{Line32f} classes (\iclclassref{Core}{Line}, and \iclclassref{Core}{Line32f}). Are provided not only as an interface for representing 2D-Lines given by two points. In actual fact, they provide an efficient mechanism to sample lines into an image grid using Bresenham's line algorithm\footnote{See \hrefn{http://de.wikipedia.org/wiki/Bresenham-Algorithmus}}. \subsubsection{Ranges} At last, there's a template class representing ranges: \iclstructref{Core}{Range} and it's extension \iclstructref{Core}{SteppingRange}. \subsection{String-Utilities} To facilitate string processing (which is still a bit complicated in C++), the header \iclheaderref{Utils}{StringUtils} is provided in the ICLUtils package. It contains functions to parse \icode{std::strings} into given data types (template based, \icode{T parse(const std::string&)} and to convert given data-types into strings (\icode{std::string str(const T&)}). Furthermore, there are variants of these functions, that work on \icode{std::vector} instances to convert or parse ranges at once.\\ Moreover, ICL provides conversion functions to translate sizes, formats, depths etc. into strings and back. These functions are located in the \iclheaderref{Utils}{Utils} and their names are \icode{translateSize}, \icode{translateFormat} etc. \subsection{Types} Types and enumerations have been presented and discussed in section \ref{sec:typedef-confusions}. Types are declared in \iclheaderref{Utils}{BasicTypes}. \subsection{XML-Parser} To be able to use xml-data and configuration files without getting an extra software-dependency, ICL uses it's own simple XML-Parser. This of course entails some issues on the complexity of XML-files, that can be parsed. Actually, some features (like namespaces or XML-schematics) are just not supported at all.\\ However, the XML-Parser works well for our purpose: It can be used to easily parse XML-documents or to query and filter nodes from the Documents tree structure. Furthermore, we use it successfully to create and to parse instances of the \icode{ConfigFile} class, that provides high convenience for using configuration files in applications (See \ref{subsec:config-file}) \subsection{Configuration Files\label{subsec:config-file}} The \iclclassref{Utils}{ConfigFile}-class can be used to facility handling of dynamic configurable applications. Of course, one can make heavily use of program arguments, however this makes is difficult to save parameters or to distribute configuration parameters via revision control systems like e.g. subversion\footnote{\hrefn{http://de.wikipedia.org/wiki/Subversion\_(Software)}}. \icode{ConfigFile} instances can be created very easily in the C++ source code, and their contents can be obtains very simply. Most of the time, a simple one-line statement is sufficient to get a certain entry at runtime. On the other side, the configuration file itself uses a simple XML-subset which facilitates both, parsing of the document as well as referencing entries non-ambiguously. Here's a tiny example: \codefile{config.xml}{config.xml} Corresponding source file: \codefile{config-file.cpp}{ConfigFile use case} And resulting configuration file: \codefile{other.xml}{Written configuration file other.xml} \subsection{ConfigEntry Utility class} The \icode{ConfigEntry}-template class \iclclassref{Qt}{ConfigEntry} provides a convenient access to data entries of a ConfigFile instance. \icode{ConfigEntry} instances are always \emph{templated} to their actual type: e.g. \icode{ConfigEntry} or \icode{ConfigEntry}. \icode{ConfigEntry} objects can be instantiated with a given key-value (which is the '.'-separated path to the referenced ConfigFile element), a default value (which is used, if the given key was not found) and the referenced ConfigFile (which is the singleton static ConfigFile instance by default). Furthermore, \icode{ConfigEntry}-instances can simply be used in the same way as their template parameter type, because they provide an implicit cast-operator for this type.\\ Using ConfigEntry instances is useful if your application also uses the \icode{ConfigFileGUI} utility class \iclclassref{Qt}{ConfigFileGUI} which is presented in \ref{subsec:configfile-gui}.\\ In addition, the header \iclheaderref{Qt}{ConfigEntry} provides some convenience macros to create local instances of the \iclclassref{ConfigEntry} class. \section{ICLCore} [Documentation: \iclpackageref{Core}]\\ The Core-package contains, apart from the image base class \icode{ImgBase} and template classes \icode{Img}, fundamental image processing functions and utility classes. In particular, it contains the header \iclheaderref{Core}{Types}, that declares most common ICL data types and enumerations. Furthermore, there are mathematic functions, random number generators, utility functions for converting and copying images and some utility structures like rectangles, points and lines. \subsection{Image classes} The image classes have been described in detail in chapter \ref{cha:img-base-functions} and \ref{cha:img-class-functions}. \subsection{Mathematic Functions} Also some fundamental mathematic functionalities are provided in the package. The header \iclheaderref{Core}{Mathematics} contains some stochastic utility template functions for computation of mean, variance and standard derivation of a range. Also color-histogram calculating functions (\icode{hist} and \icode{channelHisto}) are available by including this header. More information can be obtained from the header documentation (mentioned above). \subsection{Random Number Generators} Formerly a part of the header \iclheaderref{Core}{Mathematics}-header, random number generators are not provided by the header \iclheaderref{Core}{Random}. Actually, only two different distributions are currently supported: uniform- and gaussian distribution. However a set of functions are provided for each distribution to facilitate random number (and random range) generation. \\ Firstly, there's an overloaded \icode{randomSeed()} function, that can be used to initialize the systems random generator engine, with an optinal \emph{seed-value}. If no seed value is given, current system time is used for initialization. Furthermore, a class called \icode{RandomSeeder} (\iclstructref{Core}{RandomSeeder}) is provided which can be instantiated on some stack to force random number generation engine initialization at program startup. \\ In Addition, there's a set of initializer functors (\iclclassref{Core}{URand},\iclclassref{Core}{URandI},\iclclassref{Core}{GRand} and \iclclassref{Core}{GRandClip}), that can be used e.g. with \icode{std::fill} to initialize range elements randomly. \section{ICLCC\label{sec:cc}} [Documentation: \iclpackageref{CC}]\\ The \textbf{C}olor \textbf{C}conversion package provides a powerful environment for converting images color formats. Additionally data layout conversion functions (\icode{interleavedToPlanar} and \icode{planarToInterleaved} are implemented in this package (available by including the \iclheaderref{CC}{CC}-header) and a powerful \icode{Converter}-class (\iclclassref{CC}{Converter}), that allows convert images with different sizes, formats and depths\footnote{or even all at once} into each other while preserving image content. \subsection{Color Conversion} As illustrated in section \ref{sec:image-features} ICL images possess a \icode{format} parameter that defines the images actual color format (e.g. RGB HLS or gray). As most algorithms process images regardless their color formats, color conversion was outsourced into the dedicated \iclpackageref{CC}-package. Basically, all color conversion algorithms are implemented with a single function called \icode{cc} (also available in the \iclheaderref{CC}{CC}-header. \icode{cc} is able to convert pixels from a source image into a given destination image. Source and destination color formats are extracted automatically from the given images. Optionally, \icode{cc} can be setup to process an images ROI only.\\ Base format for color conversion is the \emph{sRGB}\footnote{\hrefn{http://en.wikipedia.org/wiki/SRGB}} format (represented by the \icode{format}-enum value \icode{formatRGB}). All other formats can be converted \emph{from} and \emph{into} the sRGB-format natively. Some other \emph{cross-conversions} are implemented by first converting source image pixels into sRGB, before conversion from sRGB into the desired destination format is available. The \iclheaderref{CC}{CC}-header provides a good overview of the implemented formats and their conversion performance.\\ If color conversion performance is not fast enough, it can be optimized using a lookup table, that can be created on automatically on demand by calling the \icode{createLUT} function.\\ By the way, the color conversion function \icode{cc} is implemented using massive template-expansion. This enables \icode{cc} to perform a \icode{depth}-conversion simultaneously (by the way, this is why \icode{iclCC.cpp} needs that much time to compile). \subsection{General Parameter Conversion} If also an images size must be adapted, there's need for some temporary buffers. The \icode{Converter} (\iclclassref{CC}{Converter}) class automatically handles internal buffers for \icode{depth}-, \icode{size}- and \icode{format}-conversion. Furthermore it provides the ability to select the order of different conversion steps. \subsection{Data Layout Conversion} As motivated in chapter \ref{cha:what-is-the-icl}, ICL's images do always use planar data layout, however sometimes available image data is interleaved, or ICL-image data must be converted into interleaved data layout. The \iclheaderref{CC}{CC}-header also provides two conversion functions for this tasks: \displaycode{interleavedToPlanar} \displaycode{planarToInterleaved} Both functions are highly optimized and -- if supported -- IPP accelerated. Additionally, both functions are able to apply a \icode{depth}-conversion too, and the provide source and destination ROI support. As there's no interleaved image format in the ICL, interleaved image data is always just represented by a single data pointer; the interleaved images ROI is determined from the given \icode{lineStep} argument. \section{ICLIO\label{sec:io}} [Documentation: \iclpackageref{IO}]\\ The IO-package provides functions and classes for reading and writing image from- and to different devices. The most common interface is the \icode{Grabber} (\iclclassref{IO}{Grabber}). All image input devices are implemented as specializations of this \icode{Grabber}-class, which provides a generic interface for grabbing images and setting camera parameters and capture modes (see section \ref{sec:grabber-feature-adjustment}). Currently, the following grabbers are available: \begin{enumerate} \item \icode{FileGrabber} [\iclclassref{IO}{FileGrabber}]:\\ For reading image files in most different formats. .ppm, .pnm and .pgm as well as a special icl-format \emph{.icl} are implemented natively without any external software dependencies. .jpeg-files can be read with a wrapper of libjpeg. All other file-formats are passed to an libImageMagic++\footnote{\hrefn{http://www.imagemagick.org/Magick++/}}-wrapper (currently this supports about 100 more or less common file formats) % \item \icode{DCGrabber} [\iclclassref{IO}{DCGrabber}]:\\ The \icode{DCGrabber} grabs images from fire-wire aka IEEE-1394 camera devices. Basically it wraps the libdc1394 library\footnote{\hrefn{http://sourceforge.net/projects/libdc1394/}} for accessing cameras, that meet the IIDC DCAM standart\footnote{\hrefn{http://de.wikipedia.org/wiki/DCAM}}, and which gave the grabber it's name. DC-Camera features, image acquisition modes and framerates are made accessible via the Grabbers feature \emph{setter}- and \emph{getter}-functions. % \item \icode{PWCGrabber} [\iclclassref{IO}{PWCGrabber}]:\\ The \textbf{P}hillips \textbf{W}eb\textbf{C}am grabber can be used to grab images from cameras that use the linux pwc kernel module\footnote{\hrefn{http://www.saillard.org/linux/pwc/}}. \item \icode{UnicapGrabber} [\iclclassref{IO}{UnicapGrabber}]:\\ This grabber wraps the \textbf{unicap}-library\footnote{\hrefn{http://unicap-imaging.org/}} which provides a generalized access to fire-wire as well as video-4-linux/video-4-linux-2 devices. \item \icode{VideoGrabber} [\iclclassref{IO}{VideoGrabber}]:\\ This grabber implementation uses libxine\footnote{\hrefn{http://www.xine-project.org}} to use video files as image data source. The VideoGrabber provides functions to adjust video playback speed (by default, videos are grabbed frame by frame, but not faster then the actual movie framerate). Furthermore, if video playback speed is close to the movie framerate, the VideoGrabber will also playback sound. \item \icode{SwissRangerGrabber} [\iclclassref{IO}{SwissRangerGrabber}]:\\ The SwissRanger is a so-called time-of-flight camera from the Mesa-Imaging company\footnote{\hrefn{http://www.mesa-imaging.ch}}. It uses a set of pulsed IR-LED's to compute depth images. This grabber uses the libmesasr-library for accessing camera features and image data. The SwissRanger camera also provides gray-scale images and a confidence map, which tells the user the estimated accuracy for each depth-pixel. \item \icode{DemoGrabber} [\iclclassref{IO}{DemoGrabber}]:\\ This is just a tiny utility class, which can be used e.g. to test an application if no appropriate source device is currently available. It creates images with a small moving red rectangle. \item \icode{XCFPublisherGrabber} [\iclclassref{IO}{XCFPublisherGrabber}]:\\ This grabber utilizes the XCF-framework to receive images via network. Images can e.g. be send from other applications using the \icode{XCFPublisher} class (\iclclassref{IO}{XCFPublisher} or the demo application \icode{icl-xcf-publisher}. \item \icode{XCFMemoryGrabber} [\iclclassref{IO}{XCFMemoryGrabber}]:\\ Here also XCF is used, but now in combination with the ActiveMemory which is an XML-based active blackbord for interprocess communication. \item \icode{XCFServerGrabber} [\iclclassref{IO}{XCFServerGrabber}]:\\ This is the 3rd implementation of an XCF-based grabber. This grabber utilized XCF's remote method invocation concept for grabbing images from an image server. \item \icode{GenericGrabber} [\iclclassref{IO}{GenericGrabber}]: \textbf{This is the grabber that should be used by your application!}\\ The \icode{GenericGrabber} internally wraps one of the other grabbers. Which actual \icode{Grabber}-instance is used internally depends on a string values, which is passed to the \icode{GenericGrabber}-constructor. This enables the programmer to create applications, that are able to work with most different input devices. Using the \icode{GenericGrabber} instead of some other Grabber instances directly provides a lot of advantages: \begin{itemize} \item Applications that use a generic grabber can simple be set up to use any input device \item In particular, one can easily switch from a camera device to stored image files \item The compilation procedure of applications doesn't have to check for a particular Grabber/library to be present or something \item There's only a single additional pointer access for this improved flexibility \end{itemize} The \icode{GenericGrabber} constructor gets two arguments. The first argument is one (or a comma-separated list) of \icode{pwc}, \icode{unicap}, \icode{dc}, \icode{file} or \icode{demoe} and determines the grabber type. If a list if given, it is searched successively for a supported device type. The second argument is also of type \icode{std::string} and provides additional information for the creation of the internal Grabber instance. As each Grabber needs other constructor parameters, this argument's syntax depends on the first argument as well. To allow to specify also arguments to a list of possible device types, the device type is also given here again. Some examples will explain this in detail: \begin{itemize} \item \icode{GenericGrabber("pwc","pwc=0")}: Creates a \icode{GenericGrabber} instance that wraps a \icode{PWCGrabber} for \texttt{/dev/video0} \item \icode{GenericGrabber("pwc,file","pwc=0,file=*.ppm")}: As above, but with file grabber fallback \item \icode{GenericGrabber("file","file=images/*.ppm")}: \icode{FileGrabber} that uses all files in \texttt{./images} with suffix \texttt{.ppm} \item \icode{GenericGrabber("dc","1")}: internally a \icode{DCGrabber} is wrapped, that uses DC-device index 1 (available devices are enumerated internally) \item \icode{GenericGrabber("unicap","*Phillips*")}: Use the first unicap device, which matches the given string pattern \item \icode{GenericGrabber("pwc,file,dc","pwc=0,file=images/*.ppm,dc=1")}: An example combination \end{itemize} In most applications, the actual Grabber, that should be used is determined by a progarg-argument (See \iclheaderref{Utils}{ProgArg}). To enhance convenience for instantiation of \icode{GenericGrabber}s using progarg input, a special macro \icode{FROM\_PROGARG} is provided. The following example illustrates how to use this macro. \codefile{generic-grabber-from-progarg.cpp}{Creation of a \icode{GenericGrabber} instance from a programm argument} Now, the resulting application (we call it \icode{app} here) can be run e.g. with \icode{app -input dc 0} to use \icode{DCGrabber} input. or with \icode{app -input file images/*.ppm} to use image files as input.\\ \textbf{Please note} that the progarg-name \icode{-input} is ICL's default for selection of an application's input device. Applications should not break with this conventions, unless, there are more than one input devices that must be specified somehow else (e.g. with \icode{-left} and \icode{-right} resp.) \end{enumerate} \subsection{What is the best Grabber for Fire-Wire devices?} Unicap is partially redundant to the \icode{DCGrabber} class, however both interfaces are completely independent. Grabbing from Fire-wire devices runs a little bit faster with the \icode{DCGrabber}-class. The \icode{DCGrabber} class is optimized to provides images with a minimal latency and is a bitter better here. \subsection{Feature Adjustment\label{sec:grabber-feature-adjustment}} The \icode{Grabber}-interface \iclclassref{IO}{Grabber} defines not only the way, how images can be obtained from \icode{Grabber}-implementations, but also how grabber parameters or properties can be got and set. All grabbers have a special list of properties, which can be obtained by the \icode{getPropertyList}-function. This function must be implemented by all classes that are derived from the \icode{Grabber} class. It returns an \icode{std::vector} that contains the \textbf{names} of the supported properties. Each property has a so-called \emph{type}, that is defined by the grabber too (the type of a property is returned by the \icode{getType(propertyName)} function. The type of the property defines how possible property value are defined. Possible types are: \begin{itemize} \item \textbf{range}\\In this case the property may have values within a defined float range (sometimes with a given fixed stepping between valid values). In a GUI , one might just use a slider to adjust such properties. \item \textbf{value-list}\\In this case the properties possible values are defined by a list of possible float values. GUI's will probably use a combo-box containing possible float values for this kind of property. \item \textbf{menu}\\Here, also a list defines allowed values, but here, values are strings. Again, such properties can best be adapted interactively by using a combo-box GUI element. \item \textbf{command}\\This is a very special property type. Commands have no values at all. A command is e.g. used to make PWC-based camera save it's current parameters to the permanent camera memory. A GUI will provide commands maybe using a push-button component. \item \textbf{info}\\ Info properties are read-only, so info-properties can't be set from externally. \end{itemize} Having estimated which type a certain property has (using the \icode{getType(propertyName)} method in the Grabber class), the property value can be got/set using \icode{getValue} and \icode{setValue} respectively. \subsection{Desired Parameters} The \icode{Grabber}-interface \iclclassref{IO}{Grabber} also provides functions that handle the Grabbers desired parameters. To explain this one must understand, that there are always two different kinds of image parameters: \begin{itemize} \item The image parameters, that result from the current settings of the input device. E.g. a firewire camera might provide an image acquisition mode with VGA resolution and RGB color format or (to save bandwidth of the firewire interface) also another VGA-mode where images are YUV-411-encoded. We call this image parameter set the \emph{device-parameters}. \item In addition there is another kind of parameters, which we call the \emph{desired parameters}. This parameters describe the size, format, etc. of images that are produced by the \icode{Grabber}-instance. All Grabbers will ensure, that new images have exactly this desired format unless a special flag is set (function \icode{setIgnoreDesiredParams}). \textbf{Note:} In general device parameters are not adapted to simplify creation of an appropriate new image, but the image in device-parameters is converted into the desired parameters. \end{itemize} We decided to have this two parameter sets to make grabber instances abstract from the underlying device, however there are situations where it is useful to be able to deactivate this mechanism: E.g. if one uses a \icode{FileGrabber} instance, he might want to read the images in the real size, or when obtaining images via network (\icode{XCFPublisherGrabber}). Most of the time camera grabbers provide RGB images when the ignore-desired-params flag is set to true. \subsection{Using Grabbers} The essential class of the IO package is the \icode{Grabber}-class. Here's a short example of how to use a Grabber \codefile{using-a-grabber.cpp}{Simple use of a \icode{Grabber}-instance} As one can see, the essential function to grab the next image from a \icode{Grabber} instance is called \icode{grab}. The \icode{grab(..)} can be called in two different ways: If no parameter of type \icode{ImgBase**} is given, the grabber internally handles a buffer image, which is adapted to appropriate parameters and returned as \icode{const ImgBase*} (\icode{const} to emphasise that the \icode{Grabber} instance owns this image. Grabbers ensure, that this image is valid and not changed until the next call to the \icode{grab()}-function or of course until the Grabber instance is released. If one wants to have read/write access to a grabbed image he has several options: \begin{itemize} \item The first option is to pass an \icode{ImgBase**} to the grab function. In this case the grabber will adapt the referenced image parameters or -- if the given \icode{ImgBase**} points to a NULL-\icode{ImgBase*} -- even create it. \textbf{Note:} That some grabbers do not support this feature, however, this is the most accurate version that has also an optimal performace. \item Another option is to copy the obtained image using the \icode{ImgBase}-function \icode{deepCopy}. This function returns a new \icode{ImgBase*} which has to be deleted lateron. \item If the user wants to have direct access to the image pixels, he has check the actual image depth. This can be bypassed by using the \icode{ImgBase}-function template \icode{convert}. \item However sometimes if one know what he's doing it might be possible just to remove the const-ness from the obtained image by using the \icode{const\_cast}-operator of C++. \end{itemize} The following example demonstrates how to access pixels from a recently grabbed image. \codefile{using-grabbed-images.cpp}{How to access pixels of grabbed images} \subsubsection{Distortion Compensation\label{subsubsec:distortion-compensation}} The \icode{Grabber}-base class also provides an interface to enable distortion compensation. The \icode{Grabber}s \icode{enableDistortion} function can be used for this. Appropriate distortion parameters can be obtained by using demo application \icode{icl-intrinsic-camera-calibration} which is discussed in detail in section \ref{subsec:icl-intrinsic-camera-calibration}. Internally, distortion compensation is always applied by using a warp-operation, i.e. pixel offset are pre-calculated and stored in a so called warp-map (see \iclclassref{Filter}{WarpOp} for further details). \subsection{File class} The \icode{File}-class \iclclassref{IO}{File} is a simple File implementation, that provides functions to read or to write files, or to check whether a given file exists. Furthermore, the \icode{File} class is able to handle zipped files if the ICL was compiled with zlib support. \section{ICLBlob\label{sec:blob}} [Documentation: \iclpackageref{Blob}]\\ The package \iclpackageref{Blob} provides functions for blob-detection/tracking. The most common class of the package is the \icode{RegionDetector} (see \iclclassref{Blob}{RegionDetector}), which performs a fast connected components analysis on images. The package subsumes different approaches for blob detection, which are presented in the following subsection. \subsection{What is a \emph{Blob}} Here, a \emph{Blob} is a set of connected pixels, that have to fulfill a certain criterion of homogeneity (e.g. all pixels have exactly the same value, or all pixels euclidian color distance to a reference color is below a certain threshold). The term of \emph{connection} must also be defined more exactly, to avoid misunderstandings: We assume pixels to be connected to all pixels in its neighborhood, which in turn is given by the 4 pixels (to the left, to the right, above and below) next to the reference pixel. Even though an also very common neighborhood defined by the 8 nearest pixels to a reference pixel is often used, we resign of it because of its higher computational complexity. \subsection{Color based Blob detection} The first part of the \iclpackageref{Blob}-package is the Color based Blob detection framework, which essentially consists of the classes \iclclassref{Blob}{ColorBlobSearcher}, \iclclassref{Blob}{DefaultColorBlobSearcher}, \iclclassref{Blob}{FoundBlob}, \iclclassref{Blob}{PixelRating} and \iclclassref{Blob}{PixelRatingGroup}. Introducing documentation can be found in the reference of the \iclclassref{Blob}{ColorBlobSearcher} class.\\ Within the color based blob detection framework, blobs are detected globally, so different blobs must have different colors. \subsection{Region detection / connected component analysis} Here, the \iclclassref{Blob}{RegionDetector} is the main class. The Region detector performs a fast connected component analysis on images and provides a list of all connected regions in an image (again using the 4-pixel neighbourhood). The \icode{Region} (see \iclclassref{Blob}{Region}) provides many function to compute region features like the center of gravity, the boundary or even local PCA\footnote{Principal Component Analysis (google)} information. Internally, the \icode{RegionDetector} uses an intermediate representation of \icode{Region}s: a list of so called \emph{scan lines}. These scan lines are horizontal image lines with a single identical gray value. Furthermore, they can efficiently be stored by a tuple $(x_{start},x_{end},y,value)$ and most region features can be calculated very efficiently on this lines (in comparison to calculate features on the image pixels). Currently there's no possibility to define certain criterions of homogeneity, than the most simple one which is currently used: all pixels within a certain region must have an identical gray value. \subsubsection{SimpleBlobSearcher} On top of the \icode{RegionDetector} class there's a higher level wrapper for detection of colored blobs in images: the \icode{SimpleBlobSearcher} (see \iclclassref{Blob}{SimpleBlobSearcher}). This class is well documented and shall not be explained here in detail. However, it's nice to know, that the \icode{SimpleBlobSearcher} provides a very simple interface for a reference-color/threshold based detection of multiple blobs in an image. \subsection{Blob-Tracking} In comparison to blob detection, \emph{blob tracking} means to track blobs/regions successively through time. Here, the ICL currently provides two different approaches: \begin{itemize} \item A simple MeanShiftTracker implementation (see \iclclassref{Blob}{MeanShiftTracker}). The mean-shift algorithm is a very common algorithm in computer vision, which is well-discussed and well interpreted in literature. Our implementation uses the basic mean shift algorithm rather than one of its various extensions. \item A generic yet powerful layered approach which's basic idea was also patented somewhere from somewhere else\footnote{we found that out later}. However, our tracking assignment mechanism better. The most common class is the \icode{PositionTracker} (see \iclclassref{Blob}{PositionTracker}) and the \icode{VectorTracker} (see \iclclassref{Blob}{VectorTracker}). For more details start reading here: \iclclassref{Blob}{PositionTracker}. \end{itemize} \section{ICLFilter\label{sec:filter}} [Documentation: \iclpackageref{Filter}]\\ The ICLFilter package provides a large variety of image filtering classes. When talking about image filters some misapprehensions can arise. To prevent this, the following sub-section will shortly introduce our understanding of image filters. \subsection{Our Definition of Image \emph{Filters}} In a most general view on image filters, a filter can be seen as a kind of black box that has N image inputs and M image outputs. However this view brings us a very generic interface for image filters which is most of the time not very feasible. \\ Most common filters (e.g. binary image operations, linear filters or neighborhood operations) only need a single input and output image. Another larger group of filters subsumes filters that get exactly two input images and that have only a single output image (e.g. arithmetical/logical per-pixel image operations or image comparison filters). To avoid a large computational overhead arising of a too general interface, we currently support two dedicated filtering interfaces for the above mentioned 1-1 and 2-1 input-output combinations. To obviate further misunderstandings, we call this filter groups \emph{unary operations} and \emph{binary operations} respectively (represented by the C++-interfaces \iclclassref{Filter}{UnaryOp} and \iclclassref{Filter}{BinaryOp}). For a better overview, the ICLFilter package documentation is grouped into the following modules: \begin{enumerate} \item Unary operations [\ref{subsec:unary-ops}] \item Binary operations [\ref{subsec:binary-ops}] \item Affine operations (also unary) [\ref{subsec:affine-ops}] \item Neighbourhood operations (also unary) [\ref{subsec:nbh-ops}] \item In-place operations [\ref{subsec:inplace-ops}] \end{enumerate} \subsection{Unary Operations\label{subsec:unary-ops}} Actually, most implemented operations have this type. All implementations in this group implement the \icode{UnaryOp}-interface (see \iclclassref{Filter}{UnaryOp}). Unary operations implement a 1-1 input/output relation which entails, that unary operations cannot be used to apply in-place operations on images\footnote{see subsection \ref{subsec:inplace-ops} for this kind of operations}. Unary operations \emph{always} use two different images: an input image and an output image. If you want to apply a unary operation on a source image you can either manage the destination image by yourself, using the \displaycode{void apply(const ImgBase *src, ImgBase **dst)} method or you can let the \icode{UnaryOp} instance manage the destination image internally by using the \displaycode{const ImgBase *apply(const ImgBase *src)} method. In both cases, the destination image will be adapted to appropriate parameters (which depend on the source image parameters and -- of course -- the certain \icode{UnaryOp}-implementation). In subsection \ref{subsec:unary-op-preview} we already presented some examples of \icode{UnaryOp} usage.\\ Most \icode{UnaryOp}-subclasses work on the source image's ROI only.\\ A list of all \icode{UnaryOp} implementation can be found in the doxygen collaboration diagram in \iclclassref{Filter}{UnaryOp} \subsubsection{The bpp-function\label{subsec:bpp}} The \icode{bpp} function is a part of the \iclpackageref{Core}-packge and it was already mentioned in section \ref{sec:raw-data-access-for-ipp-calls}. In some special cases (e.g. see the example in subsection \ref{subsec:check-only}), the \icode{ImgBase**}-interface is somehow too general, keeping in mind, that we only need it to enable a function to re-allocate a given image for depth adaption. However, if the programmer is sure that his image has correct depth, he might have to face the following complications: if the destination image is of type \icode{Img} i.e. it's no pointer type, the naive approach to get an \icode{ImgBase**}: \displaycode{Img image; ImgBase **base = &ℑ } leads to a compiler error because C++ inheritance works only with pointers, but not with pointers to pointers. In other words: a C++ pointer of type \icode{derived*} has always also the type \icode{base*}\footnote{assuming \icode{derived} inherits \icode{base}}, but \icode{derived**} and \icode{base**} have no common type at all. This is why the type of \icode{&&image} is \icode{Img**} but not \icode{ImgBase**}. As a matter of fact, \icode{&&image} can also not even be casted easily to the \icode{ImgBase**} type. Furthermore, the application will crash, if after all depth was incorrect, because the implementation would then maybe try to call \icode{delete} on a pointer to an object on the stack.\\ All this problems are solved by the \icode{bpp}-function. It returns an instance of an internal utility structure which, then again can be casted \emph{implicitly} into an \icode{ImgBase**}. It also checks whether the given image was mistakenly reallocated and notifies this with an error message. \subsubsection{The ClipToROI Flag\label{subsec:clip-to-roi}} The \icode{UnaryOp}-class has two special flags, that can be set from externally to influence how input and output images are processed. If the \emph{ClipToROI}-flag is set to \icode{false} (by default, it is set to \icode{true}), the output image size is set to the source image size. As still only the source images ROI is processed, the destination images ROI is set up appropriately here. E.g. if the source image has VGA size\footnote{$640 \times{}480$ pixels} and a centered ROI of size QVGA\footnote{Quater-VGA = $320\times{}240$ pixels}, The destination image will be set up with this parameters as well, however all pixels that are outside it's ROI are not changed. If otherwise, \emph{ClipToROI} is set to \icode{true}, the destination image is clipped to these pixels that are actually determined from the source image. This means for the example above, that the destination image's size is set to the ROI size of the source image (QVGA)\footnote{In some cases, the destination image size/ROI-size becomes even smaller (see subsection \ref{subsec:nbh-ops} for more information about this)}. \subsubsection{The CheckOnly Flag\label{subsec:check-only}} In a few situations, the destination image shall neither be adapted to have the source image size nor to have the size of the source images ROI. E.g. if you want to store results from different \icode{UnaryOp}'s into a single image. In this case, the programmer can set the \icode{UnaryOp's} \emph{CheckOnly} flag to \icode{true}. Now the \icode{UnaryOp}'s \icode{apply}-function will no longer apply any changes to the destination image. It will only check, whether the current destination image parameters are appropriate. I.e. channel count and depth matches (not always exactly to the source image, but these parameters have to meet those that are expected by the certain \icode{UnaryOp} implementation). Furthermore, the destination image's size and ROI size must have been set up correctly, before. Most of the time, it is necessary \textbf{to set ClipToROI to \icode{false} if you set \emph{CheckOnly} to \icode{true}}. The following example shows a common usage of these two flags. \codefile{check-only-example.cpp}{Example for usage of ClipToROI and the CheckOnly flags} This is the shown result image: \includegraphics[width=300pt]{media/check-only-example-output} \subsubsection{UnaryOp-pipes} The generic interface of the \icode{UnarypOp} class already suggests successive pipelining of \icode{UnaryOp} instances. Such a pipelining mechanism is provided by the \icode{UnaryOpPipe}-class (see \iclclassref{Filter}{UnaryOpPipe}). Furthermore, the \icode{UnarpOpPipe} class itself inherits the \icode{UnaryOp}-interface by passing given input images to the first \emph{piped} \icode{UnaryOp}-instance and letting the last \icode{UnarypOp} instance use the given output image as it's destination image. The following code section gives a short example for the usage of the \icode{UnarypOpPipe} class. \codefile{unary-op-pipe-example.cpp}{Example for usage of the UnaryOpPipe class} \subsection{Binary Operations\label{subsec:binary-ops}} In contrast to unary operations, binary operations (base class \iclclassref{Filter}{BinaryOp} implement a 2-1 input/output relationship. Binary operations are e.g. a pixel-wise addition and multiplication of images or even a proximity measurement pixel-wise comparison of two images. The \icode{BinaryOp} class uses another signature for its \icode{apply}-function: \displaycode{void apply(const ImgBase *operand1, const ImgBase *operand2, ImgBase **dst)} As you can see, the only difference to the \icode{UnaryOp}'s interface is an additional second source image argument. All other aspects like ROI-handling or adaption of the given destination image remain equal. In addition to this, when calling the \icode{apply}-function of \icode{BinaryOp} instances, the two source images must have compatible parameters. This means in general, that the two given source images must have identical parameters, however there are some classes where this is not necessary or even not useful. \subsubsection{Example: Arithmetical Operations\label{subsec:arithmeticl-unary-ops}} A common prototype for binary operations is the \icode{UnaryArtithmeticalOp}-class (see \iclclassref{Filter}{UnaryArtithmeticalOp}). Basically, this class provides pixel-wise addition, subtraction, multiplication and division of images. In addition to this, a special, but also quiet common pixel-wise operation can be used: the absolute value of the difference two images. However, this could also be emulated by a successive use of a binary \emph{minus}-operations followed by a unary \emph{abs} operation, the combination is very useful because it's much faster. The speed advantage is also discussed in subsection \ref{subsec:perf-issues-for-unary-and-binary-ops}. \subsection{Performance issues for successive use of BinaryOp/UnaryOp instances\label{subsec:perf-issues-for-unary-and-binary-ops}} As mentioned in subsection \ref{subsec:arithmeticl-unary-ops}, there are different approaches to compute e.g. the pixel-wise absolute value of differences between two images (which is just an example for complex pixel-wise operations on images). The following code example demonstrates 4 possibilities. \codefile{abs-diff-successive-ops.cpp}{Comparison of different methods for calculating the pixel-wise absolute value for the differences between two images\label{codefile:successive-ops}} We have to discuss two different performance relevant aspects: \begin{itemize} \item \textbf{When Intel IPP is used, the actual execution of the arithmetical operation becomes about twice as fast.}\\ Ok, as presented in section \ref{sec:optimal-performace} we use Intel IPP to obtain optimal performace, however IPP's function set is quickly exhausted when deliberating \todo{translate correctly!} about all possible arithmetical operations with 3, 4 or even more operands. Hence operations have to be concatenated to obtain more possibilities by using combinations of them. But if this is done, each single operation will need a temporary image, and lots of data must flow through the processor and the RAM $\rightarrow$ processor interface, which leads us directly to the next point: \item \textbf{How often must image data be pushed through the common CPU-architecture bottleneck \emph{memory-interface (RAM $\rightarrow$ processor cache)}?}\\ When applying simple operations on data, processing speed depends mainly on the throughput of the processors memory interface. When applying arithmetical operations step by step, a single pixel from source and destination images must be read several times (assuming cache is not large enough to hold all image data). This entails, that also whole images must be read more then once, which directly stresses the memory interface. \end{itemize} In listing \ref{codefile:successive-ops} four different alternatives for a simple arithmetical task were presented: \begin{enumerate} \item \textbf{Baseline: using IPP based \icode{BinaryArithmeticalOp}.} [$\Delta t$: 3.3ms]\\ This is -- apart from some computational overhead that arises from the image and channel management -- presumably the most efficient implementation possible. \item \textbf{Successive use of \icode{BinaryOp} and \icode{UnaryOp} instance.} [$\Delta t$: 5.9ms]\\ Here, also IPP is used, but due to the separation into two parts, computation time is nearly twice as long. \item \textbf{Pure C++-implementation using \icode{Img::combine}} [$\Delta t$: 6.0ms]\\ Even though, IPP performs the actual arithmetical operation much faster, pure C++ nearly reaches the performance of the variant above. \item \textbf{Operator based implementation using ICLQuick} [$\Delta t$: 6.2ms]\\ This is just slightly slower than the 2nd variant although same operators are used internally. We reach this performance here by using a static image buffer selection mechanism to avoid online memory-reallocation and by using shallow copies. \item \textbf{Pure C++-implementation using two runs} [$\Delta t$: 7.4ms]\\ In comparison to the 3rd version, this one is significantly slower, however the performance difference is also not close to 100\%. As the operations (\icode{-} and \icode{fabs}) have not changed, we can assume, that transferring data of two images (source and destination image) to the processor takes about 1.4ms in this case. \end{enumerate} On balance, we can conclude that it's of course optimal to use dedicated operators (like \icode{BinaryArithmeticalOp(absSubOp)}) if one is available for the desired operation. If not, most of the time it's optimal to apply the operation in a single run (e.g. using the \icode{Img::combine} template function). \subsection{Affine Operations\label{subsec:affine-ops}} Affine operators apply an affine transformation to pixel-\emph{locations} (e.g. image rotation, or translation). ICL's \icode{AffineOp} (see \iclclassref{Filter}{AffineOp}) class allows to concate sequences of \emph{translations}, \emph{rotations} and \emph{scalings}. Internally, the actual affine transformation is applied using a affine transformation $3 \times{}3$ matrix. Concatenation can be optimized by pre-multiplying the transformation matrices. \subsection{Neighbourhood Operations\label{subsec:nbh-ops}} Another large set of common image processing functions can be formulated as neighbourhood function. Theoretically this are all image operators, where the value of a result pixel $R(x,y)$ depends on a neighbourhood of the corresponding source pixel $N(S(x,y))$. Most of the time, the neighbourhood is defined by a squared region with center $(x,y)$ with a fixed width/height. ICL's base class for neighbourhood operations is \icode{NeighborhoodOp} (see \iclclassref{Filter}{NeighborhoodOp}). Currently the following subclasses are defined: \begin{itemize} \item \textbf{\iclclassref{Filter}{ConvolutionOp}}\\ The \icode{ConvolutionOp} provides a set of common fixed (and optimized) filters (like sobel, laplace, gaussian, etc.) as well as an interface to define custom linear image filters. The \icode{ConvolutionKernel} (see \iclclassref{Filter}{ConvolutionFilter}) class can be used here. \item \textbf{\iclclassref{Filter}{MedianOp}}\\ The very common image enhancement filter. \item \textbf{\iclclassref{Filter}{MorphologicalOp}}\\ Morphological operations like \emph{dilation}, \emph{opening} or \emph{closing} are also a very common tool in image processing. All morphological operators can also be applied on gray scale images, which is described in detail in the IPP manual. Again, this class provides an interface for both, fixed and optimized common operations as well as an interface to use user defined masks. \item \textbf{\iclclassref{Filter}{WienerOp}}\\ The \icode{WienerOp} class implements the wiener filter to remove adaptive noise from images\footnote{\hrefn{http://de.wikipedia.org/wiki/Wiener-Filter}}. Currently, for this operation is no C++ fallback available. \end{itemize} \subsubsection{Destination Image Size Adaption} All \icode{NeighborhoodOp} implementations use a special mechanism to determine the destination images (ROI) size. Hence a linear filter can only be applied correctly at pixel positions where no kernel elements are outside the source image rect, the destination image always becomes slightly smaller than the source image, unless the \emph{ClipToROI}-flag (see subsection \ref{subsec:clip-to-roi}) is set to false. In this case, the destination images size is -- as usual -- set to the source image size, but the destination images ROI size becomes a bit smaller in comparison to the source images ROI size.\\ Here's common example: we use a VGA\footnote{$640 \times{}480$} source image S with a ROI set to the upper left quarter (offset $(0,0)$, size $320 \times{}240$). Besides, we use a common $3\times{}3$-sobel filter. If \emph{ClipToROI} remains true, the destination images size is set to $318 \times{}238$ (one pixel to each side was removed) and it's ROI is set to the whole image. Otherwise, the destination images size is set to VGA, while it's ROI is adapted appropriately: (offset $(1,1)$, size $318 \times{}238$). \subsection{In-Place Operations\label{subsec:inplace-ops}} In contrast to unary operations (see subsection \ref{subsec:unary-ops}) the \icode{InplaceOp} (see \iclclassref{Filter}{InplaceOp}), provides an interface to apply \emph{in-place} operations on images. Currently, there are two certain implementation of the \icode{UnaryOp} interface: \begin{itemize} \item \iclclassref{Filter}{InplaceArithmeticalOp}\\ Arithmetical operators e.g. to add a constant value to all pixels, or to multiply all pixels by a constant. \item \iclclassref{Filter}{InplaceLogicalOp}\\ Logical and binary inplace operations. \end{itemize} The common interface for inplace operations is the \displaycode{ImgBase *apply(ImgBase *srcDst)} function, which always returns the given source image \icode{srcDst} (to be able to use nested apply-expressions like \icode{OP1.apply(OP2.apply(image))}). Inplace operations can also be implemented using the \icode{Img::forEach} function template, however the current arithmetical and logical operators provide also IPP-support which may speed up computation time significantly. \section{ICLQuick\label{sec:quick}} [Documentation: \iclpackageref{Quick}] First, the part \emph{Quick} of the package name must be clarified. Actually, this package should be named \emph{ICLQuickAndDirty}, to emphasise that it was actually designed for prototyping only. This means, that ICLQuick provides functions and functors that are \emph{as convenient as possible}. Only in the remaining range, we tried to make the code fast (actually, the shallow copy concept of images contributed the major part for this). In contrast to the other packages, ICLQuick contains \emph{ready-to-use} global functions only.\\[10pt] \textbf{The problem}\\ In most vision applications, there are parts of the code, that have to be as fast as possible, while the speed of other parts doesn't matter at all. For these parts, e.g. within the initialization section of an application users can easily use ICLQuick's functions to implement faster, and to keep their code tidier. Furthermore, ICL can also be used to create offline applications, where the speed issue is also less important. Currently the ICLQuick interface consists of two headers: \iclheaderref{Quick}{Quick} and \iclheaderref{Quick}{QuickRegions}.\\[10pt] \textbf{Please note:}\\ The \iclheaderref{Quick}{Quick}-header also contains a \displaycode{using namespace icl} statement! \subsection{ICLQuick's image type ImgQ} Inside the ICLQuick headers, a special image type named \icode{ImgQ} is used, but\\ \textbf{\icode{ImgQ} is just a typedef to \icode{Img32f}}.\\ I.e. ICLQuick is designed for float-images only, which underlines it's prototyping character. To facilitate usage if ICLQuick's function in combination with images of other types, conversion functions are provided (see subsection \ref{subsec:quick-functions}/creators). However the choice of the float image type was obvious as floats are a good compromise between performance and flexibility (see section \ref{sec:image-value-domain} for more details).\\ In this package, all functions work on image references or instances directly, i.e. no pointers are used. Functions that create new images directly return an instance of \icode{ImgQ}, however images are not copied twice deeply here because of ICL's shallow copy concept (see section \ref{sec:shallow-copy-concept}). \subsection{ICLQuick Functions\label{subsec:quick-functions}} The function set of the ICLQuick package can be divided into the following subsets. Most of this functions are declared in the header \iclheaderref{Quick}{Quick}. \begin{itemize} \item \textbf{Creator Functions}\\ Creator functions can be used for a simple creation of new Images. The functions \icode{zeros} and \icode{ones} provide a matlab/octave like interface for image creation: \displaycode{ImgQ a=zeros(640,480,3); // empty 3 channel VGA image } \displaycode{ImgQ b=ones(800,600,formatGray); // gray SVGA image initial. to 1} The \icode{ones} function can easily be combined with the \icode{operator*}(also defined in subsection \iclheaderref{Quick}{Quick}) to create images that are initialized with any value. \displaycode{ImgQ c=ones(320,240,3)*3.14; // 3-channel QVGA image initial. to pi} In addition to this functions, a 3rd very common creator function named \icode{create} is provided. \icode{create} creates a test image which can be selected by one of the following string identifier: \icode{woman}, \icode{house}, \icode{tree}, \icode{parrot}, \icode{windows} and \icode{flowers}. \displaycode{ImgQ d=create("parrot"); // wonderful parrot image} Internally, image data is stored in simple byte arrays, that contains the binary data of JPEG-compressed images. Hence, \icode{create} can be used without any image file.\\ To read an image from hard disc, a simple \icode{load} function is also provided: \displaycode{ImgQ e=load("image.png"); // loads an image from file} \item \textbf{Converter Functions}\\ As mentioned before, ICLQuick functions work only on \icode{ImgQ} (a.k.a \icode{ImgQ}) images. To facilitate processing of images with another depth, the \icode{cvt} function can be used. \icode{cvt} is overloaded for all \icode{Img}-types and for all \icode{Img}-pointer-types and always returns a deeply converted(/copied) instance of the source image.\\ To convert instances of type \icode{ImgQ} into another depth, another set of converter function is provided: \icode{cvt8u}, \icode{cvt16s}, ..., \icode{cvt64f}. The following example illustrates use of \icode{cvt}-functions: \codefile{quick-cvt-example.cpp}{Example for the ICLQuick-converter-functions} \item \textbf{Filtering Functions}\\ Again, a definition of \emph{filters} must be found. Here, we have no special interface, except that all so called \emph{filtering functions} get an \icode{const ImgQ&} as first argument partly followed by further arguments) and return a new \icode{ImgQ} instance.\\ \begin{itemize} \item The \icode{filter}-function is a simple wrapper for some common \icode{UnaryOp}s (see subsection \ref{subsec:unary-ops}). \icode{filter} gets an additional string argument to specify the certain operation, e.g. \displaycode{ImgQ a = filter(create("parrot"),"laplace");} \item \icode{cc} overloads the \icode{cc}-function from the ICLCC package (see \iclheaderref{CC}{CC}) for \icode{ImgQ} instances. \item \icode{rgb}, \icode{gray}, \icode{hls} and \icode{lab} convert an image into a certain color format. \item \icode{scale} can be used to scale an image to a give size or by a given factor. \item \icode{channel} extracts a single channel from a given image. \item \icode{levels} can be used to reduces each image channels possible count of intensity levels. E.g. levels \icode{levels(myImage,2)} creates a binary image and \icode{levels(myImage,4)} removes all but the 2 most significant bits from an images value domain. \item \icode{thresh} also creates a binary image but here, the threshold can be given as well. \item \icode{copy} creates a \emph{deep} copy of a given image. \item \icode{copyroi} extracts an images ROI and returns it as a new image. \item \icode{norm} normalizes an images intensity range to range $[0,255]$ \item \icode{flipx} and \icode{flipy} flip images about either the x- or y-axis. \end{itemize} Due to the input- output nature of these functions, they can easily and intuitively be concatenated: \displaycode{Img8u i = cvt8u(gray(scale(create("parrot"),0.2)));} \item \textbf{Output Functions}\\ The output function set consists of two essential functions: \icode{save} and \icode{show}. The \icode{save} function works really straight forward. It gets an image argument and a file name argument and it saves the given image to a file of that name.\\ The \icode{show} function is much more complicated. But before trying to understand how it works, it has to be explained what is does! \\[10pt] \textbf{You can call the \icode{show}-function anywhere in your code to show a given image in a special window (which runs in a brand new process)}.\\[10pt] This is -- as you might have noticed -- a very powerful debugging tool. Internally, the \icode{show}-function writes the given image to hard-disc into a temporary file and runs and external program using C++'s \icode{system} function. The external program, which is called is \icode{icl-xv} on default. So if you want to use the \icode{show} function, you have to ensure, that at least \icode{icl-xv} can be found in your \icode{PATH}-variable. We use \icode{icl-xv} because, it is able to delete a given file after it has read it's image data. By this means no temporary files remains on your system. However, if another image viewer tool shall be used, you can call the \icode{showSetup} function\footnote{also a part of the \iclheaderref{Quick}{Quick}-header}, to set up another tool for this. If your favourite viewer is not able to delete visualized images (which is more than probable), you can also define a \icode{system}-delete-command (something like \icode{rm -rf \%s} where \icode{\%s}-token is automatically replace by the temporary image file name) you can define how long the \icode{show}-function shall wait until the temporary image is deleted. \item \textbf{Arithmetical Operators}\\ The ICLFilter package (see \ref{sec:filter}) provides a large set of operators. However, sometimes a more intuitive way to use operators is useful. E.g. if you want to determine the pixel-wise absolute difference between two images the ICLQuick-header allows you to simple write: \displaycode{ImgQ x = icl::abs(a-b); } As a matter of fact, this is even not significantly slower than using nested \icode{UnaryOP} instances (see subsection \ref{subsec:perf-issues-for-unary-and-binary-ops}).\\ Currently, the following arithmetical operators are available: \icode{+},\icode{-},\icode{*} and \icode{/} for \icode{ImgQ}-\icode{ImgQ}, for \icode{ImgQ}-\icode{float} and for \icode{float}-\icode{ImgQ} operands. Furthermore, the unary operator \icode{-} to invert an \icode{ImgQ} instances pixel values can be used. \item \textbf{Arithmetical Functions}\\ In addition to the \icode{abs}-function which was alredy used above, the folowing unary arithmetical function are declared in the \iclheaderref{Quick}{Quick}-header: \icode{exp}, \icode{sqrt}, \icode{ln} and \icode{sqr}\footnote{\icode{power of two}}. These functions work equal to filtering functions. \item \textbf{Logical Operators and Binary-Functions}\\ Like arithmetical operators, also logical operators can be used to write very code very intuitively. ICLQuick provides the logical operators \icode{||} and \icode{&&} which apply their specific operation pixel- and channel-wise. \\ Binary operators are not that easy to define in \icode{float}-valued images, hence binary operators were implemented as template-functions. The type parameter, which determines the internal type to perform binary operation on, must always be given explicitly. Possible binary functions are \icode{binOR}, \icode{binAND} and \icode{binXOR}. \item \textbf{Image Concatenation Operators}\\ These operators can be used to concatenate images horizontally, vertically or channel-wise.\\[10pt] \textbf{Please Note:} we use the operators \icode{|}, \icode{,} and \icode{\%} for \icode{ImgQ} operands here in a very special way, which is not conform to the C++ standard.\\[10pt] The \icode{,}-operator is used for horizontal image concatenation, e.g.: \displaycode{ImgQ a=create("flowers"),b=create("parrot");} \displaycode{ImgQ c = a,b;} If source and destination image heights or channel counts are different, the smaller one is padded with additional black pixels. Please note, that C++ evaluates the \icode{,}-operator always after function arguments are detected, so if you want to pass a \icode{,}-operator expression directly to another function, you have to use explicit braches: \displaycode{ImgQ c=filter(a,b,"gauss"); // Error: a,b are two arguments} \displaycode{ImgQ c=filter((a,b),"gauss"); // works!} For vertical concatenation, the \icode{\%}, which behaves like the \icode{,}-operator, is provided. Finally, the \icode{|}-operator can be used for channel-wise concatenation, e.g.: \displaycode{ImgQ channel1=..., channel2=..., channel3=...;} \displaycode{ImgQ image = c1|c2|c3; // 3 channel image} creates a 3 channel image. Please note that image concatenation between images $A$ and $B$ runs in $O(N_A+N_B)$ where $N_X$ is $WIDTH_X \times{}HEIGHT_X \times{} CHANNELS_X$. \item \textbf{Data/ROI Assignment Functions}\\ This set consists of two wrappers for the \icode{Img}'s \icode{deepCopyROI}-function please take a look at the following example: \codefile{quick-copy-roi-example.cpp}{Example for ICLQuick functions \icode{roi} and \icode{data}} \item \textbf{Drawing Functions}\\ This function set can be used to render graphical primitives into an image. Graphical primitives that can be drawn are rectangles, points, lines, triangles, circles and text.\\ \textbf{Please note} that in most online applications, this is \textbf{not} the recommended way to visualize the current system state (e.g. tracking results or class labels). The drawing functions should only be used for debugging purpose, as they are not optimized at all. The drawing function set works like a \textbf{single threaded} state machine. Current drawing parameters like draw-color or text size can be set using global functions as well. This tiny example explains how it works. \codefile{quick-drawing-example.cpp}{Example for ICLQuick's drawing functions} If you execute this application, the following image is shown: \includegraphics[width=200pt]{media/quick-house.png} Of course, there are more convenient ways to draw pictures, but in some cases, an image must be annotated somehow. Here ICLQuick's drawing functions will be very helpful. \item \textbf{tic/toc}\\ The function \icode{tic} and \icode{toc} can be used to measure a time interval. \icode{tic} starts a global timer and \icode{toc} stops it and prints the length of the time period to standard-out. \item \textbf{Region Detection/Analysis}\\ The Region detection functions are located in the header \iclheaderref{Quick}{QuickRegions}. They can be used for ad-hoc region analysis on \icode{ImgQ} instances. \end{itemize} \section{ICLGeom\label{sec:geom}} [Documentation: \iclpackageref{Geom}] The Geom package contains classes for 3D geometry representation. In many applications, e.g. object detection, it is not sufficient to detect an objects (x,y)-coordinates in an image, but also to determine it's 3D position in the scene. The fundamental class of this package is the \icode{Camera}-class (see \iclclassref{Geom}{Camera}). In contrast to a grabber, a camera is represented by extrinsic and intrinsic camera parameters \todo{add camera main axis offset sometimes}. We implemented ICL's camera model close the OpenGL camera model, so currently, a \icode{Camera} is represented by \begin{itemize} \item A position vector $\vec{p}$, which determines the camera's logical center in the world. for a pinhole camera, this would be the position of the pinhole. \item A norm vector $\vec{n}$, which determines the view direction of the camera (currently the normal vector intersects the virtual image plane always in it's center \todo{adapt this in the code first!}). \item An \emph{up}-vector $\vec{u}$ which defines the rotation of the image plane about the normal vector. \item A view port $v=(v_x,v_y,v_w,v_h)$ (where $v_x$ and $v_y$ are generally $0$). The viewport defines the transformation from logical image coordinates into pixel coordinates. \item The focal length $f$ of the camera (im mm). \item The near and far clipping plane $z_{far}$ and $z_{near}$ which are only used for OpenGL based visualization. \end{itemize} The camera projection model transforms a world point $p^W=(p^W_x,p^W_y,p^W_z,1)$ (we use homogeneous coordinates) into image coordinates $p^I=(p^I_x,p^I_y)$ by a concatenation of homogeneous transformations. The first step transforms $p^w$ into the camera frame (not the image coordinate system, which is 2D only) where we call it $p^C$: \begin{equation} p^C = C p^w \end{equation} where $C$ is the camera matrix which is created as follows \begin{equation} C = \left(\begin{array}{cc} R_C & d_C\\ 0 & 1\\ \end{array}\right) \end{equation} with \begin{equation} R_C = \left(\begin{array}{ccc} h_x & u_x & n_x \\ h_y & u_y & n_y \\ h_z & u_z & n_z \\ \end{array}\right) \end{equation} The horizontal vector $h$ can be computed by $h=u\times{}n$ and with $d_C=-R_C*p$.\\ The next transformation is the perspective projection, which can also be expressed by using a homogeneous matrix P: \begin{equation} P = \left(\begin{array}{cccc} f & 0 & p_x & 0\\ 0 & f & p_y & 0\\ 0 & 0 & A & B\\ 0 & 0 & -1 & 0\\ \end{array}\right) \end{equation} where $A=(z_{far}+z_{near})/(z_{near}-z_{far})$ and $B = (2 \cdot{}z_{far} \cdot{}z_{near})/(z_{near}-z_{far})$ are used for scaling z values into the range between the near and the far clipping plane\footnote{$p=(p_x,p_y) is the offset between the image center and the principle point of the camera. This is not yet used!$}. $P$ is used to transform points from the camera frame into a normalized image plane: \begin{equation} p^N = P p^C \end{equation} In the next step, points $p^N$ have to be transformed by the view-port matrix $V$ into image coordinates. \begin{equation} p^I = \mbox{hom}(V p^N) \end{equation} where $\mbox{hom(x,y,z,h)=(x/h,y/h,z/h,1)}$ and % float dx = (m_viewPort.left()+m_viewPort.right())/2; % float dy = (m_viewPort.top()+m_viewPort.bottom())/2; % float slope = iclMin(m_viewPort.width/2,m_viewPort.height/2); % return Mat ( slope , 0 , 0 , dx, % 0 , slope , 0 , dy, % 0 , 0 , 1 , 0 , % 0 , 0 , 0 , 1 ); \begin{equation} V = \left(\begin{array}{cccc} s_V & 0 & 0 & \Delta x_V\\ 0 & s_V & 0 & \Delta y_V\\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1\\ \end{array}\right) \end{equation} with $\Delta x_V= v_x+v_w/2$, $\Delta y_V= v_y+v_h/2$ and $s_V = \mbox{min}(v_w/2,v_h/2)$.\\ The resulting transformation is \begin{equation} p^I = \mbox{hom}(V P C p^W) \end{equation} Where only the first two components of $p^I$ are used. Once have defined the camera transformation model further functions like moving or rotating the camera, or creating view rays for image pixels can be implemented straight forward. Take a look at the \icode{Camera} reference for more details (see \iclclassref{Geom}{Camera}). \subsection{Camera Calibration Applications\label{subsec:icl-intrinsic-camera-calibration}} Currently, ICL provides two camera calibration tools: \begin{itemize} \item \icode{icl-intrinsic-camera-calibration}\\ This tool allows to compute lens-(un)distortion parameters automatically. Currently only a single radial distortion model is provided\todo{implement more general distortion model}. The model was taken from the ARToolkit library\footnote{See \hrefn{http://www.hitl.washington.edu/artoolkit/}}. Given a distortion parameter vector $\vec{\Theta}=(x_0,y_0,f,s)$ the undistortion of a distorted point $(x,y)_{dist}$ works as follows: \begin{equation*} (x,y)_{undist}^\tau = (p x + x_0, p y + y_0)^\tau \end{equation*} % where % \begin{equation*} (x,y)^\tau = s \cdot{}(x_{dist}-x_0,y_{dist}-y_0)^\tau \end{equation*} % and % \begin{equation*} p = 1-10^{-8}f(x*x+y*y) \end{equation*} % The point $(x_0,y_0)^\tau$ is the distortion center, $f$ is the radial distortion factor and $s$ is a scaling factor, which is used to removed black image edges in case of pincussion distortion. The current optimization procedure was also taken from the ARToolkit library which was published under GPL License as well. It searches best parameters using a hierarchical search approach. The undistortion error is computed from the straightness of the lines that connect the points on the calibration pattern. If the automatic calibration is not able to find appropriate parameters, the user is able to adjust them manually as well\footnote{This is e.g. necessary for the SwissRanger cameras, which need a very high distortion coefficient $f$}.\\ In contrast to the ARTookit calibration tool, \icode{icl-intrinsic-camera-calibration} is able to detect the calibration pattern autonomously. Furthermore, you can run this application with the program-argument \icode{-cp} and e.g. \icode{-nx 4 -ny 3} to make the program spawn an extra window, which shows a calibration pattern\footnote{in this case showing 12 round spots, 4 in x- and 3 in y-direction} that can either be used for print out or directly for calibration by pointing the camera to the monitor. The calibration result can be printed to standard out.\\ The resulting parameter vector can be passed to \icode{Grabber} instances as explained in \ref{subsubsec:distortion-compensation}. \iclfigure{intrinsic-calib-screenshot}{GUI-screenshot of icl-intrinsic-camera-calibration application. Here, a 2nd screen is used to visualize the 4x3 calibration pattern. A self-organizing-map (SOM) is used to fit a grid to the detection points.}{width=300pt} \subsubsection{GUI Parameters} Internally, the calibration pattern is detected as follows. After converting the input image into a gray-level image, a local threshold operation is applied (see \iclclassref{Filter}{LocalThresholdOp} for details). The values of the GUI elements \emph{mask size} and \emph{threshold} are passed to the underlying \icode{LocalThresholdOp}-instance. Subsequently, a dilation (see \iclclassref{Filter}{MorphologicalOp}) is applied to the thresholded image in order to suppress single black pixels. The \emph{use morphed image}-toggle button can be used to enable this step (by default, is not activated). The resulting binary image is then used as input for a connected component analysis (see \iclclassref{Blob}{RegionDetector}, which is parameterized by the sliders labelled \emph{roundness} and \emph{min blob size} in order to filter out all regions, except those that are small and round enough. \\ If the amount of matching regions meets the spot count from the calibration pattern\footnote{12 for the example above}, the display in the upper right GUI corner shows a green \emph{good}-string. Now, the user can press the \emph{add}-button to fit a grid into the detected points that is stored internally. The grid is fitted by using a SOM\footnote{see \hrefn{http://en.wikipedia.org/wiki/Self-organizing\_map}} which is implemented in the ICLAlgorithms package (see \iclclassref{Algorithms}{SOM}).\\ Within the calibration process, all straight lines, i.e horizontal, vertical and diagonal ones, in all currently stored grids are use for calculating the calibration error. \item \icode{icl-extrinsic-camera-calibration}\\ \end{itemize}