3rdparty/opencv-4.5.4/modules/gapi/doc/slides/gapi_overview.org

#+TITLE:     OpenCV 4.4 Graph API
#+AUTHOR:    Dmitry Matveev\newline Intel Corporation
#+OPTIONS: H:2 toc:t num:t
#+LATEX_CLASS: beamer
#+LATEX_CLASS_OPTIONS: [presentation]
#+LATEX_HEADER: \usepackage{transparent} \usepackage{listings} \usepackage{pgfplots} \usepackage{mtheme.sty/beamerthememetropolis}
#+LATEX_HEADER: \setbeamertemplate{frame footer}{OpenCV 4.4 G-API: Overview and programming by example}
#+BEAMER_HEADER: \subtitle{Overview and programming by example}
#+BEAMER_HEADER: \titlegraphic{ \vspace*{3cm}\hspace*{5cm} {\transparent{0.2}\includegraphics[height=\textheight]{ocv_logo.eps}}}
#+COLUMNS: %45ITEM %10BEAMER_ENV(Env) %10BEAMER_ACT(Act) %4BEAMER_COL(Col) %8BEAMER_OPT(Opt)
* G-API: What is, why, what's for?
** OpenCV evolution in one slide
*** Version 1.x -- Library inception
- Just a set of CV functions + helpers around (visualization, IO);
*** Version 2.x -- Library rewrite
- OpenCV meets C++, ~cv::Mat~ replaces ~IplImage*~;
*** Version 3.0 -- Welcome Transparent API (T-API)
- ~cv::UMat~ is introduced as a /transparent/ addition to
  ~cv::Mat~;
- With ~cv::UMat~, an OpenCL kernel can be enqeueud instead of
  immediately running C code;
- ~cv::UMat~ data is kept on a /device/ until explicitly queried.
** OpenCV evolution in one slide (cont'd)
# FIXME: Learn proper page-breaking!
*** Version 4.0 -- Welcome Graph API (G-API)
- A new separate module (not a full library rewrite);
- A framework (or even a /meta/-framework);
- Usage model:
  - /Express/ an image/vision processing graph and then /execute/ it;
  - Fine-tune execution without changes in the graph;
- Similar to Halide -- separates logic from
  platform details.
- More than Halide:
  - Kernels can be written in unconstrained platform-native code;
  - Halide can serve as a backend (one of many).
** OpenCV evolution in one slide (cont'd)
# FIXME: Learn proper page-breaking!
*** Version 4.2 -- New horizons
- Introduced in-graph inference via OpenVINO™ Toolkit;
- Introduced video-oriented Streaming execution mode;
- Extended focus from individual image processing to the full
  application pipeline optimization.
*** Version 4.4 -- More on video
- Introduced a notion of stateful kernels;
  - The road to object tracking, background subtraction, etc. in the
    graph;
- Added more video-oriented operations (feature detection, Optical
  flow).
** Why G-API?
*** Why introduce a new execution model?
- Ultimately it is all about optimizations;
  - or at least about a /possibility/ to optimize;
- A CV algorithm is usually not a single function call, but a
  composition of functions;
- Different models operate at different levels of knowledge on the
  algorithm (problem) we run.
** Why G-API? (cont'd)
# FIXME: Learn proper page-breaking!
*** Why introduce a new execution model?
- *Traditional* -- every function can be optimized (e.g. vectorized)
  and parallelized, the rest is up to programmer to care about.
- *Queue-based* -- kernels are enqueued dynamically with no guarantee
  where the end is or what is called next;
- *Graph-based* -- nearly all information is there, some compiler
  magic can be done!
** What is G-API for?
*** Bring the value of graph model with OpenCV where it makes sense:
- *Memory consumption* can be reduced dramatically;
- *Memory access* can be optimized to maximize cache reuse;
- *Parallelism* can be applied automatically where it is hard to do
  it manually;
  - It also becomes more efficient when working with graphs;
- *Heterogeneity* gets extra benefits like:
  - Avoiding unnecessary data transfers;
  - Shadowing transfer costs with parallel host co-execution;
  - Improving system throughput with frame-level pipelining.
* Programming with G-API
** G-API Basics
*** G-API Concepts
- *Graphs* are built by applying /operations/ to /data objects/;
  - API itself has no "graphs", it is expression-based instead;
- *Data objects* do not hold actual data, only capture /dependencies/;
- *Operations* consume and produce data objects.
- A graph is defined by specifying its /boundaries/ with data objects:
  - What data objects are /inputs/ to the graph?
  - What are its /outputs/?
** The code is worth a thousand words
   :PROPERTIES:
   :BEAMER_opt: shrink=42
   :END:
#+BEGIN_SRC C++
#include <opencv2/gapi.hpp>                            // G-API framework header
#include <opencv2/gapi/imgproc.hpp>                    // cv::gapi::blur()
#include <opencv2/highgui.hpp>                         // cv::imread/imwrite
int main(int argc, char *argv[]) {
    if (argc < 3) return 1;
    cv::GMat in;                                       // Express the graph:
    cv::GMat out = cv::gapi::blur(in, cv::Size(3,3));  // `out` is a result of `blur` of `in`
    cv::Mat in_mat = cv::imread(argv[1]);              // Get the real data
    cv::Mat out_mat;                                   // Output buffer (may be empty)
    cv::GComputation(cv::GIn(in), cv::GOut(out))       // Declare a graph from `in` to `out`
        .apply(cv::gin(in_mat), cv::gout(out_mat));    // ...and run it immediately
    cv::imwrite(argv[2], out_mat);                     // Save the result
    return 0;
}
#+END_SRC
** The code is worth a thousand words
   :PROPERTIES:
   :BEAMER_opt: shrink=42
   :END:
*** Traditional OpenCV                                        :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.45
    :END:
#+BEGIN_SRC C++
#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
int main(int argc, char *argv[]) {
    using namespace cv;
    if (argc != 3) return 1;
    Mat in_mat = imread(argv[1]);
    Mat gx, gy;
    Sobel(in_mat, gx, CV_32F, 1, 0);
    Sobel(in_mat, gy, CV_32F, 0, 1);
    Mat mag, out_mat;
    sqrt(gx.mul(gx) + gy.mul(gy), mag);
    mag.convertTo(out_mat, CV_8U);
    imwrite(argv[2], out_mat);
    return 0;
}
#+END_SRC
*** OpenCV G-API                                              :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.5
    :END:
#+BEGIN_SRC C++
#include <opencv2/gapi.hpp>
#include <opencv2/gapi/core.hpp>
#include <opencv2/gapi/imgproc.hpp>
#include <opencv2/highgui.hpp>
int main(int argc, char *argv[]) {
    using namespace cv;
    if (argc != 3) return 1;
    GMat in;
    GMat gx  = gapi::Sobel(in, CV_32F, 1, 0);
    GMat gy  = gapi::Sobel(in, CV_32F, 0, 1);
    GMat mag = gapi::sqrt(  gapi::mul(gx, gx)
                          + gapi::mul(gy, gy));
    GMat out = gapi::convertTo(mag, CV_8U);
    GComputation sobel(GIn(in), GOut(out));
    Mat in_mat = imread(argv[1]), out_mat;
    sobel.apply(in_mat, out_mat);
    imwrite(argv[2], out_mat);
    return 0;
}
#+END_SRC
** The code is worth a thousand words (cont'd)
# FIXME: sections!!!
*** What we have just learned?
- G-API functions mimic their traditional OpenCV ancestors;
- No real data is required to construct a graph;
- Graph construction and graph execution are separate steps.
*** What else?
- Graph is first /expressed/ and then /captured/ in an object;
- Graph constructor defines /protocol/; user can pass vectors of
  inputs/outputs like
  #+BEGIN_SRC C++
cv::GComputation(cv::GIn(...), cv::GOut(...))
  #+END_SRC
- Calls to ~.apply()~ must conform to graph's protocol
** On data objects
Graph *protocol* defines what arguments a computation was defined on
(both inputs and outputs), and what are the *shapes* (or types) of
those arguments:
  | *Shape*      | *Argument*       | Size                        |
  |--------------+------------------+-----------------------------|
  | ~GMat~       | ~Mat~            | Static; defined during      |
  |              |                  | graph compilation           |
  |--------------+------------------+-----------------------------|
  | ~GScalar~    | ~Scalar~         | 4 x ~double~                |
  |--------------+------------------+-----------------------------|
  | ~GArray<T>~  | ~std::vector<T>~ | Dynamic; defined in runtime |
  |--------------+------------------+-----------------------------|
  | ~GOpaque<T>~ | ~T~              | Static, ~sizeof(T)~         |
~GScalar~ may be value-initialized at construction time to allow
  expressions like ~GMat a = 2*(b + 1)~.
** On operations and kernels
    :PROPERTIES:
    :BEAMER_opt: shrink=22
    :END:
***                                                           :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.45
    :END:
- Graphs are built with *Operations* over virtual *Data*;
- *Operations* define interfaces (literally);
- *Kernels* are implementations to *Operations* (like in OOP);
- An *Operation* is platform-agnostic, a *kernel* is not;
- *Kernels* are implemented for *Backends*, the latter provide
  APIs to write kernels;
- Users can /add/ their *own* operations and kernels,
  and also /redefine/ "standard" kernels their *own* way.
***                                                          :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.45
    :END:
#+BEGIN_SRC dot :file "000-ops-kernels.eps" :cmdline "-Kdot -Teps"
digraph G {
node [shape=box];
rankdir=BT;
Gr [label="Graph"];
Op [label="Operation\nA"];
{rank=same
Impl1 [label="Kernel\nA:2"];
Impl2 [label="Kernel\nA:1"];
}
Op -> Gr [dir=back, label="'consists of'"];
Impl1 -> Op [];
Impl2 -> Op [label="'is implemented by'"];
node [shape=note,style=dashed];
{rank=same
Op;
CommentOp [label="Abstract:\ndeclared via\nG_API_OP()"];
}
{rank=same
Comment1 [label="Platform:\ndefined with\nOpenCL backend"];
Comment2 [label="Platform:\ndefined with\nOpenCV backend"];
}
CommentOp -> Op      [constraint=false, style=dashed, arrowhead=none];
Comment1  -> Impl1   [style=dashed, arrowhead=none];
Comment2  -> Impl2   [style=dashed, arrowhead=none];
}
#+END_SRC
** On operations and kernels (cont'd)
*** Defining an operation
- A type name (every operation is a C++ type);
- Operation signature (similar to ~std::function<>~);
- Operation identifier (a string);
- Metadata callback -- describe what is the output value format(s),
  given the input and arguments.
- Use ~OpType::on(...)~ to use a new kernel ~OpType~ to construct graphs.
#+LaTeX: {\footnotesize
#+BEGIN_SRC C++
G_API_OP(GSqrt,<GMat(GMat)>,"org.opencv.core.math.sqrt") {
    static GMatDesc outMeta(GMatDesc in) { return in; }
};
#+END_SRC
#+LaTeX: }
** On operations and kernels (cont'd)
*** ~GSqrt~ vs. ~cv::gapi::sqrt()~
- How a *type* relates to a *functions* from the example?
- These functions are just wrappers over ~::on~:
  #+LaTeX: {\scriptsize
  #+BEGIN_SRC C++
  G_API_OP(GSqrt,<GMat(GMat)>,"org.opencv.core.math.sqrt") {
      static GMatDesc outMeta(GMatDesc in) { return in; }
  };
  GMat gapi::sqrt(const GMat& src) { return GSqrt::on(src); }
  #+END_SRC
  #+LaTeX: }
- Why -- Doxygen, default parameters, 1:n mapping:
  #+LaTeX: {\scriptsize
  #+BEGIN_SRC C++
  cv::GMat custom::unsharpMask(const cv::GMat &src,
                               const int       sigma,
                               const float     strength) {
      cv::GMat blurred   = cv::gapi::medianBlur(src, sigma);
      cv::GMat laplacian = cv::gapi::Laplacian(blurred, CV_8U);
      return (src - (laplacian * strength));
  }
  #+END_SRC
  #+LaTeX: }
** On operations and kernels (cont'd)
*** Implementing an operation
- Depends on the backend and its API;
- Common part for all backends: refer to operation being implemented
  using its /type/.
*** OpenCV backend
- OpenCV backend is the default one: OpenCV kernel is a wrapped OpenCV
  function:
  #+LaTeX: {\footnotesize
  #+BEGIN_SRC C++
  GAPI_OCV_KERNEL(GCPUSqrt, cv::gapi::core::GSqrt) {
      static void run(const cv::Mat& in, cv::Mat &out) {
          cv::sqrt(in, out);
      }
  };
  #+END_SRC
  #+LaTeX: }
** Operations and Kernels (cont'd)
# FIXME!!!
*** Fluid backend
- Fluid backend operates with row-by-row kernels and schedules its
  execution to optimize data locality:
  #+LaTeX: {\footnotesize
  #+BEGIN_SRC C++
  GAPI_FLUID_KERNEL(GFluidSqrt, cv::gapi::core::GSqrt, false) {
      static const int Window = 1;
      static void run(const View &in, Buffer &out) {
          hal::sqrt32f(in .InLine <float>(0)
                       out.OutLine<float>(0),
                       out.length());
      }
  };
  #+END_SRC
  #+LaTeX: }
- Note ~run~ changes signature but still is derived from the operation
  signature.
** Operations and Kernels (cont'd)
*** Specifying which kernels to use
- Graph execution model is defined by kernels which are available/used;
- Kernels can be specified via the graph compilation arguments:
  #+LaTeX: {\footnotesize
  #+BEGIN_SRC C++
  #include <opencv2/gapi/fluid/core.hpp>
  #include <opencv2/gapi/fluid/imgproc.hpp>
  ...
  auto pkg = cv::gapi::combine(cv::gapi::core::fluid::kernels(),
                               cv::gapi::imgproc::fluid::kernels());
  sobel.apply(in_mat, out_mat, cv::compile_args(pkg));
  #+END_SRC
  #+LaTeX: }
- Users can combine kernels of different backends and G-API will partition
  the execution among those automatically.
** Heterogeneity in G-API
    :PROPERTIES:
    :BEAMER_opt: shrink=35
    :END:
*** Automatic subgraph partitioning in G-API
***                                                           :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.18
    :END:
#+BEGIN_SRC dot :file "010-hetero-init.eps" :cmdline "-Kdot -Teps"
digraph G {
rankdir=TB;
ranksep=0.3;
node [shape=box margin=0 height=0.25];
A; B; C;
node [shape=ellipse];
GMat0;
GMat1;
GMat2;
GMat3;
GMat0 -> A -> GMat1 -> B -> GMat2;
GMat2 -> C;
GMat0 -> C -> GMat3
subgraph cluster {style=invis; A; GMat1; B; GMat2; C};
}
#+END_SRC
The initial graph: operations are not resolved yet.
***                                                           :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.18
    :END:
#+BEGIN_SRC dot :file "011-hetero-homo.eps" :cmdline "-Kdot -Teps"
digraph G {
rankdir=TB;
ranksep=0.3;
node [shape=box margin=0 height=0.25];
A; B; C;
node [shape=ellipse];
GMat0;
GMat1;
GMat2;
GMat3;
GMat0 -> A -> GMat1 -> B -> GMat2;
GMat2 -> C;
GMat0 -> C -> GMat3
subgraph cluster {style=filled;color=azure2; A; GMat1; B; GMat2; C};
}
#+END_SRC
All operations are handled by the same backend.
***                                                           :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.18
    :END:
#+BEGIN_SRC dot :file "012-hetero-a.eps" :cmdline "-Kdot -Teps"
digraph G {
rankdir=TB;
ranksep=0.3;
node [shape=box margin=0 height=0.25];
A; B; C;
node [shape=ellipse];
GMat0;
GMat1;
GMat2;
GMat3;
GMat0 -> A -> GMat1 -> B -> GMat2;
GMat2 -> C;
GMat0 -> C -> GMat3
subgraph cluster_1 {style=filled;color=azure2; A; GMat1; B; }
subgraph cluster_2 {style=filled;color=ivory2; C};
}
#+END_SRC
~A~ & ~B~ are of backend ~1~, ~C~ is of backend ~2~.
***                                                           :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.18
    :END:
#+BEGIN_SRC dot :file "013-hetero-b.eps" :cmdline "-Kdot -Teps"
digraph G {
rankdir=TB;
ranksep=0.3;
node [shape=box margin=0 height=0.25];
A; B; C;
node [shape=ellipse];
GMat0;
GMat1;
GMat2;
GMat3;
GMat0 -> A -> GMat1 -> B -> GMat2;
GMat2 -> C;
GMat0 -> C -> GMat3
subgraph cluster_1 {style=filled;color=azure2; A};
subgraph cluster_2 {style=filled;color=ivory2; B};
subgraph cluster_3 {style=filled;color=azure2; C};
}
#+END_SRC
~A~ & ~C~ are of backend ~1~, ~B~ is of backend ~2~.
** Heterogeneity in G-API
*** Heterogeneity summary
- G-API automatically partitions its graph in subgraphs (called "islands")
  based on the available kernels;
- Adjacent kernels taken from the same backend are "fused" into the same
  "island";
- G-API implements a two-level execution model:
  - Islands are executed at the top level by a G-API's *Executor*;
  - Island internals are run at the bottom level by its *Backend*;
- G-API fully delegates the low-level execution and memory management to backends.
* Inference and Streaming
** Inference with G-API
*** In-graph inference example
- Starting with OpencV 4.2 (2019), G-API allows to integrate ~infer~
  operations into the graph:
  #+LaTeX: {\scriptsize
  #+BEGIN_SRC C++
  G_API_NET(ObjDetect, <cv::GMat(cv::GMat)>, "pdf.example.od");
  cv::GMat in;
  cv::GMat blob = cv::gapi::infer<ObjDetect>(bgr);
  cv::GOpaque<cv::Size> size = cv::gapi::streaming::size(bgr);
  cv::GArray<cv::Rect>  objs = cv::gapi::streaming::parseSSD(blob, size);
  cv::GComputation pipelne(cv::GIn(in), cv::GOut(objs));
  #+END_SRC
  #+LaTeX: }
- Starting with OpenCV 4.5 (2020), G-API will provide more streaming-
  and NN-oriented operations out of the box.
** Inference with G-API
*** What is the difference?
- ~ObjDetect~ is not an operation, ~cv::gapi::infer<T>~ is;
- ~cv::gapi::infer<T>~ is a *generic* operation, where ~T=ObjDetect~ describes
  the calling convention:
  - How many inputs the network consumes,
  - How many outputs the network produces.
- Inference data types are ~GMat~ only:
  - Representing an image, then preprocessed automatically;
  - Representing a blob (n-dimensional ~Mat~), then passed as-is.
- Inference *backends* only need to implement a single generic operation ~infer~.
** Inference with G-API
*** But how does it run?
- Since ~infer~ is an *Operation*, backends may provide *Kernels* implenting it;
- The only publicly available inference backend now is *OpenVINO™*:
  - Brings its ~infer~ kernel atop of the Inference Engine;
- NN model data is passed through G-API compile arguments (like kernels);
- Every NN backend provides its own structure to configure the network (like
  a kernel API).
** Inference with G-API
*** Passing OpenVINO™ parameters to G-API
- ~ObjDetect~ example:
  #+LaTeX: {\footnotesize
  #+BEGIN_SRC C++
  auto face_net = cv::gapi::ie::Params<ObjDetect> {
      face_xml_path,        // path to the topology IR
      face_bin_path,        // path to the topology weights
      face_device_string,   // OpenVINO plugin (device) string
  };
  auto networks = cv::gapi::networks(face_net);
  pipeline.compile(.., cv::compile_args(..., networks));
  #+END_SRC
  #+LaTeX: }
- ~AgeGender~ requires binding Op's outputs to NN layers:
  #+LaTeX: {\footnotesize
  #+BEGIN_SRC C++
  auto age_net = cv::gapi::ie::Params<AgeGender> {
      ...
  }.cfgOutputLayers({"age_conv3", "prob"}); // array<string,2> !
  #+END_SRC
  #+LaTeX: }
** Streaming with G-API
#+BEGIN_SRC dot :file 020-fd-demo.eps :cmdline "-Kdot -Teps"
digraph {
  rankdir=LR;
  node [shape=box];
  cap [label=Capture];
  dec [label=Decode];
  res [label=Resize];
  cnn [label=Infer];
  vis [label=Visualize];
  cap -> dec;
  dec -> res;
  res -> cnn;
  cnn -> vis;
}
#+END_SRC
Anatomy of a regular video analytics application
** Streaming with G-API
#+BEGIN_SRC dot :file 021-fd-serial.eps :cmdline "-Kdot -Teps"
digraph {
  node [shape=box margin=0 width=0.3 height=0.4]
  nodesep=0.2;
  rankdir=LR;
  subgraph cluster0 {
  colorscheme=blues9
  pp [label="..." shape=plaintext];
  v0 [label=V];
  label="Frame N-1";
  color=7;
  }
  subgraph cluster1 {
  colorscheme=blues9
  c1 [label=C];
  d1 [label=D];
  r1 [label=R];
  i1 [label=I];
  v1 [label=V];
  label="Frame N";
  color=6;
  }
  subgraph cluster2 {
  colorscheme=blues9
  c2 [label=C];
  nn [label="..." shape=plaintext];
  label="Frame N+1";
  color=5;
  }
  c1 -> d1 -> r1 -> i1 -> v1;
  pp-> v0;
  v0 -> c1 [style=invis];
  v1 -> c2 [style=invis];
  c2 -> nn;
}
#+END_SRC
Serial execution of the sample video analytics application
** Streaming with G-API
    :PROPERTIES:
    :BEAMER_opt: shrink
    :END:
#+BEGIN_SRC dot :file 022-fd-pipelined.eps :cmdline "-Kdot -Teps"
digraph {
  nodesep=0.2;
  ranksep=0.2;
  node [margin=0 width=0.4 height=0.2];
  node [shape=plaintext]
  Camera [label="Camera:"];
  GPU [label="GPU:"];
  FPGA [label="FPGA:"];
  CPU [label="CPU:"];
  Time [label="Time:"];
  t6  [label="T6"];
  t7  [label="T7"];
  t8  [label="T8"];
  t9  [label="T9"];
  t10 [label="T10"];
  tnn [label="..."];
  node [shape=box margin=0 width=0.4 height=0.4 colorscheme=blues9]
  node [color=9] V3;
  node [color=8] F4; V4;
  node [color=7] DR5; F5; V5;
  node [color=6] C6; DR6; F6; V6;
  node [color=5] C7; DR7; F7; V7;
  node [color=4] C8; DR8; F8;
  node [color=3] C9; DR9;
  node [color=2] C10;
  {rank=same; rankdir=LR; Camera C6 C7 C8 C9 C10}
  Camera -> C6 -> C7 -> C8 -> C9 -> C10 [style=invis];
  {rank=same; rankdir=LR; GPU DR5 DR6 DR7 DR8 DR9}
  GPU -> DR5 -> DR6 -> DR7 -> DR8 -> DR9 [style=invis];
  C6 -> DR5 [style=invis];
  C6 -> DR6 [constraint=false];
  C7 -> DR7 [constraint=false];
  C8 -> DR8 [constraint=false];
  C9 -> DR9 [constraint=false];
  {rank=same; rankdir=LR; FPGA F4 F5 F6 F7 F8}
  FPGA -> F4 -> F5 -> F6 -> F7 -> F8 [style=invis];
  DR5 -> F4 [style=invis];
  DR5 -> F5 [constraint=false];
  DR6 -> F6 [constraint=false];
  DR7 -> F7 [constraint=false];
  DR8 -> F8 [constraint=false];
  {rank=same; rankdir=LR; CPU V3 V4 V5 V6 V7}
  CPU -> V3 -> V4 -> V5 -> V6 -> V7 [style=invis];
  F4 -> V3 [style=invis];
  F4 -> V4 [constraint=false];
  F5 -> V5 [constraint=false];
  F6 -> V6 [constraint=false];
  F7 -> V7 [constraint=false];
  {rank=same; rankdir=LR; Time t6 t7 t8 t9 t10 tnn}
  Time -> t6 -> t7 -> t8 -> t9 -> t10 -> tnn [style=invis];
  CPU -> Time [style=invis];
  V3 -> t6  [style=invis];
  V4 -> t7  [style=invis];
  V5 -> t8  [style=invis];
  V6 -> t9  [style=invis];
  V7 -> t10 [style=invis];
}
#+END_SRC
Pipelined execution for the video analytics application
** Streaming with G-API: Example
**** Serial mode (4.0)                                        :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.45
    :END:
#+LaTeX: {\tiny
#+BEGIN_SRC C++
pipeline = cv::GComputation(...);
cv::VideoCapture cap(input);
cv::Mat in_frame;
std::vector<cv::Rect> out_faces;
while (cap.read(in_frame)) {
    pipeline.apply(cv::gin(in_frame),
                   cv::gout(out_faces),
                   cv::compile_args(kernels,
                                    networks));
    // Process results
    ...
}
#+END_SRC
#+LaTeX: }
**** Streaming mode (since 4.2)                               :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.45
    :END:
#+LaTeX: {\tiny
#+BEGIN_SRC C++
pipeline = cv::GComputation(...);
auto in_src = cv::gapi::wip::make_src
    <cv::gapi::wip::GCaptureSource>(input)
auto cc = pipeline.compileStreaming
    (cv::compile_args(kernels, networks))
cc.setSource(cv::gin(in_src));
cc.start();
std::vector<cv::Rect> out_faces;
while (cc.pull(cv::gout(out_faces))) {
    // Process results
    ...
}
#+END_SRC
#+LaTeX: }
**** More information
#+LaTeX: {\footnotesize
https://opencv.org/hybrid-cv-dl-pipelines-with-opencv-4-4-g-api/
#+LaTeX: }
* Latest features
** Latest features
*** Python API
- Initial Python3 binding is available now in ~master~ (future 4.5);
- Only basic CV functionality is supported (~core~ & ~imgproc~ namespaces,
  selecting backends);
- Adding more programmability, inference, and streaming is next.
** Latest features
*** Python API
#+LaTeX: {\footnotesize
#+BEGIN_SRC Python
import numpy as np
import cv2 as cv
sz  = (1280, 720)
in1 = np.random.randint(0, 100, sz).astype(np.uint8)
in2 = np.random.randint(0, 100, sz).astype(np.uint8)
g_in1 = cv.GMat()
g_in2 = cv.GMat()
g_out = cv.gapi.add(g_in1, g_in2)
gr    = cv.GComputation(g_in1, g_in2, g_out)
pkg   = cv.gapi.core.fluid.kernels()
out   = gr.apply(in1, in2, args=cv.compile_args(pkg))
#+END_SRC
#+LaTeX: }
* Understanding the "G-Effect"
** Understanding the "G-Effect"
*** What is "G-Effect"?
- G-API is not only an API, but also an /implementation/;
  - i.e. it does some work already!
- We call "G-Effect" any measurable improvement which G-API demonstrates
  against traditional methods;
- So far the list is:
  - Memory consumption;
  - Performance;
  - Programmer efforts.
Note: in the following slides, all measurements are taken on
Intel\textregistered{} Core\texttrademark-i5 6600 CPU.
** Understanding the "G-Effect"
# FIXME
*** Memory consumption: Sobel Edge Detector
- G-API/Fluid backend is designed to minimize footprint:
#+LaTeX: {\footnotesize
| Input       | OpenCV | G-API/Fluid | Factor |
|             |    MiB |         MiB | Times  |
|-------------+--------+-------------+--------|
| 512 x 512   |  17.33 |        0.59 |  28.9x |
| 640 x 480   |  20.29 |        0.62 |  32.8x |
| 1280 x 720  |  60.73 |        0.72 |  83.9x |
| 1920 x 1080 | 136.53 |        0.83 | 164.7x |
| 3840 x 2160 | 545.88 |        1.22 | 447.4x |
#+LaTeX: }
- The detector itself can be written manually in two ~for~
  loops, but G-API covers cases more complex than that;
- OpenCV code requires changes to shrink footprint.
** Understanding the "G-Effect"
*** Performance: Sobel Edge Detector
- G-API/Fluid backend also optimizes cache reuse:
#+LaTeX: {\footnotesize
| Input       | OpenCV | G-API/Fluid | Factor |
|             |     ms |          ms |  Times |
|-------------+--------+-------------+--------|
| 320 x 240   |   1.16 |        0.53 |  2.17x |
| 640 x 480   |   5.66 |        1.89 |  2.99x |
| 1280 x 720  |  17.24 |        5.26 |  3.28x |
| 1920 x 1080 |  39.04 |       12.29 |  3.18x |
| 3840 x 2160 | 219.57 |       51.22 |  4.29x |
#+LaTeX: }
- The more data is processed, the bigger "G-Effect" is.
** Understanding the "G-Effect"
*** Relative speed-up based on cache efficiency
#+BEGIN_LATEX
\begin{figure}
  \begin{tikzpicture}
    \begin{axis}[
      xlabel={Image size},
      ylabel={Relative speed-up},
      nodes near coords,
      width=0.8\textwidth,
      xtick=data,
      xticklabels={QVGA, VGA, HD, FHD, UHD},
      height=4.5cm,
    ]
    \addplot plot coordinates {(1, 1.0) (2, 1.38) (3, 1.51) (4, 1.46) (5, 1.97)};
    \end{axis}
  \end{tikzpicture}
\end{figure}
#+END_LATEX
The higher resolution is, the higher relative speed-up is (with
speed-up on QVGA taken as 1.0).
* Resources on G-API
** Resources on G-API
   :PROPERTIES:
   :BEAMER_opt: shrink
   :END:
*** Repository
- https://github.com/opencv/opencv (see ~modules/gapi~)
*** Article
- https://opencv.org/hybrid-cv-dl-pipelines-with-opencv-4-4-g-api/
*** Documentation
- https://docs.opencv.org/4.4.0/d0/d1e/gapi.html
*** Tutorials
- https://docs.opencv.org/4.4.0/df/d7e/tutorial_table_of_content_gapi.html
* Thank you!