Deformable DETR

Helpful YouTube Video

The main benefits of Deformable DETR are:

Reduces the memory and computational cost of DETR through deformable attention where the attention modules only attend to a small set of key sampling points around a reference. It can achieve better performance than DETR (especially on small objects) with 10x less epochs. Additionally, it can have higher resolution features since you don’t need to attend to the full input.

Deformable Convolution is able to attend to sparse spatial locations with a convolution, but it lacks the element relation modeling of attention (all elements can relate to all other elements). This paper introduces Deformable Attention to attend to a small set of sampling locations as a pre-filter for prominent key elements out of all the feature pixel maps. They are able to use multi-scale features without requiring an FPN.

Related Work §

There are three main approaches to improving the complexity of attention:

1. Restrict the attention pattern to be a fixed local window. This results in decreased complexity, but it loses global information.
2. Learn data-dependent sparse attention. Deformable DETR uses this approach.
3. Use the low-rank property in self-attention to reduce the complexity (use linear algebra to reduce the matrix multiplies to the most significant elements).

Deformable Attention §

Deformable DETR Version of Deformable Attention §

Deformable DETR was inspired by Deformable Convolution and modifies the attention module to learn to focus on a small fixed set of sampling points predicted from the features of query elements. It is just slightly slower than traditional convolution under the same FLOPs. It is slower because you are accessing memory in a random order (vs. a conv layer which always accesses memory in the same order) so memory optimization and caching doesn’t work great.

The above diagram shows a single scale feature map $x$ ($l=1$) and 3 attention heads in green, blue, and yellow ($M=3$). You use $K=3$ sampling points.

Notation:

• $N_q$ is the number of query features ($z_q$) you have. For the encoder, this is $H\times W$ (you pass your image through a backbone that gives a downsampled feature map. You then flatten this feature map into a sequence of length $HW$. See DETR). For the decoder, this is the number of objects you want to detect ($N$).
• $K$ is the number of sampling points for each query feature to attend to the feature map. It should be much less than grid_height * grid_width.
• $z_q\in {\mathbb{R}}^C$ is the feature vector of query element $q$. For the encoder, this could be one pixel of the input feature map $x$. For the decoder, this could be an object query.
• ${\hat{p}}_q\in [0,1{]}^2$ is the normalized coordinates of the reference point for each query element $q$. These are normalized from the reference point $p_q$. For the encoder, the reference point can be the pixel of the input feature map. For the decoder, this can be predicted from its object query embedding via a linear projection + sigmoid.
• $x^l\in {\mathbb{R}}^{C\times H_l\times W_l},l=1,\dots ,L$ are the input feature maps extracted by a CNN backbone at multiple scales. In the above diagram only a single feature map is used.

Link to original

Bounding Box Prediction §

You start with a reference point that is used as the initial guess of the box center. The reference point is predicted as a 2D normalized coordinate (${\hat{p}}_q$) using a linear project + sigmoid that takes an object query as input.

The reference point is used as the initial guess as a box center and then a detection head predicts the relative offsets w.r.t. the reference points: ${\hat{\mathbf{b}}}_q=\left\{\sigma \left(b_{qx}+{\sigma }^{-1}\left({\hat{p}}_{qx}\right)\right),\sigma \left(b_{qy}+{\sigma }^{-1}\left({\hat{p}}_{qy}\right)\right),\sigma \left(b_{qw}\right),\sigma \left(b_{qh}\right)\right\}$

where:

• $b_{q\{x,y,w,h\}}\in \mathbb{R}$
• $\sigma$ and ${\sigma }^{-1}$ are the sigmoid and inverse sigmoid functions
• ${\hat{b}}_q$ will be normalized coordinates of the form ${\hat{\mathbf{b}}}_q\in [0,1{]}^4$.

Iterative Bounding Box Refinement §

Each decoder layer refines the bounding boxes based on the predictions from the previous layers. Suppose there are $D$ decoder layers (e.g. $D=6$), then given a normalized bounding box ${\hat{\mathbf{b}}}_q^{d-1}$ predicted by the ($d-1$)-th decoder layer, the $d$-th decoder layer will refine the box as: ${\hat{\mathbf{b}}}_q^d=\left\{\sigma \left(\Delta b_{qx}^d+{\sigma }^{-1}\left({\hat{b}}_{qx}^{d-1}\right)\right),\sigma \left(\Delta b_{qy}^d+{\sigma }^{-1}\left({\hat{b}}_{qy}^{d-1}\right)\right),\sigma \left(\Delta b_{qw}^d+{\sigma }^{-1}\left({\hat{b}}_{qw}^{d-1}\right)\right),\sigma \left(\Delta b_{qh}^d+{\sigma }^{-1}\left({\hat{b}}_{qh}^{d-1}\right)\right)\right\}$

The initial box uses the reference point (${\hat{p}}_q$) as the center with width = 0.1 and height = 0.1. Using the above notation this is: ${\hat{b}}_{qx}^0={\hat{p}}_{qx},{\hat{b}}_{qy}^0={\hat{p}}_{qy},{\hat{b}}_{qw}^0=0.1\text{, and }{\hat{b}}_{qh}^0=0.1$

To stabilize training, the gradients only back propagate through $\Delta b_{q\{x,y,w,h\}}^d$ and are blocked at ${\sigma }^{-1}\left({\hat{b}}_{q\{x,y,w,h\}}^{d-1}\right)$.

Two-Stage Deformable DETR §

In the original DETR, object queries in the decoder are irrelevant to the current image (they are learned during training and do not change depending on the inference image). Inspired by two-stage object detectors, Deformable DETR explores a variant of Deformable DETR for generating region proposals as the first stage. The generated region proposals will be fed into the decoder as object queries for further refinement, forming a two-stage Deformable DETR.

In the first stage, given the output feature maps of the encoder, a detection head is applied to each pixel. The detection head consists of:

• A 3-layer FFN for bounding box regression.
• A linear projection for bounding box binary classification (this is similar to an objectness score - 0 means background and 1 means foreground).

Given the predicted bounding boxes from the first stage, the top scoring bounding boxes are picked as region proposals. In the second stage, these proposals are fed into the decoder as the initial boxes for iterative bounding box refinement.

Loss §

Focal loss with loss weight of 2 is used for bounding box classification.

Notation §