How to choose threshold for ORB features without template matching

[phirestalker]

I see that I can choose the number of features that constitute a minimum match for the template matcher. I have two problems with this. The first concern is that the template matcher (assumedly) is not going to catch crops, or close-ups. I would assume that it would not match photos where one is just the subject, and the other has some boxes or text splattered over it. Secondly, the template matching process is sooo slow on my hardware.

I also see that there is a distance setting, but as far as I understand it, that will only determine how close the found features are, and not how many have to match. Is there a way to set this currently, or is there another way to match images with the differences that I mentioned above?

From my limited searching and writing a small video duplicate finder with OpenCV years ago (think OpenCV 2), I believe that the matching is done with a "database" that is indexed into buckets (or maybe multiple layer). This is a complete over simplification, but maybe close? This can be searched much quicker than looping through and manually comparing each feature one by one. Is there anything that could be sped up with a GPU, or is this a CPU bound problem?

[scrubbbbs]

The first concern is that the template matcher (assumedly) is not going to catch crops, or close-ups

It is quite good at catching crops. More features are needed to catch higher levels of cropping. No way around that.

I would assume that it would not match photos where one is just the subject, and the other has some boxes or text splattered over it. Secondly, the template matching process is sooo slow on my hardware.

It will catch modified crops provided it isn't too much and you also disable the dct matching part (see below). As for speed you might just use the orb algo , set it to maximum of 1 match, and play with the distance parameter and score (-with score '<num' etc. Since that is almost the same thing as what template matcher is doing. The template matcher is like a more restricted/higher-res version of orb search...

I also see that there is a distance setting, but as far as I understand it, that will only determine how close the found features are, and not how many have to match.

There is no setting for how many have to match. Only a few have to match for it to work. It works like this

• (1) first we find features between A and B. The -p.odt determines the sensitivity here, there is no # of matches requirement
• (2) then we see if the found features can solve an affine transform from A to B -- to solve the system needs only a small number of features (maybe 3-4) IIRC. Which explains why I didn't make it a setting.
• (3) then we apply that transform over A and the inverse over B, to get the common patch or "area of interest" between the two images.
• (4) then the dct hash is applied to test the patch -- this can be disabled by setting -p.tdht to something high. This is the part that will reject patches that have a lot of modifications.

From my limited searching and writing a small video duplicate finder with OpenCV years ago (think OpenCV 2), I believe that the matching is done with a "database" that is indexed into buckets (or maybe multiple layer).

The feature matching is using cv::BFMatcher -- "brute force" -- which looks at all features. There is a locale-sensitive hashtable (FLANN) which could be used with ORB features, that is probably what you are recalling. It could be faster, but maybe not, as the hashtable would have to be built each time for a small number of features (<1000 by default).

I would like to port cbird to use OpenCV 4 (current using 2.4), maybe this will be one area to revisit.

[phirestalker]

As for speed you might just use the orb algo , set it to maximum of 1 match, and play with the distance parameter and score (-with score '<num' etc.

Is there any info on how the scoring works. For instance, whether larger or smaller numbers will make closer matches?

• (1) first we find features between A and B. The -p.odt determines the sensitivity here, there is no # of matches requirement
• (2) then we see if the found features can solve an affine transform from A to B -- to solve the system needs only a small number of features (maybe 3-4) IIRC. Which explains why I didn't make it a setting.
• (3) then we apply that transform over A and the inverse over B, to get the common patch or "area of interest" between the two images.
• (4) then the dct hash is applied to test the patch -- this can be disabled by setting -p.tdht to something high. This is the part that will reject patches that have a lot of modifications.

I am exclusively using ORB features for similar scans now (-p.alg orb). Is the first quoted advice the only means of fine-tuning the matching? Are these steps only for the DCT algo?

The feature matching is using cv::BFMatcher -- "brute force" -- which looks at all features. There is a locale-sensitive hashtable (FLANN) which could be used with ORB features, that is probably what you are recalling. It could be faster, but maybe not, as the hashtable would have to be built each time for a small number of features (<1000 by default).

I am going to have to get into studying this stuff. Also, I will be glad to test anything you want if you decide to try out other matching or indexing methods.

[scrubbbbs]

Is there any info on how the scoring works. For instance, whether larger or smaller numbers will make closer matches?

The scoring is different for each algo; lower score is better. I can't remember off the top of my head for orb, but a quick look at the code later, it seems to use the median distance for all matching features (all distances being less than -p.odt), then divides by the number of matching features. It also multiplies by 1000 to keep the score from being < 1.

So having more features (-i.nfeat) and a lower value for -p.odt will give lower scores to matches. And the relation is reciprocal so it should get better at weeding out poor matches as more features are added. As for the threshold, it just needs to be high enough, but not too high as that would boost poor matches.

I am exclusively using ORB features for similar scans now (-p.alg orb). Is the first quoted advice the only means of fine-tuning the matching? Are these steps only for the DCT algo?

My last message was concerned with template matching. The orb algo is a separate thing although both do use ORB features. I think you are asking about the orb algo??

If so, the tunables are:

• -p.odt -- ORB distance threshold
• -p.nfeat -- number of features per image. There was a user that lowered this to 100 (default 400), but I would think it has to be increased to detect higher levels of cropping.
• -p.rsize -- sets the image size (downscaling) prior to feature detection. If this is too low, the desired number of features won't be found.

Also, I will be glad to test anything you want if you decide to try out other matching or indexing methods.

Thanks, I appreciate that. I'm getting close to another binary release. Are you on Linux/Windows?

Since OpenCV 4 is on my list for 0.8, that will possibly enable more work in this area or performance improvements. There is a new feature descriptor called KAZE which might be better.

[phirestalker]

If so, the tunables are:

• -p.odt -- ORB distance threshold
• -p.nfeat -- number of features per image. There was a user that lowered this to 100 (default 400), but I would think it has to be increased to detect higher levels of cropping.
• -p.rsize -- sets the image size (downscaling) prior to feature detection. If this is too low, the desired number of features won't be found.

sounds good. I think I can figure this out.

Since OpenCV 4 is on my list for 0.8, that will possibly enable more work in this area or performance improvements. There is a new feature descriptor called KAZE which might be better.

Also, I forgot to mention earlier that when I was using the ORB features, I also used BRIEF to condense them to 64 bits, and it supposedly sped up lookups or comparisons. It also made it take up less space. Maybe this new KAZE will have something like that.

Oh, related question. If I am using orb, do I even need to enable the vertical and horizontal checks for flipped images? I know that ORB is very good at rotations, but I wasn't sure about this.

[scrubbbbs]

Oh, related question. If I am using orb, do I even need to enable the vertical and horizontal checks for flipped images? I know that ORB is very good at rotations, but I wasn't sure about this.

You do, the descriptor doesn't account for mirror/flip.