ANPR accuracy test

12 Jul 2019

We recently updated our OCR platform with a newer version of TensorFlow and made improvements to the way images are pre-processed. The results are pretty good for the first cut, but there is definitely room for improvement.

The ANPR process

  1. Wake up the camera after a threshold speed reading from the radar
  2. Start processing the video stream at 3fps
  3. Discard any frames with no moving vehicle in the specified area / direction
  4. Start recording the video stream once a vehicle is detected
  5. Try to detect the number plate
  6. Try to read the text on the number plate once it was detected
  7. Stop recording after a pre-configured time interval or when the vehicle is gone from the view
  8. Send the event details to the cloud DB

The video and other media are uploaded on demand to save the cellular bandwidth.

The actual OCR step takes only 40ms, so in theory the camera can process more than 3 frames per second without slowing everything else down.

Variables and the success rate

The success of the number plate detection and the OCR very much depend on the lighting conditions. The normal daytime allows for at least 1/500 – 1/125 shutter speed at ISO64, which is enough to catch a decent view of a number plate in a suburban area. 1/350 is probably the lower limit for an 80kmh zone.

The slower the shutter speed, the more smudged the images appear. The sample below shows the beginning of twilight hours. You can see how FFP511 (40kmh, misread) is harder to read than FDE573 (20kmh).

Slower shutter speed results in smudged images, but they are still readable

The closer the camera can get to the plate the higher the maximum readable vehicle speed is. The plates in the sample above were read from 5-10m distance in twilight. Many of the cars already had their headlights on.

25 minutes later, at the very end of the twilight the success rate was dropping considerably.

There were no successful OCR reads after 17:23. Most of the cars had their headlights on and the street lights started turning on as well.

Compare that to crisp daylight images below giving us 91% OCR accuracy. The images above and below this paragraph appear equally bright, but the difference is in the shutter speed. The one below had a lot more light available, so the shutter speed was faster and the number plates appear crisper and brighter.

Shortfalls

There are clearly a few tricky number plate reading areas the cam struggles with:

  • plates with spaces
  • plates shorter than 5 characters
  • plates with all letters or unusual combinations

None of these 6 plates were read by the cam even though 4 of them should be in the “easy” category. This is a deficiency of the training data set and will improve as we get more of similar plates in and re-train the model.

Thresholds

The confidence thresholds are configurable. There is a trade-off between the system only returning results that it is fairly confident are correct and returning a result for every input it is given.

For the results in this article, the confidence threshold was set to 40% and compared to the overall confidence for a given prediction, which was calculated by multiplying the confidence values for each of its characters.

Reducing the confidence levels will increase the number of successful reads, but will also increase the number of incorrect ones.

Images 1 and 2 in the example below were discarded and the 3rd one was read correctly as FFL162. It is amazing how a neural net manages to read such dim and low contrast images while discarding some seemingly sharp ones.

OCR accuracy stats

The best hourly ANPR accuracy on the test day was 91% with overall accuracy at 88%.

The best hour

  • Time: 09:00 – 10:00
  • Vehicles: 39
  • No plate detected: 7%

From the detected plates:

  • Unverifiable: 0
  • Incorrect OCR: 2.78%
  • No OCR: 5.56%
  • Correct: 91.67%

Overall

  • Time: 09:00 – 17:00
  • Vehicles: 450
  • No plate detected: 3.56%

From the detected plates:

  • Unverifiable: 1.61%
  • Incorrect OCR: 5.53%
  • No OCR: 5.99%
  • Correct: 86.87%

Where to next

As mentioned before, this is one of the initial tests in a real life deployment on a moderately busy suburban road with a 50kmh speed limit. We identified a few gaps in the training dataset that need to be filled and run through another round of training.

There are also ways we can adjust the confidence level calculation to get more plates read. Getting the exposure right for optimal plate presentation will help with the dark plates.

Altogether, these measures will add a few percentage points each.

Overall, this is not a bad result for a 2W battery-powered ANPR device.