Skip to content

Machine-readable zone/travel document (MRZ / MRTD) detector and recognizer using deep learning

License

Notifications You must be signed in to change notification settings

DoubangoTelecom/ultimateMRZ-SDK

Repository files navigation


This is state-of-the-art Machine Readable Zone / Travel Documents (MRZ / MRTD) dectector and recognizer using deep learning.

Unlike other solutions you can find on the web, you don't need to adjust the camera/image to define a Region Of Interest (ROI). We also don't try to use small ROI to decrease the processing time or false-positives. The whole image (up to 4K supported) is processed and every pixel is checked. No matter if the MRZ lines are small, far away, blurred, partially occluded, skewed or slanted, our implementation can accurately detect and recognize every character.

The detector is agnostic and doesn't decode (recognize/OCR) the text to check it against some pre-defined rules (regular expressions) which means we support all MRZ types (Travel Documents 1/2/3, MRVA, MRVB...) regardless the font, content, shape or country. You can use our implementation to extract information from local identity cards, passports, enhanced driver licenses, visas, resident cards...

In addition to being highly accurate our implementation is very fast and can run at up to 50fps on ARM devices (iPhone7) and 114fps on x86_64 (Core i7-4790K) using HD images (720p). See benchmark section for more information.

Next video (https://youtu.be/AO5XdbLK9Do) shows VideoRecognizer sample Running on Android:
VideoRecognizer Running on Android


The Github repository contains the SDK binaries for Android, iOS, Raspberry Pi and Windows. You can contact us if you want binaries for other platforms.

Don't take our word for it, come check our implementation. No registration, license key or internet connection is needed, just clone the code and start coding/testing. Everything runs on the device, no data is leaving your computer. The code released on Github comes with many ready-to-use samples to help you get started easily.

You can also check our online cloud-based implementation (no registration required) at https://www.doubango.org/webapps/mrz/ to check out the accuracy and precision before starting to play with the SDK.

Sample applications

The source code comes with #4 sample applications: Benchmark, VideoRecognizer, Parser and Validation. All sample applications are open source and don’t require registration or license key.

The source code is at samples.

Video Recognizer

This application should be used as reference implementation by any developer trying to add ultimateMRZ to their products. It shows how to detect and recognize MRZ lines at realtime using live video stream from the camera.

Benchmark

This application is used to check everything is ok and running as fast as expected. The information about the maximum frame rate (50fps on iPhone7 and 114fps on core i7) could be checked using this application.

Parser

This is a C++ sample application using regular expressions to extract the MRZ fields. If you want to extract the expiry date, document number, holder's name or birth date then, this is the right application. More info about the parser: https://www.doubango.org/SDKs/mrz/docs/MRZ_parser.html

Validation

MRZ lines contains check digits to validate the field. This C++ sample application shows how to validate the MRZ fields to make sure the OCR result from the recognizer is valid. Using the confidence score and validation process allows reaching 100% accuracy.

More info about data validation: https://www.doubango.org/SDKs/mrz/docs/Data_validation.html

Trying the samples

Android

To try the sample applications on Android:

  • Open Android Studio and select Open an existing Android Studio project alt text

  • Navigate to /samples, select android folder and click Open alt text

  • Select the sample you want to try (e.g. videorecognize), the device (e.g. samsung SM-G975F) and press run alt text

iOS

To try the sample applications on iOS just open the corresponding Xcode project. For example, the Xcode project for the VideoRecognizer sample is at ultimateMRZ-SDK/samples/ios/VideoRecognizer/VideoRecognizer.xcodeproj

Windows

To try the sample applications on iOS just open the corresponding Visual Studio project. For example, the VS project for the VideoRecognizer sample is at samples/c++/recognizer/recognizer.vcxproj.

Raspberry Pi Linux, and Others

For Raspberry Pi and other Linux systems you need to build the sample applications from source. More info at samples/c++/README.md.

Getting started

The SDK works on many platforms and comes with support for many programming languages.

Adding the SDK to your project

This Github repository contains binaries for Android, iOS, Raspberry Pi, Linux and Windows. The next sections explain how to add the SDK to an existing project.

Android

The SDK is distributed as an Android Studio module and you can add it as reference or you can also build it and add the AAR to your project. But, the easiest way to add the SDK to your project is by directly including the source. In your build.gradle file add:

android {

      # This is the block to add within "android { } " section
      sourceSets {
         main {
             jniLibs.srcDirs += ['path-to-your-ultimateMRZ-SDK/binaries/android/jniLibs']
             java.srcDirs += ['path-to-your-ultimateMRZ-SDK/java/android']
             assets.srcDirs += ['path-to-your-ultimateMRZ-SDK/assets/models']
         }
      }
}

iOS

On iOS we provide a FAT C++ framework for Xcode. The framework has a single C++ header file which means you can easily write an Obj-C wrapper around it if you want to use Swift language. Please contact us if you want to use the framework with Swift.

The framework is at binaries/ios/ultimate_mrz-sdk.framework.

Make sure you're using latest Xcode version. In the next sections we're using Xcode Version 11.3 (11C29).

  • Select your target (let's say Test), select General section then, Framework, Libraries, and Embedded Content and press + to browse to binaries/ios/ultimate_mrz-sdk.framework to add the framework. alt text

  • Select your target (let's say Test), select Build Settings section then, Framework Search Paths and press + to add path to the folder containing the framework (should be ultimateMRZ-SDK/binaries/ios) alt text

  • Right click on your project, select Add Files to... and browse to ultimateMRZ-SDK/assets to select models the folder. alt text

    The previous action should add the models and configuration files to the bundle resources: alt text

That's it.

Raspberry Pi, Linux, Windows and Others

The shared libraries are under ultimateMRZ-SDK/binaries/. The header file at c++. You can use any C++ compiler/linker.

Using the API

It's hard to be lost when you try to use the API as there are only 3 useful functions: init, process and deInit.

The C++ API is defined at https://www.doubango.org/SDKs/mrz/docs/cpp-api.html.

Android (Java)

	import org.doubango.ultimateMrz.Sdk.ULTMRZ_SDK_IMAGE_TYPE;
	import org.doubango.ultimateMrz.Sdk.UltMrzSdkEngine;
	import org.doubango.ultimateMrz.Sdk.UltMrzSdkResult;

	// JSON configuration string
	// More info at https://www.doubango.org/SDKs/mrz/docs/Configuration_options.html
	final static String CONFIG = "{" +
		"\"debug_level\": \"info\"," + 
		"\"debug_write_input_image_enabled\": false," + 
		"\"debug_internal_data_path\": \".\"," + 
		"" + 
		"\"num_threads\": -1," + 
		"\"gpgpu_enabled\": true," + 
		"\"gpgpu_workload_balancing_enabled\": true," + 
		"" + 
		"\"segmenter_accuracy\": \"high\"," + 
		"\"interpolation\": \"bilinear\"," + 
		"\"min_num_lines\": 2," + 
		"\"roi\": [0, 0, 0, 0]," + 
		"\"min_score\": 0.0" + 
	"}";

	@Override
	protected void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		

		// Initialize the engine
		assert UltMrzSdkEngine.init(
				getAssets(),
				CONFIG
		).isOK();
	}

	// Camera listener: https://developer.android.com/reference/android/media/ImageReader.OnImageAvailableListener
	final ImageReader.OnImageAvailableListener mOnImageAvailableListener = new ImageReader.OnImageAvailableListener() {

		@Override
		public void onImageAvailable(ImageReader reader) {
				try {
				    final Image image = reader.acquireLatestImage();
				    if (image == null) {
				        return;
				    }

				    // MRZ recognition
				    final int exifOrientation = 1; // Normal (landscape) - no rotation
				    final Image.Plane[] planes = image.getPlanes();
				    final UltMrzSdkResult result = UltMrzSdkEngine.process(
				        ULTMRZ_SDK_IMAGE_TYPE.ULTMRZ_SDK_IMAGE_TYPE_YUV420P,
				        planes[0].getBuffer(),
				        planes[1].getBuffer(),
				        planes[2].getBuffer(),
				        image.getWidth(),
				        image.getHeight(),
				        planes[0].getRowStride(),
				        planes[1].getRowStride(),
				        planes[2].getRowStride(),
				        planes[1].getPixelStride(),
				        exifOrientation
				    );
				    assert result.isOK();

				    image.close();

				} catch (final Exception e) {
				   e.printStackTrace();
				}
		}
	};

	@Override
	public void onDestroy() {
		// DeInitialize the engine
		assert UltMrzSdkEngine.deInit().isOK();

		super.onDestroy();
	}

iOS, Windows, Raspberry Pi, Linux and Others (C++)

	#include <ultimateMRZ-SDK-API-PUBLIC.h> // C++ API: https://www.doubango.org/SDKs/mrz/docs/cpp-api.html
	
	// JSON configuration string
	// More info at https://www.doubango.org/SDKs/mrz/docs/Configuration_options.html
	static const char* __jsonConfig =
	"{"
	"\"debug_level\": \"info\","
	"\"debug_write_input_image_enabled\": false,"
	"\"debug_internal_data_path\": \".\","
	""
	"\"num_threads\": -1,"
	"\"gpgpu_enabled\": true,"
	#if defined(__arm__) || defined(__thumb__) || defined(__TARGET_ARCH_ARM) || defined(__TARGET_ARCH_THUMB) || defined(_ARM) || defined(_M_ARM) || defined(_M_ARMT) || defined(__arm) || defined(__aarch64__)
	"\"gpgpu_workload_balancing_enabled\": true,"
	#else // Disable GPGPU/CPU work balancing on x86 and enable it on ARM -> https://devtalk.nvidia.com/default/topic/494659/execute-kernels-without-100-cpu-busy-wait-/
	"\"gpgpu_workload_balancing_enabled\": false,"
	#endif
	""
	"\"segmenter_accuracy\": \"high\","
	"\"interpolation\": \"bilinear\","
	"\"min_num_lines\": 2,"
	"\"roi\": [0, 0, 0, 0],"
	"\"min_score\": 0.0"
	"}";

	// Local variable
	UltMrzSdkResult result(0, "OK", "{}");

	// Initialize the engine (should be done once)
	ULTMRZ_SDK_ASSERT((result = UltMrzSdkEngine::init(
		__jsonConfig
	)).isOK());

	// Processing (detection + recognition)
	// Call this function for every video frame
	const void* imageData = nullptr;
	ULTMRZ_SDK_ASSERT((*result_ = UltMrzSdkEngine::process(
			ULTMRZ_SDK_IMAGE_TYPE_RGB24,
			imageData,
			imageWidth,
			imageHeight
		)).isOK());

	// DeInit
	// Call this function before exiting the app to free the allocate resources
	// You must not call process() after calling this function
	ULTMRZ_SDK_ASSERT((result = UltMrzSdkEngine::deInit()).isOK());

Again, please check the sample applications for more information on how to use the API.

Technical questions

Please check our discussion group or twitter account