Showcasing how UiPath Document Understanding can extract signatures from bank checks and compare them with another signature stored in UiPath Data Service for validity.
Signatures are one of the most critical data elements we see in documents every day. There are many use cases for signature verification that require manual involvement. However, UiPath Document Understanding provides the capability to extract and compare signatures in documents automatically.
Bank checks are of different sizes and slightly different layouts. We used a dataset available in Kaggle to select some samples to build our use case.
Download public dataset for checks: Kaggle dataset link
The requirement is to extract the name, check date, amount, check number, payment account number, check holder's account number, and signature.
The solution should cover the following steps.
Read all checks in the input folder.
Extract the information and save it in an excel file
Check if a signature exists in the document and extract it as an image
Download the signature from the primary data source (Data Service entity)
Compare the two signatures for verification
Note: Video is also available for your reference at the end.
Step 1: Prepare Your Solution
Create a project in UiPath Studio and include the following dependencies:
UiPath.DocumentUnderstanding.ML.Activities
UiPath.IntelligentOCR.Activities
UiPath.OCR.Activities
UiPath.MLService.Activities
UiPath.DataService.Activities
ImageActivities.Activities
Dependency | Description |
UiPath.DocumentUnderstanding.ML.Activities | Activity package that enables Machine Learning Document Understanding feature |
UiPath.IntelligentOCR.Activities | Core activities that enable the usage of a complete document processing framework |
UiPath.OCR.Activities | Includes all the OCR engines required for document digitization |
UiPath.MLService.Activities | Contains the activities required to connect with other machine learning models in AI Center, including Signature Comparison model |
UiPath.DataService.Activities | Contains activities required to connect and query Entities in Data Service |
ImageActivities.Activities | Contains activities required to crop and edit images for higher contrast |
Prepare the configuration data that includes the input folder path, the path where extracted signatures are stored, the path where files downloaded from Data Service is stored, and the path where extracted check data are stored. In addition, include the OCR endpoint and the API key in the configuration file.
UiPath Document OCR Endpoint: https://du.uipath.com/ocr
Step 2: Deploy the Signature Comparison Skill
Deploying the Signature Comparison AI model in UiPath is very easy. Log in to your cloud platform and navigate to the AI Center to perform the following actions.
Create a project in AI Center to start creating models
Navigate to ML Packages and locate the UiPath Image Analysis packages under Out-of-the-box packages.
Create a package using the Signature Comparison model.
The model is not retrainable. Hence, we can directly create an ML Skill using the package.
Step 3: Creating Data Service Entity
The Data Service Entity acts as a data storage location that holds all signatures. The signature is saved along with the name of the customer. The signature file is also renamed with the customer name.
Perform the following steps to configure the entity
Create a Data Service Entity with two columns to hold the Customer Name and the Signature File.
Upload the records to the entity through the Data Service Entity Interface
The entity should look similar to the following image.
Step 4: Studio Workflow to Extract the Signature
The Document Understanding workflow consists of several sections: taxonomy creation, digitization, data extraction, and validation.
Taxonomy Creation
Taxonomy enables the robot to understand what fields to extract and the data types expected for each field. Use the Taxonomy Manager to create the document type as defined.
Create a Group named "Financial."
Create a Category under the group named "Incoming."
Create the Document Type as "Checks"
Use the document type created to define the fields we need to extract.
Define the data type of the Signature field as Boolean to check if signature exists before extracting the signature
Document Understanding Workflow Creation
The first few steps of the document understanding workflow are the same as any document understanding flow. The stages consist of the following:
Loading the Taxonomy
Loop through the input documents
Digitize using UiPath Document OCR (default OCR engine)
Template-based data extraction using Form Extractor
The Form Extractor is used for the data extraction because the checks used for the use case share a similar layout. Further, the Form Extractor is capable of identifying handwritten text and signatures.
Ensure to select the Signature field as a field used for signature detection in the Template Manager, as shown below.
Once done with template creation, configure the Data Extraction Scope activity to use Form Extractor to extract all the fields.
Use the Validation Station activity and run it a couple of times to see how the values get extracted. The signature field should show up as True if it could detect the signature.
Workflow for Signature Extraction
Every value extracted from the document returns the coordinates where the text is in the document. The region coordinates play an important role in extracting the signature as an image from the document. The region coordinates are mentioned in the Extraction Results variable provided by the Data Extraction Scope activity. The sample output provided below as a JSON structure helps us identify how to navigate in the JSON to locate the regions (named BOX).
Accessing the Field requires navigating through the following:
Variable -> Results Document -> Fields (<index>) -> Results Data PointAccessing the BOX of the given field requires navigating through the following
Variable -> Results Document -> Fields (Index) -> Results Data Point -> Values (<index>) -> Reference -> Tokens (<index>) -> Boxes (<box index>)The structure of the sample output result is as follows.
ExtractionResult
{
DocumentId="Cheque083654",
ExtractorPayloads=ExtractorPayload[0] { },
ResultsDocument=ResultsDocument
{
Bounds=ResultsDocumentBounds
{
PageCount=1,
StartPage=0,
TextLength=576,
TextStartIndex=0
},
DataVersion=1,
DocumentCategory="Payments",
DocumentGroup="Financial",
DocumentTypeDataVersion=0,
DocumentTypeField=ResultsValue
{
Components=ResultsDataPoint[0] { },
Confidence=1,
DerivedFields=ResultsDerivedField[0] { },
OcrConfidence=1,
OperatorConfirmed=true,
Reference=ResultsContentReference
{
TextLength=0,
TextStartIndex=0,
Tokens=ResultsValueTokens[0] { }
},
UnformattedValue="",
Value="Checks"
},
DocumentTypeId="Financial.Payments.Checks",
DocumentTypeName="Checks",
DocumentTypeSource=Automatic,
Fields=ResultsDataPoint[7]
{
ResultsDataPoint
{
DataSource=Automatic,
DataVersion=0,
FieldId="Financial.Payments.Checks.PayName",
FieldName="Pay Name",
FieldType=Text,
IsMissing=false,
OperatorConfirmed=true,
Values=ResultsValue[1]
{
ResultsValue
{
Components=ResultsDataPoint[0] { },
Confidence=1,
DerivedFields=ResultsDerivedField[0] { },
OcrConfidence=1,
OperatorConfirmed=true,
Reference=ResultsContentReference
{
TextLength=22,
TextStartIndex=252,
Tokens=ResultsValueTokens[1]
{
ResultsValueTokens
{
Boxes=Box[3]
{
T:199,L:420,W:366,H:130,
T:199,L:829,W:302,H:130,
T:199,L:1152,W:280,H:130
},
Page=0,
PageHeight=1093,
PageWidth=2372,
TextLength=22,
TextStartIndex=252
}
}
},
UnformattedValue="",
Value="DadabaD. Pradeep Kumar"
}
}
},
ResultsDataPoint
{
DataSource=Automatic,
DataVersion=0,
FieldId="Financial.Payments.Checks.Amount",
FieldName="Amount",
FieldType=Text,
IsMissing=false,
OperatorConfirmed=true,
Values=ResultsValue[1]
{
ResultsValue
{
Components=ResultsDataPoint[0] { },
Confidence=1,
DerivedFields=ResultsDerivedField[0] { },
OcrConfidence=1,
OperatorConfirmed=true,
Reference=ResultsContentReference
{
TextLength=11,
TextStartIndex=336,
Tokens=ResultsValueTokens[1]
{
ResultsValueTokens
{
Boxes=Box[1]
{
T:407,L:1829,W:418,H:96
},
Page=0,
PageHeight=1093,
PageWidth=2372,
TextLength=11,
TextStartIndex=336
}
}
},
UnformattedValue="",
Value="88.00,000/-"
}
}
},
....
As we see in the sample result, the above-mentioned structure repeats for all the fields. We can navigate to each field using indexes and the unique field identifier (Field ID).
Let's follow the following steps to extract the signature from the document.
Identify Signature Field
The signature field is identified using the unique field identifier. The identifier is created based on the configuration you did in the Taxonomy Manager. The filed identifier is built by combining Group, Category, Document Type, and the Field Name.
In our example, the signature field ID is:
Financial.Incoming.Check.SignaturePlace an Assign activity and configure it using the following formula to extract the signature field.
SignatureField = ValidatedExtractionResults.ResultsDocument.Fields.Where(Function (f) f.FieldId = "Financial.Incoming.Check.Signature").FirstThe data type of the signature field variable should be:
UiPath.DocumentProcessing.Contracts.Results.ResultsDataPoint Checking if Signature Exists
Once the signature field is extracted, it is important to check whether the extraction was able to capture the signature. We can easily check if the document contains a signature using the properties of the signature field using an IF condition.
IF SignatureField.IsMissing
{
//Proceed with signature extraction
}
Else
{
Log message "Does not have a signature"
}Identify Signature Field Box
Signature Field extracted provides access to the field's value and region. Use the following formula in another Assign activity to capture the BOX.
Configure the Value property of the Assign activity with the following formula
SignatureFieldVariable.Values(0).Reference.Tokens(0).Boxes(0)Create a new variable to hold the Signature Box information. The data type of the field is:
UiPath.DocumentProcessing.Contracts.BoxLoad Document as Image
The document we process could be of any file format. For example, the check received could be in PDF or image format. If the file is not an image, it is required to convert the document into a JPG file using a file conversion activity.
PDF files can be converted into Image files using UiPath PDF activities
The image file is now ready to be loaded as an image object in the workflow. We can use the Load Image activity to perform this action.
Cropping to Retrieve the Signature
The BOX extracted earlier contains the information needed to crop the signature from the document image. Use an Invoke Code activity and configure it with the following arguments.
Configure the Invoke Code activity with the following code to crop the signature.
ExtractedSignature = CheckImage.SimpleCrop(New Rectangle(cint(SignatureBox.Left), cint(SignatureBox.Top), cint(SignatureBox.Width), cint(SignatureBox.Height)))The code returns the cropped signature as an image object to the main workflow.
The cropped image should go through a simple enhancement step to enhance the ability to read it in the comparison stage. Use the Image Contract activity available under the Image Activities pack to perform this step.
Set the Contrast Factor property of the activity to 100 and save the output of the activity as a JPG image file.
Now you have the signature as an image file, ready for comparison.
Step 5: Studio Workflow to Compare Signatures
Create a separate workflow to perform the signature download from Data Service and the Comparison.
Configure the input arguments of the new workflow to have the following:
Customer Name mentioned in the document
The file path for the extracted signature
The folder path to save the downloaded signature
The new workflow should start by querying the data service with the customer's name.
Configure Studio to Access Data Service Entity
Access the Manage Entities option in the top Ribbon and configure it to enable Studio to access the entity you created.
Query Entity to Download the Signature File
The data in the entity is saved using the customer name as the primary reference. We can query the entity using the customer name provided in the input argument to access the record and download the file.
Use the Query Entity Records activity and configure it to access the record using the Customer Name field.
Use the output of the above activity to download the file using the Download File From Record Field activity.
The workflow should look similar to the following screenshot.
Using Signature Comparison Model
The signature comparison model deployed in AI Center requires two signatures as the input. The input is expected to be in a comma-separated string.
Use the two variables that contain the file paths to both signature files and concatenate them into one string separated by a comma (",").
String ConcatenatedSignatureFiles = DownloadedSignaturePath + "," + in_ExtractedSignaturePathPlace an ML Skill activity and configure it by providing the combined file paths as the input and getting the JSON response as the output.
The JSON response contains the matching confidence and a few other information you can use in your validation logic.
Use UiPath Web API activities to deserialize the JSON string and access the values returned
Once done, invoke this workflow from the previous workflow we created to complete the whole process. Run the process and experience how seamlessly UiPath extracts and compares the signatures.
Watch the video explaining the whole process.
Conclusion
UiPath Document Understanding compared with machine learning and image processing functionalities enable seamless signature extraction and comparison. The article showcased a bank check processing use case where UiPath automates the signature extraction and comparison to validate checks.
Additional Helpful Links:
Find more similar content under Tutorials: UiPath Document Understanding Concepts
Want to know more about Document Understanding? Follow a collection of Tutorials here to get in-depth knowledge.
Don't forget to subscribe and follow for more updates.. 👇🏻