Agent Observability

Need

To fully exploit automated Meta-Data extraction across a set of documents. Analysts must be able to validate the accuracy of the automatically extracted data in a way that does not lose all the speed gain from the automated extraction process.

To realize the speed improvements of automated Meta-Data extraction, Analysts must be able to validate and correct the accuracy of the automatically extracted data from an individual document while keeping up with the volume and pace of the process.

Idea

Users of the Heat Shield tool use the algorithm observability screen to view the results of the extraction algorithm (batch process). Because of the data overload in this domain, Heat Shield was designed so that users do not have to manually validate every extraction from the algorithm. The observability screen gives users insights into the potential extraction problems, so that users can select a sub-set of extractions for manual validation. Also, a quality control mechanism has the users validate a sample of extractions. After completing the extraction validation (on the extraction validation screen) users are shown the probable remainder of error based on an extrapolation of the error found during the validation and quality control process.

Resilient Cognitive Solutions’ Heat Shield interface allows Analysts to see extracted Meta-Data in the context of the document from which it was obtained. By hue-coding the extracted data in the source document, the proper context for validating extracted Meta-Data can be quickly established.

Benefit

The Heat Shield system was designed to help users overcome data overload. By showing the users where the algorithm struggled with the extraction, they are able to point their manual validation effort in the most beneficial places. Also, the observability of the estimated residual error provides powerful work management insights.

The Heat Shield Meta-Data validation screen allows Analysts to direct their attention to those Meta-Data extractions that need to be validated, and then use the original source document as the proper context for manually validating candidate Meta-Data extractions. The interface creates a new Joint Cognitive System (analyst and extraction algorithm) which is faster and more accurate than either could be alone.