Revolutionizing Compliance Document Reviews with Machine Learning
In six weeks, HatchWorks AI built a prototype machine learning service that predicts the compliance status of a document and points to the exact text behind its verdict: 80% accuracy on test data, against a 50% benchmark set by stakeholders.
A six-week prototype to take the manual work out of compliance review
HatchWorks was engaged by a leading cybersecurity consultancy on a six-week strategic initiative to develop a prototype machine learning service. This service, leveraging the client's data, was designed to predict the compliance status of a document.
Once fully realized, this service promises to significantly reduce the time consultants invest in manually assessing customer compliance documents.
A vast team of consultants, reading documents by hand
The manual review of customer compliance documentation was a significant bottleneck. Employing a vast team of consultants for this seemingly straightforward task was resource-intensive and fraught with inconsistencies and subjective interpretations.
Given their lack of in-house machine learning capabilities, the client sought HatchWorks' expertise.
Where the manual process broke down
-
01
Streamlining workflows for consistency
Reviews varied from consultant to consultant, with subjective interpretations of the same requirements.
-
02
Upholding quality standards
Keeping every assessment to the same bar was hard when the work was repetitive and resource-intensive.
-
03
Keeping consultants engaged
Skilled consultants spent their time on a seemingly straightforward task instead of higher-value work.
Six weeks to validate feasibility
-
01
Training data from past projects
The client provided structured datasets from past projects to train the ML model. The data was cleansed and structured for model training.
-
02
PCI-DSS requirements as queries
Compliance requirements, specifically from the PCI-DSS standard, were ingested and transformed into optimized queries.
-
03
Embedding retrieval, fine-tuned decision
These queries enabled an embedding model to sift through vast documentation, extracting relevant contexts. Each context was processed by a fine-tuned language model, which determined compliance based on historical data.
-
04
Collaborative validation
Collaborative virtual sessions with client stakeholders ensured data quality, model accuracy, and alignment throughout.
Two models, one verdict: the embedding model finds the passages that matter, the fine-tuned language model decides whether they meet the requirement.
Feasibility validated, with the benchmark cleared by 30 points
By the project's end, the language model showcased its capability to assess compliance documentation, pinpointing specific text references. Achieving this within six weeks underscored the potential of the approach.
The model reached 80% accuracy against test data, surpassing the initial 50% benchmark set by stakeholders. The prototype validated feasibility and highlighted areas for future improvement, especially with more data.
The 50% benchmark was the initial target set by client stakeholders before the engagement.
A vast team of consultants reading every compliance document by hand
Inconsistent results and subjective interpretations across reviewers
No in-house machine learning capability to change the approach
A prototype service that predicts compliance status from the client's own data
Verdicts that pinpoint specific text references, ready for a consultant to verify
80% accuracy on test data, with a clear path to improve as more data arrives
The prototype, in numbers
model accuracy against test data
initial benchmark set by stakeholders, surpassed
from kickoff to a working prototype
The approach
In their words
“It has been a pleasure working with HatchWorks for almost two years now. They work seamlessly with my team, have overlapping time zones for efficient Agile development, and have a governance model to ensure success. HatchWorks has become our partner for several successful ongoing projects. I like their US-based Headquarters, and quality delivery teams working in Latin America. I plan to continue working with HatchWorks for our mutual journey and partnership.”
About the work
HatchWorks AI turns AI into ROI by automating key business processes, transforming data, deploying intelligent agents, and shipping AI-powered products that deliver measurable results. Its proprietary Generative-Driven Development methodology is a repeatable path from idea to production, blending AI, agents, and engineering to ship faster with less risk.
Have a manual review process worth automating?
See how HatchWorks AI validates machine learning ideas in weeks, using your own data.