Automating assessments with AI

Dawid Naude, Director, Pathfindr

We recently reflected on a common theme we’re noticing in the work we’re delivering. The projects had different names like “accreditation certifier”, “clause validator” or “skilled migrant assessment”, but they were all doing the same thing, assessing something.

What we also realised is that AI is often great at a first draft, or a last pass. In this case, an initial assessment.

Assessing contracts

These were tasks that in the past were hard for computers to assess because of the text content. Take a contract - in the screenshot below is an example from a recent projects where AI has been told to find discrpencies from rules in a contract. If you look at the rule and relevant exerpt columns, you’ll notice that they are completely different in language but the same in meaning, and the tool recognises this. With rule number 4, it’s picked up that the minimum liability should be $2 million not $1 million, flags the discrepancy and then allows you to view the source.



In this example the problem was that for non-disclosure agreements, most lawyers were typically looking for the same things. IP protection, liability and a few others.

Video and course content assessments

Another example was assessing regulated training content to see if it meets professional standards. In the past someone would manually review every minute of every course to make sure that it met professional standards, but what we discovered is that what they were looking for was very achievable to instruct to an LLM agent.

So now a solution reviews all the training content for markers on professional standards, readability, quality. This process used to take thousands of hours per year, now reduced by 90%.

Another example is automating the review of the content provided by a candidate for a skills assessment. What often happens in a complex assessment, like the skilled migrant assessment performed by many industry associations, is that most of the time is taken up by reviewing academic certificates to see if they meet what was mentioned in the application, or that both sides of the driver’s licence are scanned, or that they’ve provided evidence of employment.

The skills assessment is still performed by a specialist, but all the initial prep work validation is done by the bot.

10 ideas for automating assessment

1. Tender response meets the requirements
2. User story quality
3. Contract criteria
4. Job application with job description
5. Tests
6. Complaints submissions (assess severity and type)
7. Compliance audits
8. Customer feedback
9. Supplier performance
10. Statements of work

Go through your current process and see what takes you long to review. Find where the bottlenecks are - who are you waiting on, and could this just initially be reviewed by AI.