Workflow Automation

1. How Workflow Automation Works

First, let me briefly explain how simulations are conducted in the application we developed. Fundamentally, various cases are fed into the system one model to make judgments and decisions, which then determine the downstream actions. However, for judgments where the system one model lacks confidence, the task is routed to an LLM (Large Language Model) to ensure a more careful decision.

The overall automation flow is shown in the chart below. Pay particular attention to the branching point after the initial system one model judgment: depending on whether the confidence score exceeds the threshold, the judgment is either adopted as-is or routed to the LLM for further evaluation.

The screen above shows the application's GUI. On the left, there are selection buttons for the system one model and the LLM. When the confidence score of the system one model is low, the decision is delegated to the LLM, and you can adjust the threshold for this using a slider. For this simulation, I set it to 0.95. Furthermore, you can input the actual workflow to be automated in natural language into the box on the right. Here, the system one model which financial product a bank customer's complaint refers to, which department should handle it, and what subsequent actions to take.

2. Verification of Simulation Accuracy

In this simulation, I used "Jev" for the system one model and "Gemini 3.8 flash" for the LLM. one of the instructions in this test includes ground-truth labels indicating which financial product each customer complaint pertains to, allowing us to verify accuracy. As shown below, the accuracy when judged solely by the system one model was 85%. By routing samples that fell below the threshold to the LLM to improve performance, the overall accuracy rose to 90%. Furthermore, for cases where the system one model had a confidence score of 0.95 or higher, the accuracy reached 93.8%, demonstrating that the system one model's judgments are highly accurate when its confidence is high. On the other hand, the accuracy for cases below the threshold that were delegated to the LLM was 75%, indicating that these were inherently difficult cases to judge.

3. Effectiveness of Workflow Automation

Ultimately, the effectiveness of workflow automation using a system one model depends on having a high proportion of high-confidence cases. The chart below shows the distribution of confidence scores for each instruction. Those in green, close to a confidence score of 1, can be considered highly accurate in their classifications. Among these, "Compliance Risk" has a distribution extending toward lower confidence scores, indicating that it was a difficult instruction for the system one model to judge. For such instructions, further refinements to the prompt will likely be needed to boost confidence scores.