The rollout of an artificial intelligence system to help child welfare centers identify abusive situations where children require protective custody has been postponed over accuracy and reliability concerns.
During trials, the AI frequently missed significant signs of severe abuse, raising alarm among staff and prompting the Children and Families Agency to delay the system’s introduction.
Since fiscal 2021, the government has invested around 1 billion yen ($6.7 million) into developing the system, intended to ease workloads and improve decision-making at child welfare centers.
The AI was trained on 5,000 cases from child welfare centers across Japan, analyzing 91 factors including the presence of injuries, a child’s reluctance to return home, a history of abuse and the caregiver’s attitude.
Based on these factors, the AI generates a risk-of-abuse score on a scale from 0 to 100—with 0 indicating no signs of abuse and high scores indicating severe danger.
This fiscal year, a prototype was tested with 100 cases across 10 municipalities.
However, in 62 of these cases, senior staff at the child welfare centers raised concerns that the AI scores were either too high or too low.
For example, the AI scored several cases in which children reported severe abuse from their mothers, including being beaten and kicked, as low as 2 or 3—while human staff found those children needed immediate placement in temporary protection.
Given the tool’s critical role in decisions affecting children’s lives, the agency has decided it is too difficult to implement the AI this fiscal year.
Instead, the agency allocated funds in this fiscal year’s supplementary budget to develop tools for transcribing and summarizing interviews at child welfare centers.
In addition to reducing the burden of record-keeping, the agency is also considering using the tools to collect text data and enhance the AI’s risk analysis capabilities in the future.