• The digital tool can generate a digital picture book with audio narration with one click
• Alibaba Cloud finetuned its Qwen LLMs to fit the learning style of autistic children

Alibaba Group recently introduced an artificial intelligence-powered tool to create picture books for autistic children, building a pathway for them to express their feelings and impressions of the world through art.

Since launching online in June, more than 50,000 people have used the feature, which generates audio-enhanced artbooks from a prompt fed into Alibaba’s Qwen family of large language models (LLMs).

The digital tool is an educational resource for children with autism spectrum disorder (ASD) – a developmental disorder affecting around one in 100 children globally, according to the World Health Organization.

Core symptoms of ASD include difficulties understanding social interactions and communicating, making visual learning formats like picture books a powerful tool for educators and parents of children with ASD.

“By rolling out the digital artbook creation tool, we want to foster emotional expression for children with autism, create an inclusive learning environment and enable acceptance and understanding from the wider society for children with special educational needs,” Chu Hanlong, who leads the project at Alibaba, told Alizila during an interview.

The digital tool harnesses the multimodal capabilities of LLMs developed by Alibaba Cloud to turn a one-sentence plot summary into engaging graphics paired with engaging writing and audio narration.

Parents and teachers can choose the genre and main character of the story, as well as tailor the content to a specific age group to match their child’s interests.

Visitors try out the digital picture book feature during the World Artificial Intelligence Conference in Shanghai. Photo credit: Alibaba Group

Finetuned LLMs

Generating words, audio and images for a digital picture book at the push of a button is no small feat. To do so, Alibaba Cloud engineers leveraged its multi-agent framework ModelScope Agent to orchestrate a host of LLMs and AI tools to perform tasks autonomously with minimal human intervention.  

“We customized the LLMs in accordance with the cognitive characteristics of children with autism, striving to create simple images and prominent characters, with a plain and direct writing style to better suit their needs,” said Li Chenliang, an AI algorithm engineer at Alibaba Cloud.

The team conducted research on the cognitive traits of children with autism, finding it best to avoid metaphors, puns, and idioms in language, and employ simple and impactful drawing with one main character.

The team also finetuned Alibaba Cloud’s image generation model to maintain consistency of the main character’s image across chapters and ensure diversity in the story settings.

Alibaba Cloud has open sourced the multi-agent framework on its AI model community ModelScope for developers worldwide.

Over 50,000 people have accessed the digital picture book feature on the Tongyi app. Photo credit: Alibaba Group

A Group Effort

Alibaba also partnered with a number of prominent institutions outside the realm of AI to bring the initiative to life.

Shanghai Animation Studio, which has created over 500 original animated pictures since 1957, gave its support to the digital platform to generate picture books featuring its iconic cartoon characters.

Alibaba also partnered up with two institutions focusing on education for children with autism to better understand the perceptions of parents of autistic children and special education teachers regarding educational interventions, their day-to-day challenges and more.

“By getting teachers and parents involved, we can combine the customized AI artbooks with other invention strategies to serve as an important tool for providing companionship and targeted learning for children with autism,” Wang Yonggu, a professor from Zhejiang University of Technology with expertise in special education, said in an interview.

Parents of autistic children and special education teachers can now access the digital tool in China on Alibaba’s Tongyi app.