Smooth interaction communication between sign language and audio."SURETALK" developer interview

I want to communicate casually with people who use different words.With recent advancement of technology, that wish has gradually been realized, for example, a foreign language automatic translator has made it easier to understand foreign languages.Similarly, people who use sign language as the first language and those who do not use sign language can not communicate without stress ... About such thoughts, about the two -way communication system developed by sign language and audio, "SURETALK".We asked the person in charge who had been involved since the 2017 project about the service.

Softbank Co., Ltd. Technology Unit Service Promotion Department SURETALK Section Keiyuki Tanaka

After engaging in the construction and maintenance of a submarine cable at an IT infrastructure business company, he went to Softbank Co., Ltd.Launched a SURETALK project in 2017 and engaged as a person in charge

The contents of sign language and audio are displayed on the screen.Communication tools that can be used without restrictions on location and time

I think the employment of people with disabilities in private companies is progressing, and I think that you have the opportunity to work with your colleagues who have some obstacles in your workplace.A variety of employees are also active in SoftBank, and some employees have hearing impairment and use sign language.At the site where you actually work, you may not be able to communicate smoothly at a meeting where employees who do not understand sign language are present, and they feel stressed, and they are inconvenient because they cannot receive the necessary information in an emergency.There was a communication issue.So, Suretalk was developed so that the other party could understand the content just by expressing it in sign language.

SURETALK is a service that uses sign language and audio to communicate in real time for hearing and hearing people.Because it is used via a personal computer or tablet, you can communicate, whether your opponent is far or in the same place.It also has a function to learn user sign language to increase the accuracy of sign language recognition.State -of -the -art AI technology is used for text conversion and sign language learning functions from sign language and voice, and supports sign language recognition and language processing issues.

The feature of Suretalk is that in addition to the function of recognizing sign language and converting text, it has a function to make audio text.Both hearing impaired and hearing people can complete communication in the same service.

手話で話した内容がどんどん認識され、テキスト表示

State of actual demonstration

I was actually shown how SURETALK was used.When talking in sign language, I was able to clearly see each operation because it was basically the basics.The image is the speed of sign language learning programs for beginners, not the speed of simultaneous interpretation, which is often seen on TV broadcasts at interviews recently.

The time to convert to text after talking is very short, and the correct sentence is displayed in a few seconds.The interaction with the speakers in a distant place was uncomfortable, and it was as smooth as when using a computer or smartphone text chat tool.

State of physical operation tracking

Also, if you look at the processing performed behind the demo video, you can see that the skeleton is tracking human movements.

Sign language recognition and verbalization are easy and difficult.Introducing cutting -edge AI technology for solution

The main function conversation uses a personal computer or tablet that can use Suretalk for those who speak sign language and those who can speak.When talking to the camera on the terminal in sign language, the characteristics of the skeleton and joints are extracted from the movement, and the word is identified by matching it with the database.The specified word is added by natural language processing (1) athletic, and (2) converted the ending to make it a natural Japanese.The processed content is displayed on both terminals.

The opponent who saw the text talks to the microphone of the terminal, recognizes the voice in the order of speaking, and displays text on the terminal.The audio can be recognized even in noise and response environments, and we plan to add a function to identify speakers in the future.

If the texted content is unintended, it can be modified by quoting the corresponding part and posting a new correction content.

Tanaka "Suretalk's sign language recognition is unique to disassemble, analyze, and perform language processing. For example, the sentence" Where is the window? "After that, if the word "toilet" is registered, you can recognize "Where is the toilet" just by replacing the "window" part? Register a large amount of phrases that match each scene.You don't need to. "

The registration function is to increase accuracy so that people can recognize different sign language movements.There are sign language sentences and word sample videos, so check the movement and register with the same sign language operation as the sample.

Tanaka: "Registration is possible even for those who have never learned sign language. Sample videos will imitate the operation, so registration work will lead to learning, so if you are interested, please try it.please give me"

It is said that about 2,000 words required for the government office window and other general exchanges are assumed to be about 2,000 words.Currently, there are still few words and sentences that can be handled by Suretalk, so it is necessary to increase the number in the future, and ultimately aim for 10,000 words registration.

SureTalkサービスが使用可能な端末とアプリケーション（2021/7/28現在）