１．Background at the beginning of research

The time has come when it is possible to develop a palm-sized system that records human behavior in millisecond intervals using highly integrated sensors such as acceleration, geomagnetism, and gyro. There is a possibility that the sensor will be worn by a human and the obtained data will be fed back to lead to the next operation. In this study, we focus on the effect of sound stimulus on human motor control and discover whether the body balance can be controlled by converting the sensor output into sound and feeding it back. There are two possible controllable methods: control by sound stimulus to the potential motor ability of humans, and control by relying on sound by learning sound stimulus, and a flexible high-performance sensor system. Focusing on development, we will conduct experiments and explore the principle and mechanism.

(1) Sensors progresses due to the development of nanotechnology

With the development of nanotechnology, especially micromachine technology, sensors that can observe directions, movement speeds, and postures, are available for observing human behavior such as acceleration, gyro, and geomagnetism. Those can be packed into LSIs about the size of a fingertip. The time has come when the sensor LSI manufacturers are searching for ways to use it.

(2) The time has come when sensors can be embedded and data can be fed back in real time.

The most promising way to use it is to carry around an ultra-small sensor, feed back the data obtained from the sensor to humans, and reflect it in the next action. Since the sensor is small and can be carried around at all times, it is expected to be applied to health care, sports training and rehabilitation.

(3) There are problems with traditional sports training and rehabilitation in the feedback techniques

There is a conventional technology that feeds back and connects it to the next operation. For example, control is performed by direct electrical stimulation of muscles, and balance sensation is controlled by vestibular electrical stimulation. It is difficult to put it into practical use due to concerns about physical effects, long latency to perception, and only rough control of front and back, left and right.

The advanced studies related to this research project include body balance control by vestibular electrical stimulation. It is known that direct electrical stimulation can affect body balance. However, no matter how safe it is, the ethical discomfort with direct electrical stimulation causes anxiety for rehabilitating patients and sports athletes. In addition, treatment methods using auditory rhythm input for patients with intractable diseases have been proposed. Effective results have been shown. However, controlling the physical balance of healthy subjects with simple auditory stimuli has been attempted at the level of basic research. However, The way that accelerometer data is converted into sound is still no attempt to derive the next movement from the potential athletic ability by giving feedback.

2．Research objectives

a）Research aim

In this research, we have a question: Can the body balance be controlled by converting the acceleration sensor data into sound and feeding it back as shown in Fig. 1?  In other words, the sound feedback that accompanies exercise explores the facts of the following two questions.

① Can it affect your potential athletic performance?

② Is it possible to control the body by learning sound stimulation?

We will set two themes, and explore whether that fact exists. Specifically, we will elucidate based on experiments, focusing on the development of a sensor system that can convert sensor data to sound at high speed and change the type of sound.

b) New principles explored in this research

There are two methods of posture control considered in this study. The first is a method to derive human motor ability by inducing an illusion from auditory stimulus. For example, if a sound is input to the right ear of a person who is going straight, it is expected that the effect of turning to the left or right can be obtained. If this principle can be discovered, the technology for rehabilitation systems that feed back sound will improve drastically. The second is a method to guide human’s posture by the sound. In this method, the subject is trained to consciously follow the sound source, the next posture is induced by the sound. The sound is converted from the sensor data.

From the above two methodologies, the following principles are expected.

(New principle 1) Auditory stimulation by sound feedback affects the potential motor ability of humans.

(New principle 2) Body balance can be controlled by learning to follow sounds.

The purpose of this research is to change the body balance control when these two principles are combined and the learning progresses to a higher degree.

c) Expected results

　Our first aim is (New Principle 1), but even if it cannot be discovered, we hope that the following results will be obtained according to (New Principle 2).

(1) Can we develop a rehabilitation system that directly induce the human posture ?

We expect a rehabilitation system that does not require equipment such as electrodes that are directly attached to the human body. Simply attach a small sensor system to our body and listen to the sound to receive a medical examination program, enabling home treatment.

(2) A system to avoid dangers such as falls of the elderly can be created

　It can be applied to a pre-risk avoidance system that analyzes the sensor data pattern immediately before sound feedback and outputs a posture correction warning by voice when it matches the fall pattern of the elderly.

(3) Expected to be applied to sports training

A system that corrects athletes’ wrong movement and the habits using real-time sensor data can be developed. In addition, it is expected that the era will come when elementary and junior high school students will improve their sports techniques using the data of top athletes according to (New Principle 2）.

(4) Assist system for the blind people can be developed

By integrating video input such as a stereo camera with data from an accelerometer, it is possible to develop an assist system that expands the range of action for the blind people. For example, the step at your feet is notified by sound to avoid danger.

(5) Can be applied to entertainment called “sound game”

It can be applied to games played with sound and can be used in the entertainment industry.

3．Research outcomes

The results obtained by this study can be classified into (i) basic human motor ability of body balance control for hearing, (ii) motor learning when voice is fed back, and body balance control. Each will be described below.

(1) Sound guidance differs depending on the type of change in the sound.

Guidance by the potential ability of body balance control by sound was impossible as described above, but as a result of conducting an experiment on Fig. 3 (b) that consciously pursues sound stimuli, changes in sound that change from side to side. Therefore, it was discovered that the amount of rotation for changing the direction of travel differs depending on the method of changing the sound when the walking direction is followed.In this experiment, the pan was changed from side to side by the sound stimulus, and an experiment was conducted to follow it. The method of the change was divided into a sigmoid function and a heaviside function, and these were randomly presented to the participants. At that time, the case of using the Heaviside function (when the sound changes from side to side in a rectangle) is better than the case of sound change using the sigmoid function (that is, when the sound moves gently from side to side). It turned out that the amount of rotation was large. This is an application that guides you to chase the voice (for example, rehabilitation), and when you want to control the amount of rotation of the body, change the pan with the heaviside function to induce more body movement. It can be expected that applications such as, will be possible.

(2) We found that sound-based body balance control is related to vision.

In this result, we found that the reaction time to sound stimuli was short when visual perception was obtained. In other words, it was clarified that the balance control by the auditory stimulus is also related to the visual information, and that the visual information helps to make a high-speed reaction.

(3) No motor learning ability by sound could be found.

According to the experiments, we have confirmed that it is impossible to improve the accuracy of body balance control by learning exercise by sound, that is, by obtaining the same sound stimulus.

(3) No motor learning ability by sound could be found.

（２）It turns out that the sound feedback system has engineering challenges.