The previous chapters focused on the way we leverage haptics for the design of either input or output systems.
The first chapter, about haptic as the sense of touch, barely mentions input.
-At the opposite, the second chapter, about haptic as the motor ability, barely mention output.
+On the opposite, the second chapter, about haptic as the motor ability, barely mentions output.
%Knowledgeable readers may be frustrated that these
This separation sometimes seems to make sense.
%It was mostly the case in the contributions discussed in these chapters.
-For example Tactons used to convey information for notifications does not necessarily require user input.
-However combining output with gestural input enable users to get more detailed information~\cite{pasquero11}.
-Multi-touch and gestural interfaces generally use visual feedback to compensate the lack of physicality of controls.
+For example, Tactons used to convey information for notifications do not necessarily require user input.
+However, combining output with gestural input enable users to get more detailed information~\cite{pasquero11}.
+Multi-touch and gestural interfaces generally use visual feedback to compensate for the lack of physicality of controls.
But the problem remains the same: whether the feedback is visual or haptic, it requires a systematic design.
-The message of this chapter is that input and output have an equivalent importance for the design of interactive systems, and must be designed together.
+The message of this chapter is that input and output have equivalent importance for the design of interactive systems, and must be designed together.
This is not as trivial as it seems to be at first sight.
-To illustrate this, I describe below two parts of rejected submissions, and explain what was wrong with them.
+To illustrate this, I describe below two parts of rejected submissions and explain what was wrong with them.
It has to do with the motivations and the lack of focus on the sensorimotor loop.
It made me realize that this is a large hole in the two previous chapters.
After this assessment, I will connect the various angles of the sensorimotor loop, and discuss what it brings to the different disciplines around HCI.
\section{The limits}
\label{sec:limits}
-During my postdoc at the University of Toronto I worked on the lack of haptic feedback in 3D gestural interaction.
+During my postdoc at the University of Toronto, I worked on the lack of haptic feedback in 3D gestural interaction.
The Microsoft Kinect\footurl{https://tiny.one/Kinect} was just released.
It featured a motion capture system affordable for households.
It followed the trend of \emph{natural} user interfaces, and was advertised as “\emph{You} are the controller”\footurl{https://tiny.one/youAreTheController}.
-Besides the discussion about the natural aspect of user interfaces and their relevance in the previous chapter, this motivation to eliminate physical controllers also eliminated many useful, not to say essential, haptic properties of physical controlers.
+Besides the discussion about the natural aspect of user interfaces and their relevance in the previous chapter, this motivation to eliminate physical controllers also eliminated many useful, not to say essential, haptic properties of physical controllers.
This was already an issue with touch interaction, but there was at least the passive haptics of the surface.
In the case of 3D gestural interaction, the users have no haptic feedback when they interact with virtual objects.
Therefore, the immediate feedback they need for direct manipulation~\cite{schneiderman83} is essentially visual or auditory.
This contributes to the overload of these modalities.
Therefore, I worked on a way to provide haptic feedback for 3D interaction.
-%Before getting into this, I will start with a technical note on the device I design and implemented for the studies belos.
-The design rationale was to keep the users' hands free since this was the main motivations of this type of sensor.
+%Before getting into this, I will start with a technical note on the device I design and implemented for the studies below.
+The design rationale was to keep the users' hands free since this was the main motivation of this type of sensor.
If users have to hold a device for haptic feedback, this device could also serve as an input controller.
-%Otherwise tactile feedback could be integrated in a controler.
+%Otherwise tactile feedback could be integrated into a controller.
Therefore we opted for a wearable haptic device for the wrist.
\paragraph{Apparatus}
There are several reasons to discuss the design and implementation of the prototype in this document.
-First it is designed for expressivity with simplicity.
-Second the iterative process is an interesting case study that led to guidelines regarding the implementation of interative systems.
+First, it is designed for expressivity with simplicity.
+Second, the iterative process is an interesting case study that led to guidelines regarding the implementation of interactive systems.
It helped me with the ideation of the concept depicted in \reffig{fig:hapticpath} in \refchap{chap:output}.
We discussed the output vocabulary of vibrotactile feedback in \refchap{chap:output}.
For the sake of simplicity, I rather opted for a straightforward design that enabled the precise control of both frequency and amplitude at the cost of a low control of the signal shape\footnote{Controlling the signal shape remains possible, with a software $\Delta\Sigma$ modulation \href{https://tiny.one/DeltaSigma}{https://tiny.one/DeltaSigma}.}.
The idea is to control the frequency and amplitude with two PWM signals generated by the timers of a microcontroller (\reffig{fig:actuatorcircuit}).
The frequency signal typically ranges between \qtyrange{1}{1000}{\hertz}.
-The amplitude is controlled with the duty cycle of a high-frequency signal.
-We used voice coil actuators, therefore they behave like low-pass filters, which stabilizes this high-frequency signal, hence reducing the amplitude of the actuator's movement.
+The amplitude is controlled by the duty cycle of a high-frequency signal.
+We used voice coil actuators, therefore they behave like low-pass filters, which stabilize this high-frequency signal, hence reducing the amplitude of the actuator's movement.
Our prototypes used \qty{16}{\mega\hertz} controllers with 8 bits timers, which gives a \qty{62.5}{\kilo\hertz} loop with 256 levels of amplitude.
-It communicated with a host computer with a serial protocol over bluetooth.
+It communicated with a host computer with a serial protocol over Bluetooth.
\input{figures/actuatorcircuit.tex}
As shown on \reffig{fig:wristbandprototypes}, the design of the prototypes was iterative.
The first two prototypes used Arduino LilyPad microcontroller boards~\cite{buechley08}.
-These board are designed for wearables, so it made sense for this project.
+These boards are designed for wearables, so they made sense for this project.
On the first prototypes, the components were sewn with conductive thread.
It caused several issues.
-First, the thin conductive thread had a non negligeable resistance.
+First, the thin conductive thread had a non-negligible resistance.
Therefore it was necessary to multiply the connections to have enough power flowing in the circuit.
Second, the elasticity of the wristband was convenient for comfort, but it caused short circuits that made the device unreliable.
To alleviate these issues, I soldered the components on small protoboards, which I connected together with conductive thread.
Now the issue was the connection between the thread and the pads.
Using conductive glue was only a temporary fix because it would not stick long to the pads.
-Given the recurring technical issues, I designed a PCB for the third prototype, which was connected to the microcontroller, battery and actuators with wires.
-This new prototype uses an Arduino Mini Pro microcontroller board\footurl{https://tiny.one/ArduinoMiniPro}, which includes the same microcontroller than the LilyPad: the Atmega 328P\footurl{https://tiny.one/ATmega328P}.
-This is a conveninent 8-bits microcontroller for small designs.
+Given the recurring technical issues, I designed a PCB for the third prototype, which was connected to the microcontroller, battery, and actuators with wires.
+This new prototype uses an Arduino Mini Pro microcontroller board\footurl{https://tiny.one/ArduinoMiniPro}, which includes the same microcontroller as the LilyPad: the Atmega 328P\footurl{https://tiny.one/ATmega328P}.
+This is a convenient 8-bits microcontroller for small designs.
The issue with it was that it only has six timer channels, and I needed eight to control four actuators in frequency and amplitude.
Therefore I used a common frequency signal for all actuators, and an individual Amplitude signal.
-This choice made sense given the type of tactile effects we planned to use, and that we will discuss in the next sections.
+This choice made sense given the type of tactile effects we planned to use, which we will discuss in the next sections.
\begin{figure}[htb]
\centering
\includegraphics[width=\textwidth]{figures/wristband3protos}
- \caption[Haptic wristband prototypes]{Three iterations of haptic wristband prototypes. On the first prototypes, components were connected with conductive thread. On the second prototype, the components were soldered on protoboards that were connected with conductive thread. The third prototype used a home-made PCB connected with wires.}
+ \caption[Haptic wristband prototypes]{Three iterations of haptic wristband prototypes. On the first prototypes, components were connected with conductive thread. On the second prototype, the components were soldered on protoboards that were connected with conductive thread. The third prototype used a homemade PCB connected with wires.}
\label{fig:wristbandprototypes}
\end{figure}
-The take-away is that prototyping interactive devices requires both keeping in mind the design rationale and the technical constraints of their implementation.
-It is critical to balance the trade-offs and making informed compromises.
-For example replacing conductive thread with wires was not a compromise afterall, and make the prototype more robust.
+The takeaway is that prototyping interactive devices requires both keeping in mind the design rationale and the technical constraints of their implementation.
+It is critical to balance the trade-offs and make informed compromises.
+For example, replacing conductive thread with wires was not a compromise after all, and make the prototype more robust.
It was the consequence of the bad choice of microcontroller board from the beginning.
The restrictions on signal shape and the common frequency were actual compromises.
But they enabled a simple and robust design while keeping an expressive output vocabulary.
Therefore, workarounds are necessary even for basic interactions such as target activation.
Kinect applications typically use a dwell cursor, that requires users to remain still over buttons for a few seconds to activate them.
We replicated these cursors as depicted on \ref{fig:dwellcursor}.
-In Kinect applications, both the immediate feedback telling users they hover a button and the dwell progress are indicated with visual feedback.
+In Kinect applications, both the immediate feedback telling users they hover a button and the dwelling progress is indicated with visual feedback.
At the time, we assumed that complementing this feedback with haptic feedback would be beneficial for users. Therefore we implemented the tactile dwell cursor the following way.
\input{figures/dwellpointing.tex}
After this animation all the actuators vibrate at \qty{250}{\hertz} for \qty{200}{\ms}, then the target is activated.
Therefore, overall users had to hover a button during \qty{3}{\s} to activate it.
-We ran an experiment with the idea to measure an increase of performance in a tactile condition over a visual condition.
+We ran an experiment with the idea to measure an increase in performance in a tactile condition over a visual condition.
This was motivated by the fact that when we tried the buttons, tactile feedback seemed to bring some benefit.
-Participants were presented a screen with an array of four by four buttons.
+We presented to participants a screen with an array of four-by-four buttons.
They had to select a series of buttons indicated with a highlight.
The details of the experiment do not matter.
We failed at detecting a significant difference between the conditions, either in selection time or error rate.
%That being said, we were not the only ones to have this sensation that something was better with tactile feedback.
%Several participants reported that tactile feedback was a good addition to visual feedback, as it reduced their visual attention.
That being said, several participants reported that tactile feedback was a good addition to visual feedback, as it reduced their visual attention.
-For example, two participants said: “without tactile feedback I have to focus more on the visual feedback” and “I found [tactile feedback] more helpful than the visual feedback because I didn't have to focus 100\% visually with the tactile redundancy.”
+For example, two participants said: “without tactile feedback, I have to focus more on the visual feedback” and “I found [tactile feedback] more helpful than the visual feedback because I didn't have to focus 100\% visually with the tactile redundancy.”
Users also appreciated that tactile feedback indicated when they hovered a button.
For example, one of the participants said: “It felt more like interacting with a physical button when activating the button produced a sharp buzz.”
Therefore, the benefits of the tactile feedback seemed to be qualitative rather than quantitative.
The dwell buttons scenario did not seem to be the most appropriate for evaluating qualitative benefits.
Another scenario for haptic 3D gestural interaction in this project was an open-source car racing game\footurl{https://sourceforge.net/projects/vdrift/} that I adapted for Kinect (\reffig{fig:cargame}).
-The users steered the car by moving their arms like if they were holding a steering wheel.
+The users steered the car by moving their arms as if they were holding a steering wheel.
The Kinect API computed a skeleton of the user.
The steering angle was computed as a function of the relative position of the hands of the skeleton, capped between \ang{-90} and \ang{90}.
Braking was mapped to \qtyrange{15}{30}{cm} between the hands and the chest, and throttle to \qtyrange{30}{50}{cm}.
The spatial location of vibrations indicated the steering angle as shown on \reffig{fig:cargame}.
The speed of the car was mapped to a modulation of the \qty{250}{\hertz} signal with a low-frequency signal between \qty{1}{\hertz} and \qty{25}{\hertz} with \qty{50}{\ms} durations.
-This modulation makes users feel like if equidistant strips covered the road.
-%the faster the car goes, the frequent are the vibrations.
+This modulation makes users feel as if equidistant strips covered the road.
+%the faster the car goes, the more frequent the vibrations.
In addition to this, the bottom actuator vibrated for \qty{200}{\ms} at \qty{100}{\hertz} when the car was braking.
\input{figures/cargame.tex}
\newcommand{\NoTactile}{\textsc{NT}\xspace}
\newcommand{\Tactile}{\textsc{T}\xspace}
-In a first pilot study we compared performance between the tactile feedback and no tactile feedback conditions.
-In hindsight there was little chances haptic sensations increased such metrics.
+In a first pilot study, we compared performance between the tactile feedback and no tactile feedback conditions.
+In hindsight, there were few chances haptic sensations increased such metrics.
However, participants reported that the tactile sensations provided them benefits in terms of realism and immersion in the game.
Therefore we conducted the following experiment, which focused on qualitative benefits.
%Initially, I performed a quantitative evaluation to compare the lap times between the tactile feedback and no tactile feedback conditions.
% Therefore I conducted the following experiment, which focused on qualitative benefits.
%In the next submission we performed a new user study and measured emotions with the PAD questionnaire (Pleasure Arousal Dominance)~\cite{mehrabian96} and presence with Witmer \etal PQ questionnaire \cite{witmer98}.
%\paragraph{Hypotheses}
-We made the hypotheses that $H_1$ tactile feedback would increase the sensation of realism of the game because of the increased sensory stimulation; and $H_2$ tactile feedback would increase the users' sensation of control, because it provides them feedback about the way the game interpret their actions.
+We made the hypotheses that $H_1$ tactile feedback would increase the sensation of realism of the game because of the increased sensory stimulation, and $H_2$ tactile feedback would increase the users' sensation of control because it provides them feedback about the way the game interprets their actions.
\subsubsection{Methodology}
-40 participants took part of the experiment (mean age 24.7 years).
+40 participants took part in the experiment (mean age \num{24.7} years).
They were instructed to drive as many laps as possible in 10 minutes.
They were advised to avoid going out of the road since it notably reduces the speed.
They stood \qty{2.5}{\meter} away from the Kinect during the game.
%Before the experiment the height of the Kinect was calibrated to make sure the participants' arms were in the sensor’s range.
-The game was displayed on a 17” laptop screen, with a $1600 \times 900$ resolution and the same sound volume was used for all subjects.
-We opted for a between subject design: half of the participants received tactile feedback (\Tactile), and the other half did not (\NoTactile).
+The game was displayed on a 17” laptop screen, with a $1600 \times 900$ resolution, and the same sound volume was used for all subjects.
+We opted for a between-subjects design: half of the participants received tactile feedback (\Tactile), and the other half did not (\NoTactile).
We explained the mapping of the tactile feedback to the participants of condition \Tactile.
Tactile feedback was not mentioned to the participants of condition \NoTactile.
Before they performed the task, the participants filled the immersion tendency questionnaire (ITQ)~\cite{witmer98}.
-It measures the ability to get involved and focused in tasks, playing habits and the tendency to get immersed in games.
+It measures the ability to get involved and focused on tasks, playing habits, and the tendency to get immersed in games.
%This questionnaire identifies three factors among these items: Involvement (ability/habits to get involved in tasks), Focus (ability to stay focused on a task) and Games (playing habits and tendency to be immersed in games).
%After the experiment we measured the user’s emotional state with the Pleasure Arousal Dominance questionnaire (PAD)~\cite{mehrabian96}.
%, which uses 7 points bipolar scales.
%We also measured spatial presence with the presence questionnaire (PQ)~[32].
-After the experiment we measured spatial presence with the presence questionnaire (PQ)~[32].
+After the experiment, we measured spatial presence with the presence questionnaire (PQ)~[32].
It measures the sensation of control, perception of sensations, distraction from the task, and realism.
%This questionnaire identifies four factors among these items: Control (sensation that user’s actions have conse-quences on the virtual environment), Sensory (how much/well the user’s senses are stimulated), Distraction (how easy the user was concentrated on the task) and Real-ism (how realistic was the simulation).
We added three additional questions about tactile feedback adapted from questions about audio feedback: 1) How much did the tactile aspects of the environment involve you? 2) How well could you identify the vibrations? 3) How well could you localize vibrations?
-We analyzed our results with T-tests, Pearson correlation and Cronbach’s alpha.
+We analyzed our results with T-tests, Pearson correlation, and Cronbach’s alpha.
\subsubsection{Results}
\reffig{fig:carpresence} shows the results of the ITQ and PQ questionnaires.
-The reliability of the ITQ questionaire is acceptable (\cralpha{0.75}).
-Participants had a mean immersion tendency of $116.05/189$ in the \Tactile condition, and $117.5/189$ in the \NoTactile condition.
+The reliability of the ITQ questionnaire is acceptable (\cralpha{0.75}).
+Participants had a mean immersion tendency of $116.05/189$ in the \Tactile condition and $117.5/189$ in the \NoTactile condition.
We did not detect any significant effect (\pEq{0.72}).
The PQ questionnaire had a good reliability in both \Tactile (\cralpha{0.89}) and \NoTactile (\cralpha{0.88}) conditions.
Regarding our research question, we first hypothesized that tactile feedback would enhance the realism of the game ($H_1$).
The results of the \emph{Realism} items of the \emph{Presence Questionnaire} support this hypothesis.
-In addition to this, five participants of the \NoTactile condition explicitly reported they did not have a sensation of speed while several participants of the \Tactile condition spontaneously mentioned feeling speed.
+In addition to this, five participants of the \NoTactile condition explicitly reported they did not have a sensation of speed while several participants of the \Tactile condition spontaneously mentioned feeling the speed.
Our second hypothesis ($H_2$) predicted that tactile feedback would enhance the user’s sensation of \emph{Control}.
The results do not support this hypothesis, therefore we cannot make any conclusion regarding this hypothesis.
However, participants indeed reported control issues.
-They had difficulties to brake because they had to move their arms very close to their chest, and it was difficult to turn at the same time.
+They had difficulties braking because they had to move their arms very close to their chest, and it was difficult to turn at the same time.
Indeed this issue was happening in both conditions and tactile feedback did not help.
%The other explanation is that the tactile feedback mapped to the direction intended to help the user to feel if the car reacted to her gestures. However we faced an interesting problem with the mapping we used. While users admitted they understood the speed information easily, they had difficulties to understand the orientation information and ignored it. When we explained this mapping to the user before the experiment, four users asked if this mapping was absolute or relative to the orientation of the hand. Our choice was to vibrate on the top side of the arm when the car was going straight, and the left or right side depending on the direction of the car (Figure 7). However usually when the user holds the imaginary wheel, the top side of the right arm faces the right of the user, and changes when the user turn the wheel. While we have no measure of this effect, this observation raises the importance of designing carefully the feedback mapped to the gestures performed by the user. More generally it is not clear if previous results on tactile cues [4,8,14,15,26] apply when the user focuses his attention to a complex task like playing a game. It led us to start a whole new study about this question, as a follow up to this work.
\subsection{Discussion}
The paradox here is that the motivation for 3D gestural interaction was to offer users \emph{natural} ways to interact with systems, without manipulating artificial artifacts.
-Whether if it was relevant or not, without haptic sensations the virtual objects manipulated do not feel real to users, and they sometimes struggle to to manipulate them.
+Whether it was relevant or not, without haptic sensations the virtual objects manipulated do not feel real to users, and they sometimes struggle to manipulate them.
Restoring haptic feedback on dwell buttons was not sufficient to make them efficient.
My hypothesis was that restoring tactile feedback would help users press them.
I anticipated that this would increase selection performance.
-However, with the way I emplemented these buttons activation required at least \qty{3}{\s}.
+However, with the way I implemented these buttons activation required at least \qty{3}{\s}.
This is very long indeed.
-Should tactile feedback help selecting buttons faster, the effect size could not be sufficient to make a significant improvement.
+Should tactile feedback help select buttons faster, the effect size could not be sufficient to make a significant improvement.
The average selection time in the \NoTactile condition was \qty{3.6}{\s}, and it was \qty{3.5}{\s} in the \Tactile.
-This is a first clue that improving haptic feedback is not sufficient.
-It requires an efficient input method as well, otherwise the eventual benefits of haptics do not compensate the inefficiency of input.
+This is the first clue that improving haptic feedback is not sufficient.
+It requires an efficient input method as well.
+Otherwise, the eventual benefits of haptics do not compensate for the inefficiency of input.
In \refsec{sec:hapticparadigms} I describe a new paradigm for 3D gestural interaction with haptic feedback, and the systematic design of both the input and output vocabulary to enable direct manipulation on a tactile display.
% > summary(time$DwellTime[time$Condition == "NoTactile"])
The qualitative study with the car racing scenario showed a similar issue.
Tactile feedback did provide qualitative benefits.
-However the low quality of inputs diluted these benefits and made them barely measurable.
+However, the low quality of inputs diluted these benefits and made them barely measurable.
The haptic feedback provided was questionable as well.
-I used spatial location around the wrist, however the task required participants to rotate their wrist.
-Therefore it might have influenced negatively on how participants interpreted the tactile cues.
-Unfortunately I did not investigate this issue at the time.
-Moreover the haptic cues inducated the steering angle of the wheel, not the car.
-The rationale is that it provides users immediate feedback of their actions, which is lacking without a physical controller.
-However it does not provide haptic feedback about the result of the action.
-Regrettably, I did not investigate this aspect neither.
-On the positive side I did measure an effect of tactile feedback on presence.
+I used spatial location around the wrist, however, the task required participants to rotate their wrists.
+Therefore it might have influenced negatively how participants interpreted the tactile cues.
+Unfortunately, I did not investigate this issue at the time.
+Moreover, the haptic cues indicated the steering angle of the wheel, not the car.
+The rationale is that it provides users immediate feedback on their actions, which is lacking without a physical controller.
+However, it does not provide haptic feedback about the result of the action.
+Regrettably, I did not investigate this aspect either.
+On the positive side, I did observe an effect of tactile feedback on presence.
This is an interesting result for the Virtual Reality community, and I regret this result was not published in the end, hence the discussion here.
However, later we investigated a related question with the effect of haptic feedback on the sense of embodiment, which we will discuss in \refsec{sec:embodiment}.
\section{Computing and the sensorimotor loop}
-In the previous chapters we discussed several examples of how haptics as the sense of touch on one side, and haptics as the motor ability on the other side provide useful interactive properties.
-In the previous section we discussed the fact that the benefits of one of them cannot necessarily compensate the issues of the other.
+In the previous chapters, we discussed several examples of how haptics as the sense of touch on one side, and haptics as the motor ability on the other side provide useful interactive properties.
+In the previous section, we discussed the fact that the benefits of one of them cannot necessarily compensate for the issues of the other.
This observation is however only the tip of the iceberg.
-It is one of the many clues showing that inputs and outputs cannot be separated, the same way that our senses and abilities cannot be separated neither.
+It is one of the many clues showing that inputs and outputs cannot be separated, the same way that our senses and abilities cannot be separated either.
-One of the major difference between humans and systems is that humans are as they are, we cannot fundamentally change the way they function.
+One of the major differences between humans and systems is that humans are as they are, we cannot fundamentally change the way they function.
Systems are different: we build them, therefore we can design and build them according to our needs.
We have no reason to build a machine that has no purpose for us.
We discuss below how the literature modeled the way humans work, and models for designing systems.
-Interestingly, both work in a similar way, with input, processing and output.
+Interestingly, both work in a similar way, with input, processing, and output.
The question is: is it like this because of an anthropocentric bias or because this is an operational optimum?
I will discuss this question with my vision of how interactive systems work and the critical role of the sensorimotor loop.
%The strongest roots of the understanding of human behavior that had the most significance on HCI research, at least related to the topic of this manuscript, is Gibson's work.
%Perception/action cycle~\cite{gibson79}
%Animals such as humans explore their environment by probing them, and perceiving properties. Perception is a combination of actions, resulting sensations, and cognitions that mixes this with memory, experience, etc.
-The understanding on human bahavior, and in particular the way humans perceive the world and act on it is the foundation of HCI.
+The understanding of human behavior, and in particular the way humans perceive the world and act on it is the foundation of HCI.
Gibson's work on this topic was pivotal to the development of the HCI community.
He studied for decades how we, as animals, perceive our environment~\cite{gibson79}.
His work mainly focused on visual perception, but he also studied other senses like the sense of touch~\cite{gibson62}.
-The idea is essentially that the perception we have of environment is not just based on a bunch of input stream from our senses.
+The idea is that the perception we have of the environment is not just based on a bunch of input streams from our senses.
It is the integration of these input streams with the exploratory movement that we made, with our memories and experience.
-This is because the input streams depend on the exploratory movements, and we make these exploratory movement to seek for information and match it with our previous knowledge to shape our perception.
+This is because the input streams depend on the exploratory movements, and we make these exploratory movements to seek information and match it with our previous knowledge to shape our perception.
The nature of the movement itself, that Lederman and Klatzky call exploratory procedures~\cite{lederman87,lederman96}, enables people to sense different properties of objects.
-For exemple with a lateral motion we can feel the texture of an object, and with a pressure we can feel its hardness.
-When we enclose an object with our hands we cans sense its volume and global shape, but if we follow its contours with our fingers we can not only sense its global but also exact shape.
+For example, we can feel the texture of an object with a lateral motion, and we can feel its hardness with a pressure.
+When we enclose an object with our hands we can sense its volume and global shape, but if we follow its contours with our fingers we can not only sense its global but also exact shape.
This intertwined relation of sensations and active exploration in our perception enables outstanding paradigms.
-For example sensory substitution consists in translating information that is typically sensed with one sense to another sense~\cite{bachyrita72}.
-Several sensory substitution devices use the sense of touch with active exploration gestures to replace vision \cite{collins73,linvill66,bliss70,gapenne03}.
+For example, sensory substitution consists in translating information that is typically sensed with one sense to another sense~\cite{bachyrita72}.
+Several sensory substitution devices use the sense of touch with active exploration gestures to replace vision \cite{collins73,linvill66,bliss70,gapenne03}.
They typically enable blind people to read printed books, or scan their environment with a haptic white cane.
Without active exploration, our brain cannot process such a complex tactile input stream.
\paragraph{Affordance}
-Gibson's theory involves a close relation between the animal (humans in our case) and its environment.
+Gibson's theory involves a close relationship between the animal (humans in our case) and its environment.
Gibson makes a difference between the physical world and the environment.
-The physical world refers everything between the smallest particles to the biggest possible objects like galaxies.
+The physical world refers to everything from the smallest particles to the biggest possible objects like galaxies.
The environment refers to what is reachable for the animal, in particular in terms of size.
This study on an animal in its environment is what Gibson calls the \defword{ecological approach of perception}.
It makes it possible to study the relations between animals and their environment.
In particular, specific combinations of properties of an animal and objects of its environment enable the animal to perform particular actions on the object.
Gibson called these relations \defword{affordance}~\cite{gibson77}.
A handle of a given size and a hand of a similar \emph{affords} grasping.
-A flat horizontal surface of the size of the bassin \emph{affords} affords sitting.
+A flat horizontal surface of the size of the pelvis \emph{affords} sitting.
These notions were not initially studied to improve the design of interactive systems.
However, HCI researchers and interaction designers applied these principles for the study and design of interactive systems.
In particular, Norman's work has a strong influence on the HCI community.
-His theory of action depicted on \reffig{fig:sevenstages} is an operational view on Gibson's ecological approach of perception~\cite{norman88}.
+His theory of action depicted on \reffig{fig:sevenstages} is an operational view of Gibson's ecological approach to perception~\cite{norman88}.
It describes seven stages to describe how people interact with objects in their environment, based on their \defword{mental model} how this object works.
-There is a central stage representing the goal the person would like to reach, and three stages for the execution of actions and the evaluation of the changes on the world.
+There is a central stage representing the goal the person would like to reach, and three stages for the execution of actions and the evaluation of the changes in the world.
Norman uses this model to explain the many usability issues that can arise when this mental model differs from the \defword{conceptual model} of how this object actually works.
\input{figures/sevenstages.tex}
-Norman also participated to the introduction of the concept of affordance to the HCI community.
-Contrary to objects found in the nature, human-made objects can leverage our knowledge about human characteristics.
-We can reduce the gaps between mental model and conceptual model by creating affordances.
+Norman also participated in the introduction of the concept of affordance to the HCI community.
+Contrary to objects found in nature, human-made objects can leverage our knowledge about human characteristics.
+We can reduce the gaps between the mental model and conceptual model by creating affordances.
Norman gives the example of doors designed in a way we know which side to pull and which side to push~\cite{norman88}.
-The pull side has a handle that affords grasping and pulling whereas the push side has a flat plate that affords pushing.
-However we cannot control all the physical properties of all the objects around us.
+The pull side has a handle that affords to grasp and pull whereas the push side has a flat plate that affords pushing.
+However, we cannot control all the physical properties of all the objects around us.
Therefore there can be a difference between the actual affordances and the affordances we perceive.
Gaver describes four possibilities~\cite{gaver91}.
-Two of them are desired: a perceived affordance and a true reject (there is no affordance, an no affordance is perceived).
-He also describes hidden affordances, and false affordances.
-Because of this, Norman makes a distinction between an affordance, as a property, and a \defword{signifier} which is a perceivable properties that advertises the existance of an affordance~\cite{norman02}.
+Two of them are desired: a perceived affordance and a true reject (there is no affordance, and no affordance is perceived).
+He also describes hidden affordances and false affordances.
+Because of this, Norman makes a distinction between an affordance, as a property, and a \defword{signifier} which is a perceivable property that advertises the existence of an affordance~\cite{norman02}.
%Perception/action cycle~\cite{gibson79}
%Sensorimotor loop~\cite{oregan01a}
%It is also used in other contexts like surgery, in which vision is required for a primary task, and haptics is used to replace vision at a different scale and point of view~\cite{robineau07}.
-There are other practical models in HCI literature that guide the design of interaction techniques and interactive devices.
+Other practical models in HCI literature guide the design of interaction techniques and interactive devices.
One of the most universal mathematical models in interaction is certainly \defword{Fitts' law} that measures the difficulty ($ID$) of a target selection movement $ID=\log_2\left(\frac{2D}{W}\right)$~\cite{fitts54}.
The model says that the longer the distance and the smallest the target width, the higher the difficulty.
With the original experiment protocol, participants perform a reciprocal left-to-right movement between two targets of width $W$ and distance $D$ (also called movement amplitude).
\defacronym{GOMS} models are examples of human behavior model that takes into account both perception and actions.
They model humans with three components: a perceptual system, a motor system, and a cognitive system~\cite{card83}.
These are therefore more generic than models such as Fitts', they model a greater diversity of behavior.
-In particular the Keystroke-Level Model (\defacronym{KLM}) defines several types of perations from mental activities to key presses~\cite{card80}.
-Typically it leverage models such as Fitts' to quantify pointing operations.
-With these models we can describe interaction techniques as sequences of atomic operations, and predict average performance based on empirically-defined rules.
+In particular, the Keystroke-Level Model (\defacronym{KLM}) defines several types of operations from mental activities to key presses~\cite{card80}.
+Typically it leverages models such as Fitts' to quantify pointing operations.
+With these models, we can describe interaction techniques as sequences of atomic operations, and predict average performance based on empirically-defined rules.
The models we discussed take into account human behavior, and to some extent the way interaction techniques work, but not necessarily how they are implemented.
-For example they do not take into account the transfer function between the pointing device and the cursor.
-They do not necessarily take into account the integration or separation of degrees of freedom~\cite{mackinlay90}, or the type of feedforward~\cite{vermeulen13}.
-In my opinion thus is a limitation for the generative aspect of these models, and I believe we must include more knowledge about the implementation into interaction models.
+For example, they do not take into account the transfer function between the pointing device and the cursor.
+They do not necessarily take into account the integration or separation of degrees of freedom~\cite{mackinlay90} or the type of feedforward~\cite{vermeulen13}.
+In my opinion, this is a limitation of the generative aspect of these models, and I believe we must include more knowledge about the implementation into interaction models.
%Human processor: ~ : KLM
Studying human behavior is useful for the design of machines for several reasons.
The first reason is to reproduce the strengths of humans.
-For example in the previous section we discussed the fundamental coupling between humans' perception and action.
+For example, in the previous section, we discussed the fundamental coupling between humans' perception and action.
Systems called \defwords{closed-loop systems}{closed-loop system}\footurl{https://en.wikipedia.org/wiki/Control_system} also leverage such a mechanism.
-For example the non-inverting and inverting amplifiers circuit depicted on \reffig{fig:amplifiers} have the output of their operational amplifier connected to one of their input through a resistor.
+For example, the non-inverting and inverting amplifiers circuit depicted on \reffig{fig:amplifiers} have the output of their operational amplifier connected to one of their input through a resistor.
The output voltage is proportional to the input voltage whose value depends on $R_1$ and $R_2$.
But most importantly, the feedback loop stabilizes the output voltage to the desired value.
This kind of control mechanism is used in many applications such as robots, domestic appliances, or drones.
It is also used in haptic devices that leverage information from people with \defword{Human-in-the-loop} models~\cite{vanderlinde02}.
-This paradigm focus on system control, and the human only exist as a parameter of the equation.
+This paradigm focuses on system control.
+The human only exists as a parameter of the equation.
Therefore despite the similarities with the way we describe human behavior, this is not our focus.
\input{figures/amplifiers.tex}
%\todo{Interest in human behavior for the design of interactive systems: 1) take inspiration of ot and reproduce a similar behavior. 2) make a better connection between humans and systems}
-The second reason to study human behavior is to improve interaction between humans and machines.
-Humans and interactive systems are distinct entities that need each other and must communicate to achieve their objective.
+The second reason to study human behavior is to improve the interaction between humans and machines.
+Humans and interactive systems are distinct entities that need each other and must communicate to achieve their objectives.
Hence we will discuss below the architecture of interactive systems, the similarities and differences with humans, and how this is critical for improving interactions.
\paragraph{Computation models}
Initially, computers and programs were essentially based on theoretical models such as \defword{$\lambda$-calculus}~\cite{church32} or \defwords{Turing machines}{turing machine}~\cite{turing38}.
-These are computing models, and they focus on solving numerical problem rather than helping people with their everyday activities.
+These are computing models, and they focus on solving numerical problems rather than helping people with their everyday activities.
A Turing machine has an infinite tape with symbols written in advance, and a pre-defined transition table that describes the behavior of the machine.
-Therefore these machines ignore their environment, in which anything can change at anytime.
+Therefore these machines ignore their environment, in which anything can change at any time.
All these models are equivalent (or Turing-equivalent), and the \defword{Church-Turing thesis} says that everything these models can compute can be implemented with an algorithm.
Wegner and Goldin explain that both this universality and this limitation are due to their inductive nature~\cite{wegner99}.
%\paragraph{Induction and co-induction}
-Induction require structures to be finite, and computation to end.
-For example the Listing~\ref{lst:induction} shows the inductive definition of a list of numbers, and a function that computes the length of a list.
-A list is buit with two constructors: either a \verb+Nil+ value for an empty list, or a \verb+Cons+ function that create a list with a number (the head) and another list (the tail).
+Induction requires structures to be finite, and computation to end.
+For example, the Listing~\ref{lst:induction} shows the inductive definition of a list of numbers and a function that computes the length of a list.
+A list is built with two constructors: either a \verb+Nil+ value for an empty list or a \verb+Cons+ function that create a list with a number (the head) and another list (the tail).
The \verb+Nil+ value ensures that the list is finite.
-It ensures in turn that the \verb+length+ function ends, because there is no recursive call on the base case (\verb+Nil+) and every list ends with Nil.
+It ensures in turn that the \verb+length+ function ends because there is no recursive call on the base case (\verb+Nil+) and every list ends with Nil.
\begin{code}[language=Coq, label=lst:induction, caption=Inductive list and example of inductive function on a list.]
Inductive list : Set :=
All information about the problem must be known in advance, the computing process is precisely defined, and the output is specified by the inputs~\cite{knuth68}.
Wegner and Goldin describe interaction as a more general model in which the machine is connected to input streams, that provide unpredictable data~\cite{goldin08}.
They model interaction with co-induction as in the example in Listing~\ref{lst:coinduction}.
-This example defines a stream of numbers, which is an non-finite structure, and a function that returns another stream in which numbers from the input stream are multplied by $2$.
+This example defines a stream of numbers, which is a non-finite structure, and a function that returns another stream in which numbers from the input stream are multiplied by \num{2}.
The co-inductive definition of the stream only has one constructor, identical to the second constructor of lists.
-The absence of a base constructor make streams infinite structures.
-Therefore recursive function on streams potentially run forever.
+The absence of a base constructor makes streams infinite structures.
+Therefore recursive functions on streams potentially run forever.
Hence, there is no equivalent of a \verb+length+ function because it would never return a value.
However, functions such as in this example still make sense.
They operate iteratively rather than globally, therefore they can operate on infinite structures.
end.
\end{code}
-We observe here that each iteration of the co-fixpoint can be inductive, as it is the case in the example.
-It shows that interaction is a general process that connects entities in the environment to enable them exchanging information.
+We observe here that each iteration of the co-fixpoint can be inductive, as in the example.
+It shows that interaction is a general process that connects entities in the environment to enable them to exchange information.
Algorithms only process information to transform known input into outputs without knowledge of the overall scheme, and no external event can change their behavior during their execution.
-Interactive systems we use everyday react to unpredictable inputs in real time.
+Interactive systems we use every day react to unpredictable inputs in real-time.
%They leverage algorithms, but the time constraints are critical.
Therefore they are not just built with algorithms.
%The input streams are unpredictable, and their behavior adapts to
They have separated input and output loops at different levels that communicate through streams.
-For example, on the system level an input loop gets input streams of input data (\eg mouse displacements), produce output streams of input events (\eg mouse move event).
-On the application level, an input loops gets output streams of input events and combine the information they convey with interaction techniques to produce an output stream of actions to be executed.
+For example, on the system level, an input loop gets input streams of input data (\eg mouse displacements) and produces output streams of input events (\eg mouse move event).
+On the application level, an input loop gets output streams of input events and combines the information they convey with interaction techniques to produce an output stream of actions to be executed.
On the application level, a graphics loop gets an input stream of graphic commands and produces an output stream of objects to be displayed.
Therefore, applications are therefore what Wegner calls \defwords{interaction machines}{interaction machine}~\cite{wegner97}.
%\defwords{Neural networks}{neural network} are other examples of interaction machines: they also get input streams and produce output streams~\cite{mcculloch43}.
\paragraph{Software architectures and interaction paradigms}
Software architectures leverage this interaction machine to describe a higher-level structure that connects users to a functional \emph{model}.
-This model, also called an \emph{abstraction}, defines the objects of the system, their properties and the operations on them.
-For example, the original \defacronym{MVC} architectures distinguishes the model with \emph{views} that describe how objects are presented to users and \emph{controllers} that define the way users can manipulate them~\cite{reenskaug79,reenskaug79a}.
+This model, also called an \emph{abstraction}, defines the objects of the system, their properties, and the operations on them.
+For example, the original \defacronym{MVC} architectures distinguish the model with \emph{views} that describe how objects are presented to users and \emph{controllers} that define the way users can manipulate them~\cite{reenskaug79,reenskaug79a}.
\defword{Arch}~\cite{arch92} and \defacronym{PAC}~\cite{coutaz87} rather combine input and outputs as a \emph{presentation} component, and add a \emph{controler} component that manages transitions between abstract inputs/outputs and domain-specific properties of the model/abstraction.
%The modern MVC architectures follow this structure as well.
The advantage of these architectures is to separate the objects of interest from the interaction with them.
-It is therefore easy to display several synchronized representations of the same object, and provide multiple ways to manipulate them.
+It is therefore easy to display several synchronized representations of the same object and provide multiple ways to manipulate them.
These interactive properties contribute to leveraging human capacities and flexibility through \defword{multimodality}~\cite{nigay95,nigay04}.
%Seeheim \cite{green85}
-In the previous section, we discussed the critical role of the sensorimotor cycle on human's perception.
-% and Gibson's ecological approach of perception.
+In the previous section, we discussed the critical role of the sensorimotor cycle on human perception.
+% and Gibson's ecological approach to perception.
The software architectures above create connections between humans and interactive systems with input and output streams.
However, leveraging the full potential of the sensorimotor loop requires an additional layer of interaction paradigm such as \defword{direct manipulation}~\cite{schneiderman83}.
The direct manipulation paradigm defines several properties: objects have to be visible and directly manipulable, and actions have to be fast, reversible, and incremental.
Implementing these properties into the design of graphical user interfaces (\defacronym{GUI}) favors their usability.
For example it contributes to some of Nielsen's heuristics~\cite{nielsen90,nielsen94}: \emph{visibility of system status}, \emph{user control and freedom}, \emph{recognition rather than recall}, and \emph{Flexibility and efficiency of use}.
The \defword{instrumental interaction} paradigm extends direct manipulation and makes the connection with software achitectures~\cite{mbl00}.
-Similarly to software architectures, it defines domain objects.
+Similar to software architectures, it defines domain objects.
Users can interact with them through \emph{interaction instruments}, which are reifications of commands.
When users perform actions on the instruments, they receive immediate feedback and the instrument performs operations on the domain objects.
The domain object returns a response to this operation.
People perceive and act on their environment thanks to their sensorimotor loop.
Therefore our perception depends both on our sensorial and motor capabilities.
They have limitations that influence the way we perceive our environment.
-There are \emph{type} limitations, for example we can perceive light waves but not magnetic fields.
-There are \emph{range} limitations, for example we hear a sound wave of \qty{400}{\hertz}, but not \qty{400}{\kilo\hertz}.
+There are \emph{type} limitations.
+For example, we can perceive light waves but not magnetic fields.
+There are \emph{range} limitations.
+For example, we hear a sound wave of \qty{400}{\hertz}, but not \qty{400}{\kilo\hertz}.
We can reach objects \qty{50}{\cm} away from us, but not \qty{50}{m}.
There is also a \emph{precision} limitation, for example we can distinguish colors of wavelength \qty{100}{\nano\metre} apart, but not \qty{1}{\nano\metre} apart.
-Precision is actually a range in difference, therefore according to Weber's law the threshold is proportional to the base stimulus value~\cite{fechner60}.
+Precision is actually a range of difference, therefore according to Weber's law, the threshold is proportional to the base stimulus value~\cite{fechner60}.
Finally, there are \emph{processing} limitations, related to our cognitive abilities to interpret signals resulting from our perceptions and actions.
Typically, illusions are distortions of what we could consider as ground truth.
Interactive systems have the same kind of limitations.
-They are limited by the inputs and outputs they receive, and computing capactities to interpret them.
+They are limited by the inputs and outputs they receive, and the computing capabilities to interpret them.
Here inputs and outputs do not necessarily refer to bit streams, but more generally streams of physical phenomenons of their environment they can sense or produce (\eg movements, sounds, light).
%Enaction\cite{varela92,thompson10}
Extending human capacities with computers marked the beginning of Human-Computer Interaction, with the pioneer visions of Bush~\cite{bush45} and Engelbart~\cite{engelbart68}.
On the other side, all machines need humans otherwise they have no purpose.
They all need instructions and data, and they all modify the environment or produce information.
-These interactions between entities, whether they are humas or machines, require communication.
+These interactions between entities, whether they are humans or machines, require communication.
What I mean by communication is to produce a physical effect on a shared environment that the other entity can perceive.
This shared environment has a common space and time, at a similar scale.
-The characteristics of this communications depend on the skills and capabilities of both entities.
-For example the most efficient communication between two humans is certainly speech.
-This is maybe the reason why there is a push towards vocal interfaces.
+The characteristics of these communications depend on the skills and capabilities of both entities.
+For example, the most efficient communication between two humans is certainly speech.
+This is maybe the reason why there is a push toward vocal interfaces.
However, so far technologies struggle with the interpretation of natural languages because of their complexity.
Vocal interfaces only recognize a limited and pre-defined set of instructions.
-But in addition to speech we make gestures with the hands, face or other body part that can totally change the meaning.
+But in addition to speech, we make gestures with the hands, face, or other body parts that can totally change the meaning.
As we discussed in the previous chapter, systems are efficient at sensing gestures, and there is still room for improvement.
Therefore this is today the main communication modality between humans and systems.
\paragraph{Seven stages of reaction}
-In \refsec{sec:humanbehavior} we discussed how people perceive their environment, and in particular interactive systems.
+In \refsec{sec:humanbehavior} we discussed how people perceive their environment and in particular interactive systems.
We presented how Norman's theory of action (see \reffig{fig:sevenstages}) explains the difference between the conceptual model of the system, and the perceptual model the users have of it based on their perception.
I suggest that interactive systems follow a similar perceptual scheme, as depicted on \reffig{fig:mysevenstages}.
% The system senses a physical effect in its environment.
% The system combines these events into input phrases with interaction techniques.
% After this step,
The input chain begins with the \emph{sensing} stage.
-Physical sensors measure physical effect in the environment.
+Physical sensors measure physical effects in the environment.
Typically they measure the user movements, but it can be various other information such as light, temperature, moisture, or vibrations.
All such information is transformed into \emph{input events}.
-At this stage we notice that the infinite richness of the world is reduced to a small number of bits.
+At this stage, we notice that the infinite richness of the world is reduced to a small number of bits.
%Let's discuss the simple example of a keypress on a keyboard.
%The only information in the digital world is whether a key is pressed or not.
-For example when a user presses a key on a keyboard, the interactive system only senses whether the key is pressed or not.
-There is usually no information about the finger that pressed it, the speed of the finger or its trajectory.
+For example, when a user presses a key on a keyboard, the interactive system only senses whether the key is pressed or not.
+There is usually no information about the finger that pressed it, the speed of the finger, or its trajectory.
%, even if it is possible \cite{marquardt11,goguey14,goguey17}.
%We usually neither know if a finger is hovering the key, if several fingers are pressing it, or even if it was pressed with the nose.
%And this is also possible to sense it and improve interaction~\cite{rekimoto03,harrison11}.
% Quantic information : stored in computer memory. Will it be observed?
%Bringing a piece of information to the physical world requires several steps.
First of all the systems must \emph{encode} the pieces of information.
-A visual encoding can be an icon, a text.
+A visual encoding can be an icon or a text.
Audio encodings can be sounds~\cite{gaver93} or melodies~\cite{brewster93}.
Various haptic encodings include vibrations~\cite{brewster04}, forces~\cite{maclean03a,pietrzak05b,pietrzak05}, or textures~\cite{pietrzak09,pietrzak06,potier16}.
%Effectors can produce light (like screens), sounds, vibrations, forces, …
-Output devices have driving electronics which require specific \emph{commands}, and turn them into \emph{physical effects}.
+Output devices have driving electronics that require specific \emph{commands} and turn them into \emph{physical effects}.
These are typically lights (like screens), sounds, vibrations, and forces.
\input{figures/sevenstages2.tex}
The funnel of evaluation designates the fact that the input stages reduce the complexity of the system's environment into a few bits.
A good design of the input chain senses the right phenomenons, at an appropriate amplitude, with a sufficient spatial and temporal resolution, and with little distortions.
-These information must be combined correctly to form a meaningful sequence of actions that the users must perform.
+This information must be combined correctly to form a meaningful sequence of actions that the users must perform.
For example, the \defword{Midas touch} problem is a usual issue with 3D gestural interaction.
-Since the sensors observe the users movements without activation or interruption, there is no obvious segmentation.
+Since the sensors observe the users' movements without activation or interruption, there is no obvious segmentation.
The system has no way to know if the users move their hand to interact with the system, or for scratching their nose for example.
\defword{Occlusion} has the opposite issue.
%, occlusion is the other problem with vision-based gesture sensors.
\paragraph{Funnel of execution}
The funnel of execution is symmetrical to the funnel of evaluation.
-%The software part of the system keeps an model~\cite{reenskaug79a} (or abstraction~\cite{coutaz87}), and it has to display parts of it in an intelligible way for users.
-The way the objects of the internal model is shown to the users can have a huge impact on how they interact with it~\cite{zhang94}.
-Therefore, the encoding part is crucial, and it is a first filter for reducing the internal complexity of the system.
-The specifications of the output device is a second filter.
+%The software part of the system keeps n model~\cite{reenskaug79a} (or abstraction~\cite{coutaz87}), and it has to display parts of it in an intelligible way for users.
+The way the objects of the internal model are shown to the users can have a huge impact on how they interact with it~\cite{zhang94}.
+Therefore, the encoding part is crucial, and it is the first filter for reducing the internal complexity of the system.
+The specification of the output device is a second filter.
There is a limitation of physical effect (\eg force, color, brightness, frequency) each device can produce in practice.
There is also a limit of precision, that depends on electronics and mechanics.
Last, the physical effect can be inconsistent for the same command.
Norman's theory of action is typically used to describe differences between the user's perceptual model and the system's conceptual model.
The adaptation of this theory to systems describes a similar difference, but the conceptual and perceptual models are inverted.
-The systems' conceptual model of the user is based on the user's perceptual model of it that decribes the way the users interact with the sytem.
+The systems' conceptual model of the user is based on the user's perceptual model of it that describes the way the users interact with the system.
This is what Norman describes with his seven stages of action (\reffig{fig:sevenstages}).
-The systems' perceptual model of the user is based on its own conceptual model that describe its ability to interact with humans.
+The systems' perceptual model of the user is based on its own conceptual model that describes its ability to interact with humans.
This is what I describe with the seven stages of reaction above (\reffig{fig:mysevenstages}).
-Usability ussues occur when these behaviors cannot connect together.
+Usability issues occur when these behaviors cannot connect together.
There is a fundamental difference between human and system behavior though.
We design the system behavior thanks to our engineering skills and scientific knowledge.
Therefore we control it and we can adapt it to our needs.
We can add sensors and actuators, tune their sensitivity and operating range, and change their software.
-At the opposite, human behavior depends on cultural and social background, experiences.
+On the opposite, human behavior depends on cultural and social background, and experiences.
We can only shape it through training and education.
-Moreover, we cannot change the human body to add another sense, or adjust their sensitivity and range for example.
+Moreover, we cannot change the human body to add another sense or adjust its sensitivity and range for example.
This is why we need tools and instruments to extend our capacities beyond human capacities.
% Therefore, we can improve interaction between a user and a system by changing the user's behavior.
The \reffig{fig:all} summarizes the difference between the two approaches we discussed in this manuscript.
The outer circle shows the connection between the output path discussed in \refchap{chap:output} detailed on \reffig{fig:hapticpath}, and the input path described in \refchap{chap:input} detailed on \reffig{fig:motorpath}.
It is similar to Abowd and Beale's interaction framework~\cite{abowd91}.
-The advantage of this approach is that it describes better the connections between the user and the system an takes both into account.
-The inner loop shown the connection between the human and the system through Norman's seven stages of action (\reffig{fig:sevenstages}), and the seven stages of reaction I described above (\reffig{fig:mysevenstages}).
+The advantage of this approach is that it describes better the connections between the user and the system and takes both into account.
+The inner loop shows the connection between the human and the system through Norman's seven stages of action (\reffig{fig:sevenstages}), and the seven stages of reaction I described above (\reffig{fig:mysevenstages}).
The advantage of this approach is that it leverages the sensorimotor and execution loops through both perception (resp. input) and action (resp. output).
% something…
\section{Contributions}
-In the previous chapters I presented contributions to improve output by leveraging the sense of touch, and input by leveraging the motor abilities.
-In this chapter we discussed in the \refsec{sec:limits} that this approach are not always sufficient to improve interaction.
+In the previous chapters, I presented contributions to improve output by leveraging the sense of touch, and input by leveraging the motor abilities.
+In this chapter, we discussed in the \refsec{sec:limits} that this approach is not always sufficient to improve interaction.
The contributions below use the orthogonal approach as discussed above to improve interaction by leveraging the sensorimotor loop.
The first contribution is two interaction paradigms that leverage gestural interaction and vibrotactile feedback.
-%The first one use semaphoric gestures to replace pointing in mid-air gestural interaction.
-The second contribution investigates the contribution of haptics on the sense of embodiment of an avatar in Virtual Reality.
+%The first one uses semaphoric gestures to replace pointing in mid-air gestural interaction.
+The second contribution investigates the contribution of haptics to the sense of embodiment of an avatar in Virtual Reality.
\subsection{Haptic interaction paradigms}
\label{sec:hapticparadigms}
-In \refsec{sec:limits} we first wanted to measure quantitative benefits of haptic feedback for gestural interaction.
-The interaction paradigm we used was so inefficient that haptic feedback could not compensate its limitations.
+In \refsec{sec:limits} we first wanted to measure the quantitative benefits of haptic feedback for gestural interaction.
+The interaction paradigm we used was so inefficient that haptic feedback could not compensate for its limitations.
Here we propose a new interaction paradigm that gets around these limitations.
Then, we will discuss a new interaction paradigm that brings direct manipulation to tactile displays.
I implemented both paradigms with the device described in \refsec{sec:limits}.
The main limitations of 3D gestural interaction we discussed in \refsec{sec:limits} are tracking difficulties and the lack of segmentation.
The users are tracked without interruption.
Therefore gesture segmentation is difficult.
-Moreover, every gesture the users perform are potentially interpreted.
-This is called \defword{Midas Touch}, as a reference of the curse of the king that turned everything he touched into gold in the Greek mythology.
+Moreover, every gesture the users perform is potentially interpreted.
+This is called \defword{Midas Touch}, as a reference to the curse of the king that turned everything he touched into gold in the Greek mythology.
There are also issues when the user is outside the sensor field of view, either on the edges or when it is occluded.
The users need additional feedback for this in order to avoid usability issues.
And even though, it makes interaction more complicated.
These issues make it difficult to use standard GUI widgets.
We discussed in \refsec{sec:limits} the simple case of buttons that require a different activation mechanism.
-3D gestural interfaces typically use dwell buttons that require users to hold their hand still over a button for a couple of second to select it.
+3D gestural interfaces typically use dwell buttons that require users to hold their hand still over a button for a couple of seconds to select it.
We proposed to simply add haptic feedback, but we were unable to measure a quantitative benefit.
I believe we need deeper changes to improve interaction in this context.
Therefore we proposed a different paradigm that does not rely on pointing \& selection.
This new paradigm relies on summoning \& selection~\cite{gupta17}.
-This paradigm leverages a combination of semaphoric gestures, continuous gestures and tactile feedback (\reffig{fig:summonexample}).
-We first defined an segmentation hand posture (an open hand) to summon the GUI elements.
+This paradigm leverages a combination of semaphoric gestures, continuous gestures, and tactile feedback (\reffig{fig:summonexample}).
+We first defined a segmentation hand posture (an open hand) to summon the GUI elements.
Then we defined a different hand posture for different kinds of widgets: buttons, sliders, knobs, switches, spinboxes, and paired buttons.
It is of course possible to add other kinds of widgets with other hand postures.
-When the users perform one of these postures, they can select one of the widgtes of this type.
+When the users perform one of these postures, they can select one of the widgets of this type.
They receive a \qty{150}{\ms}/\qty{350}{\hertz} vibration pulse to confirm the selection, and the currently selected widget is visually highlighted.
If the GUI has several widgets of this type, the users can disambiguate with a continuous movement.
-Then they perform a gesture for manipulating the widget, and receive immediate haptic feedback with continuous vibrations on the thumb and index finger.
-For example they can pinch and drag to move a slider knob.
-They can release the knob by releasing the pinch, and release the slider by performing the segmentation gesture again.
+Then they perform a gesture for manipulating the widget and receive immediate haptic feedback with continuous vibrations on the thumb and index finger.
+For example, they can pinch and drag to move a slider knob.
+They can release the knob by releasing the pinch and release the slider by performing the segmentation gesture again.
% Upon summoning, a 150ms pulse is played in both rings to indicate that the slider is summoned. When the user enters the drag state, a continuous pulse starts playing in both rings to mirror the grip of the slider bar. The pulse stops upon exit from the drag state. To reduce any per- ceived irritability from the vibration, the amplitude was set just above the perceivable level and the frequency was set at 350Hz. The 150ms pulse is played again upon release.
-We conduct user studies to measure the benefits of this new paradigm.
-In the first one we showed that this paradigm avoids the Midas touch issues, and we compared two disambiguation mechanisms.
-In the second study we showed that this paradigm has quantitative and qualitative benefits compared to midair pointing.
+We conducted user studies to measure the benefits of this new paradigm.
+In the first one, we showed that this paradigm avoids the Midas touch issues, and we compared two disambiguation mechanisms.
+In the second study, we showed that this paradigm has quantitative and qualitative benefits compared to midair pointing.
Despite these benefits, this new paradigm has challenges that are still to be addressed.
-In particular it relies on semaphoric gestures that users have to know.
+In particular, it relies on semaphoric gestures that users have to know.
It contradicts Nielsen's \emph{recognition rather than recall} heuristic~\cite{nielsen90,nielsen94}, which is one of the essential benefits of the point \& select paradigm.
-Therefore, we still have to evaluate the discoverability and learnability of the gestures, and improve them if necessary~\cite{cockburn14}.
+Therefore, we still have to evaluate the discoverability and learnability of the gestures and improve them if necessary~\cite{cockburn14}.
We can for example encourage learnability and discoverability with feedforward visual cues in the vicinity of the widgets~\cite{malacria13}.
\input{figures/summonexample.tex}
It is one of the most important concepts of GUIs.
Its properties provide valuable usability benefits that highly contributed to the success of GUIs over command line interfaces.
Yet, this paradigm was tailored for visual interfaces.
-The question whether this concept could be used or adapted to tactile display was open.
+The question of whether this concept could be used or adapted to tactile display was open.
Therefore we studied the adaptation of direct manipulation to tactile displays~\cite{pietrzak15,gupta16,gupta16a}.
The most challenging direct manipulation property for tactile display is certainly the one stating that objects of interest have to be visible.
Contrary to vision, it is difficult to perceive an overview of the environment at a glance with the sense of touch.
-Vision is particularly efficient at glancing not only because of the high density and sensitivity of photoreceptor cells in the retina, but also because of the high mobility of eyes and the ability of the brain to process this sensorimotor loop.
+Vision is particularly efficient at glancing not only because of the high density and sensitivity of photoreceptor cells in the retina but also because of the high mobility of the eyes and the ability of the brain to process this sensorimotor loop.
Therefore, we leveraged the sensorimotor loop with the sense of touch and the motor ability to make objects of interest \emph{touchable} and \emph{explorable}.
In this new paradigm, the users can control a pointer that they can move continuously with gestures and perceive with tactile feedback.
When the cursor moves, it feels like a vibration moving continuously on the skin.
This property makes the tactile space explorable.
-Then, the sensation is different when the cursor hovers a target or moves on the background, which makes objects touchable.
+Then, the sensation is different when the cursor hovers over a target or moves over the background, which makes objects touchable.
Input modifiers such as the number of contact points or the number of contact repetitions are used to switch between the \emph{idle}, \emph{tracking}, and \emph{dragging} stages~\cite{buxton90}.
With these interactions we can implement fundamental direct manipulation interaction techniques such as \emph{pointing}, \emph{selection}, and \emph{manipulation}.
We implemented this paradigm with a proof of concept 1D \ang{360} tactile display around the wrist (\reffig{fig:tactiledm}).
-We used the prototype described in \refsec{sec:limits}, which has four EAI C2 tactors.
+We used the prototype described in \refsec{sec:limits}, which has four EAI C2 tactors.
The continuously moving cursor is implemented with the funelling illusion I described in \refchap{chap:output} and illustrated on \reffig{fig:illusions}.
The display is divided into four quarters in between the actuators.
The cursor is a phantom sensation created by interpolating the signal amplitude of the two edge actuators of the corresponding quarter.
Targets are represented with a \qty{250}{\hertz} frequency and the background with \qty{100}{\hertz}.
-Not only it is an easily distinguishable vibration, but the \qty{100}{\hertz} is subtle and avoids or at least reduce numbness.
+Not only it is an easily distinguishable vibration, but the \qty{100}{\hertz} is subtle and avoids or at least reduces numbness.
The inputs use a multitouch smartwatch.
-Up and down swypes move the cursor in either directions.
+Up and down swypes move the cursor in either direction.
The tracking states trigger with one contact point and the dragging state triggers with two contact points.
The cursor is not felt in the idle state, to avoid numbness.
The details about feedback and states machines are presented in the paper~\cite{gupta16}.
\input{figures/tactiledm.tex}
We validated the concept with user evaluations of the proof of concept prototype.
-First we validated that users are able to navigate and distinguish targets with a JND experiment on the maximum number of targets they were able to count.
-In average, participants were able to count up to 19 targets.
-Then we evaluated the pointing performance, and confirmed it globally follows Fitts' law.
-We note however that participants made faster selections when the targets were exactly on an actuator position compared to in-between.
+First, we validated that users are able to navigate and distinguish targets with a JND experiment on the maximum number of targets they were able to count.
+On average, participants were able to count up to 19 targets.
+Then we evaluated the pointing performance and confirmed it globally follows Fitts' law.
+We note however participants made faster selections when the targets were exactly over an actuator position compared to in-between.
We proposed a refined pointing model that takes into account this observation.
-Finally we designed two tactile menus with 4 and 8 items and we showed that participants were fast and accurate.
+Finally, we designed two tactile menus with 4 and 8 items and we showed that participants were fast and accurate.
\subsection{The sense of embodiment in Virtual Reality}
\label{sec:embodiment}
-We discussed in \refsec{sec:qualitative} that haptic feedback has qualitative benefits, in particular when it restores haptic sensations that are non existent or limited in gestural or multi-touch interaction.
+We discussed in \refsec{sec:qualitative} that haptic feedback has qualitative benefits, in particular when it restores haptic sensations that are non-existent or limited in gestural or multi-touch interaction.
This is especially important in \defwords{virtual environments}{virtual environment} in which we would like to immerse users.
Slater defined \defword{immersion} as “the extent to which the actual system delivers a surrounding environment”~\cite{slater97}.
-Therefore, this notion refers to technological aspects that contribute to immerse the user in the virtual environment.
+Therefore, this notion refers to technological aspects that contribute to immersing the user in the virtual environment.
\defword{Presence} is rather the subjective feeling of being inside a virtual environment~\cite{slater09,slater99}.
-Witmer and Singer proposed a questionnaire to measure user's presence in a virtual environment~\cite{witmer98}.
+Witmer and Singer proposed a questionnaire to measure users' presence in a virtual environment~\cite{witmer98}.
They identified four factors that influence the feeling of presence: the ability to \emph{control} objects, \emph{sensory} stimulation, \emph{distraction} from the real world, and \emph{realism}.
We used this questionnaire in the study presented in \refsec{sec:qualitative} and observed that haptic feedback improved presence, in particular sensory and realism factors.
-%Virtual Reality headsets create immersive virtual environments that takes the whole field of view of users.
+%Virtual Reality headsets create immersive virtual environments that take the whole field of view of users.
%Therefore they cannot even see their physical body
The users are generally represented in virtual environments with an \defword{avatar}.
This avatar usually has a visual representation, which is not necessarily realistic or even human~\cite{olivier20}.
The users explore the virtual environment through this avatar.
-They can also perform operations thatare impossible in the physical world, like telekinesis or teleportation.
+They can also perform operations that are impossible in the physical world, like telekinesis or teleportation.
In fact, the appearance or behavior of the avatar has an influence on the way the users behave in the virtual environment.
For example, the \defword{Proteus effect} describes the way the visual representation of an avatar influences the behavior of the users that control it~\cite{yee07}.
-At the opposite, visuo-tactile stimulation can lead people to consider a rubber hand as part of their body~\cite{botvinick98}, or that they have a sixth finger on their hand~\cite{hoyet16}.
-These effects are examples of extension of the \defword{sense of embodiment} of a virtual body~\cite{kilteni12}.
-%, or artificial artefacts such as protheses or tools for example~\cite{devignemont11}.
+On the opposite, visuotactile stimulation can lead people to consider a rubber hand as part of their body~\cite{botvinick98}, or that they have a sixth finger on their hand~\cite{hoyet16}.
+These effects are examples of extensions of the \defword{sense of embodiment} of a virtual body~\cite{kilteni12}.
+%, or artificial artifacts such as prostheses or tools for example~\cite{devignemont11}.
% Embodiment: E is embodied if and only if some properties of E are processed in the same way as the properties of one’s body.
Kilteni \etal discuss three subcomponents of the sense of embodiment that were extensively studied in the literature~\cite{kilteni12}.
-\defword{Self location} refers to the “volume in space where one feels to be
+\defword{Self-location} refers to the “volume in space where one feels to be
located”.
\defword{Agency} refers to “the sense of having global motor control” over the virtual body.
And \defword{ownership} refers to “one’s self-attribution of a body”.
\subsubsection{Methodologies for measuring the sense of embodiment}
-There is a number of questionaires in the literature to measure the sense of embodiment.
+There is a number of questionnaires in the literature to measure the sense of embodiment.
We discuss some of them in one of our studies~\cite{richard22}.
%We can measure the embodiment of an avatar in a virtual environment with questionnaires~\cite{roth20,gonzalezfranco18,peck21}.
-There are recent attemps to standardize these questionnaires.
+There are recent attempts to standardize these questionnaires.
For example Roth \etal propose a questionnaire with subcomponents: \defword{ownership}, \defword{agency}, and perceived change in the \defword{body schema}~\cite{roth20}.
-The latter notion is larger than \emph{self location} as it refers to any difference the users may perceive between their own body and the avatar.
+The latter notion is larger than \emph{self-location} as it refers to any difference the users may perceive between their own body and the avatar.
Gonzalez Franco and Peck proposed another questionnaire in which they added to Kilteni \etal's subcomponents : \emph{tactile sensations}, \emph{external appearance}, and \emph{response to external stimuli}~\cite{gonzalezfranco18}.
They later improved and simplified their questionnaire, and evaluated it with many different tasks~\cite{peck21}.
The subcomponents of this new questionnaire are: \emph{appearance}, \emph{response}, \emph{ownership}, and \emph{multi-sensory}~\cite{peck21}.
%This does not mean that agency, touch or localization are not important for embodiment, (Kilteni et al., 2012), but rather that they are related to other senses and instead contribute to one of the four prominent embodiment categories. The questions on motor control and agency were mostly assigned to the Response category
These questionnaires are typically used in controlled experiments after the participants performed a specific task in a virtual environment.
-We compare the overall embodiment and its subcomponents score in two or more conditions to identitfy the effects of these condition.
+We compare the overall embodiment and its subcomponents score in two or more conditions to identify the effects of these conditions.
The experimental protocol we can use depends on the task.
-For example some studies use a threat like a virtual fire or sharp blade as an objective measurement of embodiment~\cite{dewez19,argelaguet16}.
+For example, some studies use a threat like a virtual fire or sharp blade as an objective measurement of embodiment~\cite{dewez19,argelaguet16}.
Subjects are considered embodied if they attempt to avoid the threat despite its virtual nature.
-The issue is that this kind of metric requires participants to be surprized by the threat.
+The issue is that this kind of metric requires participants to be surprised by the threat.
However, this cannot be guaranteed with a \defword{within-subjects design} in which participants perform all the conditions one after the other.
-In such situations the experiment must follow a \defword{between-subjects design}, in which separate groups of participants perform a different condition.
+In such situations, the experiment must follow a \defword{between-subjects design}, in which separate groups of participants perform a different condition.
% There are however several other factors that influence the choice of experimental setup.
% For example, between-subjects studies require more participants to reach the same statistical power.
% Each participant of a within-subject study provides less data per condition if we would like to keep the same experiment duration.
In a between-subjects study, users are assigned to one of the conditions.
There is therefore potentially a bias if the groups are not well balanced.
We investigated this effect on embodiment studies~\cite{richard22}.
-We experimented a visuo-motor task with a synchronous condition and an asynchronous condition with a latency of \qty{300}{\ms} between the inputs and output response.
+We experimented a visuomotor task with a synchronous condition and an asynchronous condition with a latency of \qty{300}{\ms} between the inputs and output response.
This value is known to have a medium effect on embodiment in the literature~\cite{botvinick98,kilteni12,kokkinara14}.
We chose a simple experimental task that requires no special equipment to facilitate replication.
-Participants were seated on a chair, with the legs on a table, and had to perform gestures with their feet (\reffig{fig:expewithin}), similarly to~\cite{kokkinara14}.
+Participants were seated on a chair, with their legs on a table, and had to perform gestures with their feet (\reffig{fig:expewithin}), similarly to~\cite{kokkinara14}.
92 participants performed this task in a balanced within-subjects design.
To study the effect of the sample size and its effect on the statistical analysis we analyzed random data subsets of 10 to 92 participants.
-To study the effect of the experiment design we simulated between-subjects design by selecting the first condition ever participant made.
+To study the effect of the experiment design we simulated between-subjects designs by selecting the first condition ever participant made.
We considered the analysis of all participants with the within-subjects design as the ground truth, which gave the same result as the literature~\cite{botvinick98,kilteni12,kokkinara14}.
\begin{figure}[htb]
\end{figure}
Our results show that all the random subsets with at least \num{40} participants with the within-subjects design gave the same result as the ground truth.
-However, regardless of the number of participants the between-subject analyses do not reveal the ground truth effect.
-Based on the debrieffing with participants, our main explanation of this phenomenon is that participants needed a reference to provide a meaningful answer for each question.
+However, regardless of the number of participants, the between-subject analyses do not reveal the ground truth effect.
+Based on the debriefing with participants, our main explanation of this phenomenon is that participants needed a reference to provide a meaningful answer for each question.
Therefore they calibrated their answers to the second condition relatively to the first one.
Hence, we could not measure the effect with the first condition only.
-We discuss recommendation and possible mitigation strategies in the paper~\cite{richard22}.
+We discuss recommendations and possible mitigation strategies in the paper~\cite{richard22}.
Interestingly, when we analyzed the second condition as a kind of calibrated between-subjects design we observed the ground truth effect.
However, the effect size was about half the effect size of the within-subject analysis.
Therefore, we wonder if both designs even measured the same phenomenon.
The study of the causes and effects of the sense of embodiment of an avatar in virtual reality is a hot topic in the Virtual Reality community.
%Results show that the sense of agency is stronger for less realistic virtual hands which also provide less mismatch between the participant's actions and the animation of the virtual hand. In contrast, the sense of ownership is increased for the human virtual hand which provides a direct mapping between the degrees of freedom of the real and virtual hand.
Interestingly, all the embodiment questionnaires such as those we discussed before have subcomponents related to the sensorimotor loop.
-It means that the sensorimotor loop is essential ot the sense of embodiment.
+It means that the sensorimotor loop is essential to the sense of embodiment.
For example, people have a stronger sense of ownership when they perform actions with a visually realistic hand, and a stronger sense of agency when they embody an abstract-looking virtual hand~\cite{argelaguet16}.
-Following this idea, we studied the effect on haptics on the sense of embodiment.
+Following this idea, we studied the effect of haptics on the sense of embodiment.
We performed a user study to compare embodiment for a drawing task with force feedback, tactile feedback, and a control condition with no haptic feedback~\cite{richard20}.
The participants were seated on a chair, and they had to paint a mandala in an immersive virtual environment with a Phantom Desktop\footnote{Today called Touch X by 3D Systems \url{https://www.3dsystems.com/haptics-devices/touch-x}} device (\reffig{fig:expeembodiment}).
-In the force feedback condition they felt the surface resistance of hard objects, and the viscosity of paint.
-In the tactile condition, they felt a \qty{250}{\hertz} vibration whose amplitude was protortional to the interpenetration distance to the canvas surface.
+In the force feedback condition, they felt the surface resistance of hard objects and the viscosity of the paint spheres at the bottom.
+In the tactile condition, they felt a \qty{250}{\hertz} vibration whose amplitude was proportional to the interpenetration distance to the canvas surface.
We attached an EAI C2 tactor to vibrate the Phantom stylus (\reffig{fig:expeembodiment}).
-In the control condition the Phantom was only used as an input device, with no force or vibration.
+In the control condition, the Phantom was only used as an input device, with no force or vibration.
We mesured embodiment with Gonzalez Franco and Peck's first standardized questionnaire\footnote{The second one was not published at the time.} with the \emph{agency}, \emph{self location}, \emph{ownership}, and \emph{tactile sensations} subcomponents~\cite{gonzalezfranco18}.
\begin{figure}[htb]
\end{figure}
We observed a stronger embodiment in the force feedback condition compared to the control condition.
-In particular participants had a higher sense of ownership.
+In particular, participants had a higher sense of ownership.
However, we did not observe these differences between the tactile and control conditions.
Besides the detailed discussion in the paper, it is important to note that in some ways this task favored the force feedback condition over the tactile condition.
Participants certainly expected to feel the stiffness of hard surfaces.
Similarly to realistic visual feedback~\cite{argelaguet16}, this realistic force feedback aspect reinforced the sense of ownership
-At the contrary the vibrotactile feedback was symbolic because participants only received tactile guidance.
-And we did not observe any improvement of embodiment.
+On the contrary the vibrotactile feedback was symbolic because participants only received tactile guidance.
+And we did not observe any improvement in embodiment.
It does not necessarily mean that the sense of embodiment requires realistic haptic feedback.
For example, non-realistic visual feedback improved the sense of agency~\cite{argelaguet16}.
But in our task force feedback \emph{constrained} the stylus tip movement to prevent it from getting through the surface, while vibrotactile feedback only \emph{guided} it.
-Therefore I believe the force feedback condition had a stronger sensorimotor integration, which helped participants focusing on the painting task rather than controlling the stylus to paint the canvas.
+Therefore I believe the force feedback condition had a stronger sensorimotor integration, which helped participants focus on the painting task rather than controlling the stylus to paint the canvas.
The workload analysis discussed in the paper gives supports this explanation.
%It gave users immediate feedback that could guide them to stay close to the spatial location of the surface.
Further studies should investigate other tasks or a variation of this one in which tactile feedback favors sensorimotor integration.
% is expected, like feeling surface textures.
-\subsection{Discussion}
-
\section{Conclusion}
projects / hdr.git / commitdiff