The message of this chapter is that input and output have equivalent importance for the design of interactive systems, and must be designed together.
This is not as trivial as it seems to be at first sight.
To illustrate this, I describe below two parts of rejected submissions and explain what was wrong with them.
-It has to do with the motivations and the lack of focus on the sensorimotor loop.
+It had to do with the motivations and the lack of focus on the sensorimotor loop.
It made me realize that this is a large hole in the two previous chapters.
-After this assessment, I will connect the various angles of the sensorimotor loop, and discuss what it brings to the different disciplines around HCI.
-\fixme{something here about how I extend it}
-I will conclude with some of my favorite contributions to explain the critical role of the sensorimotor loop.
-
+Therefore I will discuss the sensorimotor loop, its connection to both human and computing models, and postion this approach to the approach of the first two paragrtaphs.
+%After this assessment, I will connect the various angles of the sensorimotor loop, and discuss its positioning
+%what it brings to the different disciplines around HCI.
+%\fixme{something here about how I extend it}
+I will conclude with contributions that illustrate the critical role of the sensorimotor loop in interaction.
%However vision is generally already used to display a lot of information.
%In the case of smartwatches, the screen real estate is limited.
%separation sometimes necessary, to isolate an input or output phenomenon.
-
-\section{The limits}
+\section{The limits of the separated approach}
\label{sec:limits}
During my postdoc at the University of Toronto, I worked on the lack of haptic feedback in 3D gestural interaction.
It featured a motion capture system affordable for households.
It followed the trend of \emph{natural} user interfaces, and was advertised as “\emph{You} are the controller”\footurl{https://tiny.one/youAreTheController}.
Besides the discussion about the natural aspect of user interfaces and their relevance in the previous chapter, this motivation to eliminate physical controllers also eliminated many useful, not to say essential, haptic properties of physical controllers.
-This was already an issue with touch interaction, but there was at least the passive haptics of the surface.
+This was already an issue with touch interaction, but there was at least the passive haptics of the interactive surface.
In the case of 3D gestural interaction, the users have no haptic feedback when they interact with virtual objects.
Therefore, the immediate feedback they need for direct manipulation~\cite{schneiderman83} is essentially visual or auditory.
This contributes to the overload of these modalities.
It helped me with the ideation of the concept depicted in \reffig{fig:hapticpath} in \refchap{chap:output}.
We discussed the output vocabulary of vibrotactile feedback in \refchap{chap:output}.
-The independent control of both the frequency and amplitude of the signal is necessary for an expressive output vocabulary.
-It requires precise actuators such as the EAI C2 tactors we used \cite{mortimer07}.
+The independent control of both the frequency and amplitude of a vibrotactile signal is necessary for an expressive output vocabulary.
+It requires precise vibrotactile actuators such as voice coil actuators.
+We used EAI C2 tactors~\cite{mortimer07}.
The typical way to drive precise vibrotactile actuators is to use a sound generation system.
This is convenient because the parameters of the signal are the same: frequency, amplitude, and shape.
The main difference is the frequency range: \qtyrange{1}{1000}{\hertz} for haptics and \qtyrange{200}{20}{\kilo\hertz} for sound.
Managing the amplitude is easier with vibrations because the required amplitude levels are much lower.
In the end, the shape parameter is in my opinion the bottleneck of complexity for the implementation of vibrotactile devices because it imposes a much higher sampling rate.
It makes the design of sound generation systems complex, especially with microcontrollers available at the time this project started.
-For the sake of simplicity, I rather opted for a straightforward design that enabled the precise control of both frequency and amplitude at the cost of a low control of the signal shape\footnote{Controlling the signal shape remains possible, with a software $\Delta\Sigma$ modulation \href{https://tiny.one/DeltaSigma}{https://tiny.one/DeltaSigma}.}.
+For the sake of simplicity, I rather opted for a straightforward design that enabled the precise control of both frequency and amplitude at the cost of a low control of the signal shape\footnote{Controlling the signal shape remains possible, with a software $\Delta\Sigma$ modulation for example \href{https://tiny.one/DeltaSigma}{https://tiny.one/DeltaSigma}.}.
The idea is to control the frequency and amplitude with two PWM signals generated by the timers of a microcontroller (\reffig{fig:actuatorcircuit}).
The frequency signal typically ranges between \qtyrange{1}{1000}{\hertz}.
The amplitude is controlled by the duty cycle of a high-frequency signal.
The first two prototypes used Arduino LilyPad microcontroller boards~\cite{buechley08}.
These boards are designed for wearables, so they made sense for this project.
On the first prototypes, the components were sewn with conductive thread.
-It caused several issues.
+It caused several issues though.
First, the thin conductive thread had a non-negligible resistance.
-Therefore it was necessary to multiply the connections to have enough power flowing in the circuit.
+Therefore it was necessary to duplicate the connections to have enough power flowing in the circuit.
Second, the elasticity of the wristband was convenient for comfort, but it caused short circuits that made the device unreliable.
To alleviate these issues, I soldered the components on small protoboards, which I connected together with conductive thread.
Now the issue was the connection between the thread and the pads.
The takeaway is that prototyping interactive devices requires both keeping in mind the design rationale and the technical constraints of their implementation.
It is critical to balance the trade-offs and make informed compromises.
For example, replacing conductive thread with wires was not a compromise after all, and make the prototype more robust.
-It was the consequence of the bad choice of microcontroller board from the beginning.
+It was a constraint motivated by a bad choice of microcontroller board from the beginning.
The restrictions on signal shape and the common frequency were actual compromises.
But they enabled a simple and robust design while keeping an expressive output vocabulary.
We ran an experiment with the idea to measure an increase in performance in a tactile condition over a visual condition.
This was motivated by the fact that when we tried the buttons, tactile feedback seemed to bring some benefit.
+We did not have a clear idea of \emph{what} was better though.
We presented to participants a screen with an array of four-by-four buttons.
They had to select a series of buttons indicated with a highlight.
The details of the experiment do not matter.
They were advised to avoid going out of the road since it notably reduces the speed.
They stood \qty{2.5}{\meter} away from the Kinect during the game.
%Before the experiment the height of the Kinect was calibrated to make sure the participants' arms were in the sensor’s range.
-The game was displayed on a 17” laptop screen, with a $1600 \times 900$ resolution, and the same sound volume was used for all subjects.
+The game was displayed on a 17” laptop screen, with a \numproduct{1600 x 900} resolution, and the same sound volume was used for all subjects.
We opted for a between-subjects design: half of the participants received tactile feedback (\Tactile), and the other half did not (\NoTactile).
We explained the mapping of the tactile feedback to the participants of condition \Tactile.
Tactile feedback was not mentioned to the participants of condition \NoTactile.
\subsection{Discussion}
The paradox here is that the motivation for 3D gestural interaction was to offer users \emph{natural} ways to interact with systems, without manipulating artificial artifacts.
-Whether it was relevant or not, without haptic sensations the virtual objects manipulated do not feel real to users, and they sometimes struggle to manipulate them.
+Whether it was relevant or not, without haptic sensations the virtual objects manipulated do not \emph{feel} real to users, and they sometimes struggle to manipulate them because of this.
Restoring haptic feedback on dwell buttons was not sufficient to make them efficient.
My hypothesis was that restoring tactile feedback would help users press them.
I anticipated that this would increase selection performance.
However, with the way I implemented these buttons activation required at least \qty{3}{\s}.
This is very long indeed.
+But it was a replication of buttons in Kinect menus.
Should tactile feedback help select buttons faster, the effect size could not be sufficient to make a significant improvement.
The average selection time in the \NoTactile condition was \qty{3.6}{\s}, and it was \qty{3.5}{\s} in the \Tactile.
This is the first clue that improving haptic feedback is not sufficient.
Tactile feedback did provide qualitative benefits.
However, the low quality of inputs diluted these benefits and made them barely measurable.
The haptic feedback provided was questionable as well.
-I used spatial location around the wrist, however, the task required participants to rotate their wrists.
+I used spatial location around the wrist.
+However, the task required participants to rotate their wrists.
Therefore it might have influenced negatively how participants interpreted the tactile cues.
Unfortunately, I did not investigate this issue at the time.
Moreover, the haptic cues indicated the steering angle of the wheel, not the car.
It is one of the many clues showing that inputs and outputs cannot be separated, the same way that our senses and abilities cannot be separated either.
One of the major differences between humans and systems is that humans are as they are, we cannot fundamentally change the way they function.
+They can learn and get experience, but we cannot give people new senses and abilities.
Systems are different: we build them, therefore we can design and build them according to our needs.
We have no reason to build a machine that has no purpose for us.
We discuss below how the literature modeled the way humans work, and models for designing systems.
-Interestingly, both work in a similar way, with input, processing, and output.
-The question is: is it like this because of an anthropocentric bias or because this is an operational optimum?
-I will discuss this question with my vision of how interactive systems work and the critical role of the sensorimotor loop.
+Interestingly, both work in a similar way, with inputs, processing, and outputs.
+%The question is: is it like this because of an anthropocentric bias or because this is an operational optimum?
+I will discuss this observation, my vision of how interactive systems work, and the critical role of the sensorimotor loop.
%Humans and systems are the same kind of entity, interacting with the world in a similar way.
%Not because it has to be, but because we took inspiration from humans to design interactive systems.
For example, in the previous section, we discussed the fundamental coupling between humans' perception and action.
Systems called \defwords{closed-loop systems}{closed-loop system}\footurl{https://en.wikipedia.org/wiki/Control_system} also leverage such a mechanism.
For example, the non-inverting and inverting amplifiers circuit depicted on \reffig{fig:amplifiers} have the output of their operational amplifier connected to one of their input through a resistor.
-The output voltage is proportional to the input voltage whose value depends on $R_1$ and $R_2$.
+As a result, the output voltage is proportional to the input voltage whose value depends on $R_1$ and $R_2$.
But most importantly, the feedback loop stabilizes the output voltage to the desired value.
This kind of control mechanism is used in many applications such as robots, domestic appliances, or drones.
It is also used in haptic devices that leverage information from people with \defword{Human-in-the-loop} models~\cite{vanderlinde02}.
-This paradigm focuses on system control.
+However, this paradigm focuses on system control.
The human only exists as a parameter of the equation.
Therefore despite the similarities with the way we describe human behavior, this is not our focus.
Hence we will discuss below the architecture of interactive systems, the similarities and differences with humans, and how this is critical for improving interactions.
\paragraph{Computation models}
-Initially, computers and programs were essentially based on theoretical models such as \defword{$\lambda$-calculus}~\cite{church32} or \defwords{Turing machines}{turing machine}~\cite{turing38}.
+Initially, computers and programs were essentially based on theoretical models such as \mbox{\defword{$\lambda$-calculus}}~\cite{church32} or \defwords{Turing machines}{turing machine}~\cite{turing38}.
These are computing models, and they focus on solving numerical problems rather than helping people with their everyday activities.
A Turing machine has an infinite tape with symbols written in advance, and a pre-defined transition table that describes the behavior of the machine.
Therefore these machines ignore their environment, in which anything can change at any time.
For example, the Listing~\ref{lst:induction} shows the inductive definition of a list of numbers and a function that computes the length of a list.
A list is built with two constructors: either a \verb+Nil+ value for an empty list or a \verb+Cons+ function that create a list with a number (the head) and another list (the tail).
The \verb+Nil+ value ensures that the list is finite.
-It ensures in turn that the \verb+length+ function ends because there is no recursive call on the base case (\verb+Nil+) and every list ends with Nil.
+It ensures in turn that the \verb+length+ function ends because there is no recursive call on the base case (\verb+Nil+) and every list ends with \verb+Nil+.
\begin{code}[language=Coq, label=lst:induction, caption=Inductive list and example of inductive function on a list.]
Inductive list : Set :=
For example, on the system level, an input loop gets input streams of input data (\eg mouse displacements) and produces output streams of input events (\eg mouse move event).
On the application level, an input loop gets output streams of input events and combines the information they convey with interaction techniques to produce an output stream of actions to be executed.
On the application level, a graphics loop gets an input stream of graphic commands and produces an output stream of objects to be displayed.
-Therefore, applications are therefore what Wegner calls \defwords{interaction machines}{interaction machine}~\cite{wegner97}.
+Therefore, applications are what Wegner calls \defwords{interaction machines}{interaction machine}~\cite{wegner97}.
%\defwords{Neural networks}{neural network} are other examples of interaction machines: they also get input streams and produce output streams~\cite{mcculloch43}.
\paragraph{Software architectures and interaction paradigms}
Software architectures leverage this interaction machine to describe a higher-level structure that connects users to a functional \emph{model}.
This model, also called an \emph{abstraction}, defines the objects of the system, their properties, and the operations on them.
For example, the original \defacronym{MVC} architectures distinguish the model with \emph{views} that describe how objects are presented to users and \emph{controllers} that define the way users can manipulate them~\cite{reenskaug79,reenskaug79a}.
-\defword{Arch}~\cite{arch92} and \defacronym{PAC}~\cite{coutaz87} rather combine input and outputs as a \emph{presentation} component, and add a \emph{controler} component that manages transitions between abstract inputs/outputs and domain-specific properties of the model/abstraction.
+\defword{Arch}~\cite{arch92} and \defacronym{PAC}~\cite{coutaz87} rather combine inputs and outputs as a \emph{presentation} component, and add a \emph{controler} component that manages transitions between abstract inputs/outputs and domain-specific properties of the model/abstraction.
%The modern MVC architectures follow this structure as well.
The advantage of these architectures is to separate the objects of interest from the interaction with them.
It is therefore easy to display several synchronized representations of the same object and provide multiple ways to manipulate them.
% thei know their interpretation of the other entities, not what they actually are
Humans and machines are autonomous entities but they are not independent.
-On one side, humans use machines to extend their limitations discussed above.
+On one side, humans use tools, machines, and instruments to extend their limitations discussed above.
Extending human capacities with computers marked the beginning of Human-Computer Interaction, with the pioneer visions of Bush~\cite{bush45} and Engelbart~\cite{engelbart68}.
On the other side, all machines need humans otherwise they have no purpose.
They all need instructions and data, and they all modify the environment or produce information.
The characteristics of these communications depend on the skills and capabilities of both entities.
For example, the most efficient communication between two humans is certainly speech.
This is maybe the reason why there is a push toward vocal interfaces.
-However, so far technologies struggle with the interpretation of natural languages because of their complexity.
+However, so far technologies struggle with the interpretation of natural languages because of their lack of flexibility compared to a human mind.
+% and the complexity of these .
Vocal interfaces only recognize a limited and pre-defined set of instructions.
But in addition to speech, we make gestures with the hands, face, or other body parts that can totally change the meaning.
-As we discussed in the previous chapter, systems are efficient at sensing gestures, and there is still room for improvement.
-Therefore this is today the main communication modality between humans and systems.
+As we discussed in the previous chapter, systems are efficient at sensing gestures, and there is even still room for improvement.
+Therefore this is today the main communication modality between humans and machines.
%It is important to note that what the second entity does not perceive this physical effect, but its own interpretation of it.
%Effectors can produce light (like screens), sounds, vibrations, forces, …
Output devices have driving electronics that require specific \emph{commands} and turn them into \emph{physical effects}.
These are typically lights (like screens), sounds, vibrations, and forces.
+This model extends software models like PAC, MVC, or Arch because it describes the interactive system on three levels.
+Similarly to these architectures, it describes the application level with the input phrase, the program, and the encoding.
+But it also describes the system level, with the input event and the command creation.
+And it also describes the connection with the physical world with the sensing and physical effect stages.
\input{figures/sevenstages2.tex}
In the previous chapters, I presented contributions to improve output by leveraging the sense of touch, and input by leveraging the motor abilities.
In this chapter, we discussed in the \refsec{sec:limits} that this approach is not always sufficient to improve interaction.
The contributions below use the orthogonal approach as discussed above to improve interaction by leveraging the sensorimotor loop.
-The first contribution is two interaction paradigms that leverage gestural interaction and vibrotactile feedback.
+The first contribution provide quantitative benefits with two interaction paradigms that leverage gestural interaction and vibrotactile feedback.
%The first one uses semaphoric gestures to replace pointing in mid-air gestural interaction.
-The second contribution investigates the contribution of haptics to the sense of embodiment of an avatar in Virtual Reality.
+The second contribution investigates qualitatives benefits of the sensorimotor loop on the sense of embodiment of an avatar in Virtual Reality.
\subsection{Haptic interaction paradigms}
\label{sec:hapticparadigms}
\paragraph{User study}
-In a between-subjects study, users are assigned to one of the conditions.
+In a between-subjects study, participants are assigned to one of the conditions.
There is therefore potentially a bias if the groups are not well balanced.
We investigated this effect on embodiment studies~\cite{richard22}.
We experimented a visuomotor task with a synchronous condition and an asynchronous condition with a latency of \qty{300}{\ms} between the inputs and output response.
Participants were seated on a chair, with their legs on a table, and had to perform gestures with their feet (\reffig{fig:expewithin}), similarly to~\cite{kokkinara14}.
92 participants performed this task in a balanced within-subjects design.
To study the effect of the sample size and its effect on the statistical analysis we analyzed random data subsets of 10 to 92 participants.
-To study the effect of the experiment design we simulated between-subjects designs by selecting the first condition ever participant made.
-We considered the analysis of all participants with the within-subjects design as the ground truth, which gave the same result as the literature~\cite{botvinick98,kilteni12,kokkinara14}.
+To study the effect of the experiment design we simulated between-subjects designs by selecting the first condition every participant made.
+We considered the analysis of all participants with the within-subjects design as the ground truth.
+Similarly to the literature this analysis shows that latency reduces the sense of embodiment~\cite{botvinick98,kilteni12,kokkinara14}.
\begin{figure}[htb]
\centering
\label{fig:expewithin}
\end{figure}
-Our results show that all the random subsets with at least \num{40} participants with the within-subjects design gave the same result as the ground truth.
-However, regardless of the number of participants, the between-subject analyses do not reveal the ground truth effect.
+Our results showed that all the random subsets with at least \num{40} participants with the within-subjects design gave the same result as the ground truth.
+However, regardless of the number of participants, we did not observe the ground truth effect with the between-subject analyses.
Based on the debriefing with participants, our main explanation of this phenomenon is that participants needed a reference to provide a meaningful answer for each question.
Therefore they calibrated their answers to the second condition relatively to the first one.
Hence, we could not measure the effect with the first condition only.
However, we did not observe these differences between the tactile and control conditions.
Besides the detailed discussion in the paper, it is important to note that in some ways this task favored the force feedback condition over the tactile condition.
Participants certainly expected to feel the stiffness of hard surfaces.
-Similarly to realistic visual feedback~\cite{argelaguet16}, this realistic force feedback aspect reinforced the sense of ownership
+Similarly to realistic visual feedback~\cite{argelaguet16}, this realistic force feedback aspect reinforced the sense of ownership.
On the contrary the vibrotactile feedback was symbolic because participants only received tactile guidance.
And we did not observe any improvement in embodiment.
It does not necessarily mean that the sense of embodiment requires realistic haptic feedback.
For example, non-realistic visual feedback improved the sense of agency~\cite{argelaguet16}.
But in our task force feedback \emph{constrained} the stylus tip movement to prevent it from getting through the surface, while vibrotactile feedback only \emph{guided} it.
-Therefore I believe the force feedback condition had a stronger sensorimotor integration, which helped participants focus on the painting task rather than controlling the stylus to paint the canvas.
+Therefore I believe the force feedback condition helped participants focus on the painting task rather than controlling the stylus to paint the canvas , which reinforced sensorimotor integration.
The workload analysis discussed in the paper gives supports this explanation.
%It gave users immediate feedback that could guide them to stay close to the spatial location of the surface.
-Further studies should investigate other tasks or a variation of this one in which tactile feedback favors sensorimotor integration.
+Further studies should investigate other tasks or a variation of this one in which vibrotactile feedback promotes sensorimotor integration.
% is expected, like feeling surface textures.
\section{Conclusion}
+Interdisciplinarity: better describe the connections between domainns involved in the design, implementation and evaluation of interactive systems.
+One of the main difficulties of interdisciplinarity is to understand each other's research questions and roles.
+A common model gives space to everybody.
+
+Sensorimotor loop and its numerous connections to many domains.
+
+Descriptive and predictive aspect: ok
+Generative aspect?
+
+%Arduino made electronics mainstream / available to hobbyists
+
+
\todo{Maybe move stuff below to the discussion…}
Wegner describe several kinds of what he calls \defwords{interaction machines}{interaction machine}.
projects / hdr.git / commitdiff