% This question will remain open.
% However we will discuss here the
-In the previous chapter we discussed haptics as the sense of touch.
+In the previous chapter, we discussed haptics as the sense of touch.
This is the common interpretation of the word “haptic”.
-However haptics also refers to our ability to touch and manipulate.
+However, haptics also refers to our ability to touch and manipulate.
%In the context of interactive systems, this refers to inputs.
-In this chapter we will focus on gestures we perform with our fingers, hands and arms.
+In this chapter, we will focus on gestures we perform with our fingers, hands, and arms.
%Most of input modalities require touch and manipulation.
It covers most of the commonly used input modalities: button, pointing interfaces, touch, and gestural interaction.
All these modalities rely on gestures, but they leverage different parameters.
-Typically, buttons only sense binary contacts regardless of which finger pressed it, or the force or speed of actuation.
+Typically, buttons only sense binary contacts regardless of which finger pressed it or the force or speed of actuation.
Pointing interfaces such as mice or touchpads sense 2D movements in addition to contacts.
-Multi-touch interfaces can even sense contacts and movements of several contact points.
+Multi-touch interfaces can even sense the contacts and movements of several contact points.
Finally, gestural interaction can be sensed in 3D in the air without contact.
%All these modalities leverage our ability to touch and manipulate.
The user produces a mechanical effect that the system will sense and interpret.
Therefore the user will play the same role as a haptic device.
With this in mind, this is not surprising that cognitive scientists call this phenomenon output, as opposed to input for computer scientists.
-The \reffig{fig:motorpath} depicts both the user and the system, which both have an hardware part, in the physical world and a software part.
-The software part of users corresponds to ideas, or the mind in general, whereas the sotware part of the system refers to the code and its execution.
+The \reffig{fig:motorpath} depicts both the user and the system, which both have a hardware part, in the physical world, and a software part.
+The software part of users corresponds to ideas, or the mind in general, whereas the software part of the system refers to the code and its execution.
-The purpose of the modalities mentioned above is essentially to sense and interpret gestures of the fingers, hands, and arms.
-This is a much more complex task that it seems at first sight.
+The purpose of the modalities mentioned above is to sense and interpret the gestures of the fingers, hands, and arms.
+This is a much more complex task than it seems at first sight.
The hand alone has 21 degrees of freedom~\cite{elkoura03} and the arm adds 7 more~\cite{nasa14}.
-The movement range of each of these degrees of freedom depends on multiple factors, including morphology, physical condition, age and gender~\cite{nasa14}.
+The movement range of each of these degrees of freedom depends on multiple factors, including morphology, physical condition, age, and gender~\cite{nasa14}.
It is therefore not surprising that input systems only sense a small part of the possible human movements.
\begin{figure}[htb]
\paragraph{Motor ability}
\begin{definition}{ability}
- The human \defwords{abilities}{ability} refer to the human capacities to act on their environment, similarly to the human \defwords{senses}{sense} that refer to the human capacities to get information from their environment.
+ Human \defwords{abilities}{ability} refer to the human capacities to act on their environment, similar to the human \defwords{senses}{sense} that refer to the human capacities to get information from their environment.
\end{definition}
%In this chapter I use the term \defword{motor ability} as the mirror of the notion of \emph{sense of touch}.
-In this chapter we focus on the \emph{motor ability}, which leverages the \emph{motor system} to touch and manipulate the environment and objects it contains.
-As depicted on \reffig{fig:motorpath}, this notion comprises both the motor system, and the associated part of the cognitive system.
-Specifically, when users sould like to perform an action they form an intention that the cognitive system will turn into execution commands for the motor system.
-The motor system, typicall muscles, tendons and articulations actuates the body, which in turn produces physical effects.
-This physical effects enable the manipulation or objects in contact with the mobile parts of the body.
+In this chapter, we focus on the \emph{motor ability}, which leverages the \emph{motor system} to touch and manipulate the environment and objects it contains.
+As depicted on \reffig{fig:motorpath}, this notion comprises both the motor system and the associated part of the cognitive system.
+Specifically, when users should like to perform an action they form an intention that the cognitive system will turn into execution commands for the motor system.
+The motor system, typically muscles, tendons, and articulations actuate the body, which in turn produces physical effects.
+These physical effects enable the manipulation of objects in contact with the mobile parts of the body.
They take the form of movements or forces.
%Elkoura03: 21 (+6 “wrist”)
%- 4 x (3 extension/flexion + 1 abduction /adduction) => 16
%The human hand has 27 degrees of freedom: 4 in each finger, 3 for extension and flexion and one for abduction and
%adduction; the thumb is more complicated and has 5 DOF,
%leaving 6 DOF for the rotation and translation of the wrist
-The hand has 21 degrees of freedom: 5 for the thumb, and 4 for each of the four other fingers~\cite{elkoura03}.
+The hand has 21 degrees of freedom: 5 for the thumb and 4 for each of the four other fingers~\cite{elkoura03}.
The arms have 7 degrees of freedom: 3 for the shoulders, 1 for the elbow, 1 for the forearm, and 2 for the wrist~\cite{nasa14}.
% 1) Shoulder horizontal Abduction/Adduction %(135/45)
% 2) Lateral/medial shoulder rotation %(46/91)
In our case, we design input systems.
Therefore we need to know the range and precision of movements we have to sense.
-However these are not the only human factors that have an impact on imput systems.
+However, these are not the only human factors that have an impact on input systems.
Users may ignore the way to perform a specific action.
The discoverability~\cite{norman13,pong19} and learnability~\cite{cockburn14,grossman09} of interaction are typical research problems in the HCI field related to this point.
Then the physical actions users have to perform can require high motor skills or training~\cite{schmidt05}.
For example, performing some multiple-key keyboard shortcuts with one hand can be challenging~\cite{pietrzak14}.
-\fixme{Extend this a little?}
+%\fixme{Extend this a little?}
\paragraph{Input systems}
%\defwork{Indirect input} systems measure the movements of an object manipulated by the user.
There are several families of technologies for this.
The list below is not necessarily meant to be exhaustive.
-Rather, the idea is to give readers an overview of today most frequent technologies.
+Rather, the idea is to give readers an overview of today's most frequent technologies.
% for input systems.
%I will describe below the most frequent technologies in input systems.
The first family is electro-mechanical components.
In this family, various types of switches and push buttons sense contacts.
-Keyboard and buttons use variations of this technology.
+Keyboards and buttons use variations of this technology.
Linear and rotary potentiometers sense continuous movements.
Joysticks typically contain two of them to sense rotations in two directions.
Encoders sense discrete movements.
Threshold-based input such as capacitive sensing not only requires adjusting a sensitivity and threshold value, but they often require an \defword{hysteresis} mechanism to avoid multiple activations as well.
Analog signals must be transformed to digital values with an Analog-to-digital converter (\defword{ADC}).
Input values often have noise that must be \defwords{filtered}{filter}.
-There are many possible filters that remove noise, at the cost of latency~\cite{casiez12}.
+Many possible filters remove noise, at the cost of latency~\cite{casiez12}.
Some kinds of input require further transformation.
-In particular, pointing input require a \defword{transfer functions} that computes the movement of the cursor on the screen depending on the physical movements of the input device.
+In particular, pointing input requires a \defword{transfer functions} that computes the movement of the cursor on the screen depending on the physical movements of the input device.
These transfer functions usually take into account the ballistic-then-corrective nature of our movements~\cite{meyer88}.
Vision-based technologies are sensible to occlusions.
Therefore the software part of the pipeline extrapolates data to fill gaps in the input streams.
-The combination of several sources of inputs is challenging as well, but provides more precision in some cases.
+The combination of several sources of inputs is challenging as well but provides more precision in some cases.
For example, data from accelerometers require mathematical integrations for position sensing.
-Not only it requires calibration, but it is also sensible to drifts due to the data precision.
+Not only it requires calibration, but it is also sensible to drift due to the data precision.
The fusion of accelerometers, gyroscopes, and magnetometers provides better tracking, at the cost of increased processing complexity.
-The software processes described in the previous paragraph are either computed on the device or on the host computer.
+The software processes described in the previous paragraph are either computed on the device or the host computer.
For example, the device systematically debounces inputs with analog low-pass filters.
They also implement hysteresis effects with a Schmitt trigger\footnote{\href{https://en.wikipedia.org/wiki/Schmitt_trigger}{https://en.wikipedia.org/wiki/Schmitt\_trigger}}.
The ADCs are either microcontrollers peripherals or dedicated components.
Filters are commonly implemented either on the device firmware or the host drivers.
For example, mouse and touchpad movements are filtered on the device.
%At the opposite, inputs of depth-cameras such as a Kinect\footnote{\href{https://en.wikipedia.org/wiki/Kinect}{https://en.wikipedia.org/wiki/Kinect}} are filtered on the host side, because the host retreives raw data and computes a skeleton for example~\cite{shotton11}.
-At the opposite, inputs of depth-cameras are filtered on the host side, because the host retreives raw data and computes a skeleton for example~\cite{shotton11}.
-The transfer function is typically computed on the host because it requires information about display.
+On the opposite, inputs of depth-cameras are filtered on the host side, because the host retrieves raw data and computes a skeleton for example~\cite{shotton11}.
+The transfer function is typically computed on the host because it requires information about the display.
-When devices are integrated in interactive systems, they are connected to hosts with a simple bus like SPI or I2C\footnote{\href{https://en.wikipedia.org/wiki/Serial_Peripheral_Interface}{https://en.wikipedia.org/wiki/Serial\_Peripheral\_Interface} \href{https://en.wikipedia.org/wiki/I\%C2\%B2C}{https://en.wikipedia.org/wiki/I\textsuperscript{2}C}}.
+When devices are integrated into interactive systems, they are connected to hosts with a simple bus like SPI or I2C\footnote{\href{https://en.wikipedia.org/wiki/Serial_Peripheral_Interface}{https://en.wikipedia.org/wiki/Serial\_Peripheral\_Interface} \href{https://en.wikipedia.org/wiki/I\%C2\%B2C}{https://en.wikipedia.org/wiki/I\textsuperscript{2}C}}.
In this case, the device implements a communication protocol that the host has to follow.
There is no standard protocol, but the overall idea is usually similar.
These buses use no correction codes, therefore they are fast but sensible to interferences.
Similar to the haptic pipeline, the motor pipeline reveals pitfalls that could lead systems to behave differently than what users had in mind.
%The first limitations are due to the limited capacities of humans.
Users may ignore the way to perform they intend to do.
-They can know the the action they have to do but it is challenging to perform.
+They can know the action they have to do but it is challenging to perform.
There can be obstacles in the physical world that prevent systems to sense these actions correctly.
-The range of physical effect can be out of the sensing range of the system.
+The range of physical effects can be out of the sensing range of the system.
The system may interpret what it sensed incorrectly.
This pipeline is therefore a profuse source of HCI research questions.
-I focus here on three categories of research questions that I adressed in my research in the last decade.
+I focus here on three categories of research questions that I addressed in my research in the last decade.
\subsection{Sensing and interpretation}
Interactive systems have a limited set of sensors, which sense a limited subset of the users' actions features.
This is sometimes a desired property: a button is pressed regardless of the actual motion that produced its activation.
When users lift a mouse, its position is not tracked anymore.
-This behaviour enables \defword{clutching}, which extends the motion range of the mouse cursor with the same required physical area.
-Sensing limitations are sometimes constrains for interaction though.
-For example occlusion phenomenons or the limited field of view of vision-based sensors can impede gesture recognition.
+This behavior enables \defword{clutching}, which extends the motion range of the mouse cursor with the same required physical area.
+Sensing limitations are sometimes constraints for the interaction though.
+For example, occlusion phenomenons or the limited field of view of vision-based sensors can impede gesture recognition.
Many factors can have a negative influence on the processing of input signals.
-For example gesture recognition failures provoke interaction errors.
+For example, gesture recognition failures provoke interaction errors.
Beyond errors, the \defword{segmentation} of gestures is a key challenge of gestural interaction because typical gesture sensors observe users all the time, even when users do not want to interact with the system.
The main consequence is the \defword{Midas touch}\footnote{\href{https://en.wikipedia.org/wiki/Midas\#Golden_Touch}{https://en.wikipedia.org/wiki/Midas\#{}Golden\_Touch}} effect that forces users to interact at all times.
In the next chapter, section~\ref{sec:summon}, I will discuss this issue and a mitigation strategy for this specific problem.
-In the section~\ref{sec:lagmeter} of this chapter I will discuss a method and measurement system we designed for characterizing the latency of input systems.
+In the section~\ref{sec:lagmeter} of this chapter, I will discuss a method and measurement system we designed for characterizing the latency of input systems.
\subsection{Input vocabulary}
-The input vocabulary follows a similar structure than the output vocabulary.
-We describe it with the lexical, syntactic, and semantic levels of languages as well.
+%The input vocabulary follows a similar structure than the output vocabulary.
+The structure of the input vocabulary is similar to the structure of the output vocabulary.
+We describe it with the lexical, syntactic, and semantic levels of languages.
I typically explain this with the example of the computer mouse because everybody used it and knows its interactive language.
Its lexical elements are \defwords{degrees of freedom}{degree of freedom} (DOF): x-y motion and button presses and releases.
They have associated data: relative displacement value on both axes, and the ID of the button that was pressed or released.
The syntactic level consists in assembling these elements to form input phrases.
-A clic is a button press followed by the release of the same button.
-There are many variations, typically a double clic that repeats a clic twice, or half-clics that replace the release by dwell.
+A click is a button press followed by the release of the same button.
+There are many variations, typically a double click that repeats a click twice, or half-clicks that replace the release by dwell.
Finally, the semantic level associates input phrases to commands and parameters.
-For example a press on an object followed by a movement and a release on another object is typically interpreted as moving the first object into the second one.
+For example, a press on an object followed by a movement and a release on another object is typically interpreted as moving the first object into the second one.
When designing a new input device, we have to specify its DOFs, their type, range, precision, and if they are integrated with each other~\cite{mackinlay90}.
For example, the x and y DOFs of a mouse are integrated because users control them together with a single movement.
%The USB HID descriptor of a mouse tells the host computer information such as the range of the values it sends, the logical values it maps to, the state of the buttons, as well as the frequency at which it sends packets.
%Such information is specific to every input device.
-The design process of an input device consists in cycles between engineering and evaluation in order to choose and document this information related to DOFs.
-In the~\ref{sec:flexiblepens} section I will describe the design of flexible pens, and in the section~\ref{sec:fingeridentification} I will explain how we leveraged finger identification for multi-touch interaction.
+The design process of an input device consists of cycles between engineering and evaluation in order to choose and document this information related to DOFs.
+In \refsec{sec:flexiblepens} I will describe the design of flexible pens, and in \refsec{sec:fingeridentification} I will explain how we leveraged finger identification for multi-touch interaction.
%design of an input device requires evaluations of the users' ability to manipulate it.
%Not only this evaluation guides the design of the device, but it also validates…
%informed choice
The term \defword{natural interaction} was already used before multitouch and gestural interaction became mainstream~\cite{bolt80}.
-These interaction styles are integrated in ubiquitous consumer electronics products such as smartphones and tablets for more than a decade now.
+These interaction styles are integrated into ubiquitous consumer electronics products such as smartphones and tablets for more than a decade.
%It took two decades for the technology to be ready for consumer electronics products
-They are nowadays refered to as natural user interfaces or NUI~\cite{wigdor11}.
+They are nowadays referred to as natural user interfaces or NUI~\cite{wigdor11}.
%Whether it is about food, energy, or interaction, the word \emph{natural} suggests the idea that the object it refers to is good for people, their health, or the environment for example.
%It is the idea that what was made by the nature is better than artificial artifacts, which are made by humans.
The first question is: what do we mean by “natural”?
It primarily refers to what exists in the nature.
-However gestures are not objects that can be found in the wild.
+However, gestures are not objects that can be found in the wild.
It also refers to what comes to mind, or prior skills.
People have skills in manipulating physical objects that we can leverage in interactive systems.
-Yet, people have different backgrounds, skills, and culture.
-Therefore what comes to somebody's mind is not necessarily the same than what somebody else is thinking.
+Yet, people have different backgrounds, skills, and cultures.
+Therefore what comes to somebody's mind is not necessarily the same as what somebody else is thinking.
The second question is: is it better because it is natural?
-This is a vast question because it depend on what we would like to improve: performance, learnability, guessability, etc.
+This is a vast question because it depends on what we would like to improve: performance, learnability, guessability, etc.
There is no general guarantee that gestures improve any of these measures.
They have to be carefully designed for this.
This is the reason why decades have passed between the first gestural interaction systems and the first successful commercial products based on gestural interaction.
-Norman discussed the concept of NUIs and concludes that what matters is not whether these interactions are natural or not~\cite{norman10}.
+Norman discussed the concept of NUIs and concluded that what matters is not whether these interactions are natural or not~\cite{norman10}.
What matters is that it is an alternative interaction modality for the design of interactive systems, and the same usability rules apply to them.
Now, if the natural essence of gestural interaction is not an essential benefit for the design of interactive systems, we can think about this modality differently.
-%The physical world has constrains that the digital world does not have.
+%The physical world has constraints that the digital world does not have.
The digital world does not have some of the limitations of the physical world.
For example, we can easily teleport inside a virtual environment.
We can move through objects or even fly.
-We can manipulate objects remotely, and independently to their weight, size, or shape.
-Hence, when when designing interactive systems with gestural interaction there is no necessity to reproduce the physical world.
+We can manipulate objects remotely and independently to their weight, size, or shape.
+Hence, when designing interactive systems with gestural interaction there is no necessity to reproduce the physical world.
We should rather focus on what we would like users to achieve, and improve performance, learnability, and so on.
-On a practical point of view, we care about the input vocabulary, the properties we would like to manipulate in the virtual environment, and the mapping between them.
+From a practical point of view, we care about the input vocabulary, the properties we would like to manipulate in the virtual environment, and the mapping between them.
%mapping between DOFs and objects manipulated
-For example, in the section~\ref{sec:interactionvr} we present two interaction techniques for immersive virtual environments.
+For example, in \refsec{sec:interactionvr} we present two interaction techniques for immersive virtual environments.
The first one is a selection technique for distant objects.
-The second one enable users to select facial expressions for ther avatar without the need to perform the same expression with their own face.
+The second one enable susers to select facial expressions for their avatar without the need to perform the same expression with their own face.
%Raycursor, facial expressions
\section{Contributions}
-With these research questions in mind, I will describe now some of my contributions leveraging the users' motor abilities for the design of interactive systems and interaction techniques.
-Instead of presenting the research below the way it was described in research papers, I will focus on hindsights and how these contributions influenced my research.
+With these research questions in mind, now I will describe some of my contributions leveraging the users' motor abilities for the design of interactive systems and interaction techniques.
+Instead of presenting the research below the way it was described in research papers, I will focus on hindsight and how these contributions influenced my research.
\subsection{Input latency measurement}
The first contribution I will present is not an interaction technique or an interactive system.
It is a methodology and study about the latency of touch-based interactive systems in the general sense.
Every interactive system has a delay between the moment users perform an action and when the system produces a response.
-This delay is refered to as \defword{end-to-end latency}.
+This delay is referred to as \defword{end-to-end latency}.
This latency is known to cause performance and usability issues~\cite{deber15,jota13,teather09,waltemate16}.
Therefore, there are research studies about strategies to mitigate these effects or reduce latency~\cite{cattan15,nancel18}.
\input{figures/lagmeter.tex}
-Before reducing latency or mitigating its effects, it is important to measure it, and understand the contribution of each part of the system to it.
+Before reducing latency or mitigating its effects, it is important to measure it and understand the contribution of each part of the system to it.
There are many possible sources of latency: input and output, software and hardware, device and host.
Typical latency measurement methods consisted in counting frames on videos made with a high-speed camera~\cite{ng12,steed08,teather09}.
-This method is simple, but tedious.
-It is difficult to make large series of measurements and it is impossible to identify the main sources of latency.
+This method is simple but tedious.
+It is difficult to make large series of measurements and it is impossible to identify the main sources of latency.
Other methods enable repetitive measures, but they are adapted to specific input types~\cite{casiez15,deber16}.
They depend on the system running on the host computer.
The moment users touch the input device, a piezoelectric vibration sensor\footnote{\href{http://www.te.com/usa-en/product-CAT-PFS0011.html}{http://www.te.com/usa-en/product-CAT- PFS0011.html}} attached to the fingertip detects the contact.
-The idea is that all touch-based input require an initial contact.
+The idea is that all touch-based inputs require an initial contact.
We connected this sensor to a custom electronic board connected to an Arduino Leonardo\footnote{\href{https://www.arduino.cc/en/Main/Arduino_BoardLeonardo}{https://www.arduino.cc/en/Main/Arduino\_BoardLeonardo}} microcontroller board.
-The board enables adjustmenting the detection threshold since the voltage we get from the piezo sensor is typically lower than $0.5V$.
+The board enables the adjustment of the detection threshold since the voltage we get from the piezo sensor is typically lower than $0.5V$.
The Arduino sends a Raw HID message to the host computer when it detects contacts.
In parallel, the input device sends an event HID packet to the host computer.
A custom application gets the RawHID packet with HIDAPI\footnote{\href{https://github.com/signal11/hidapi}{https://github.com/signal11/hidapi}}, and the device HID packet with libpointing~\cite{casiez11a}.
\subsubsection{Results and discussion}
-What I realized with this project is that nothing in interaction can be considered as instantaneous.
+What I realized with this project is that nothing in interaction can be considered instantaneous.
When we measured the responsiveness of the piezoelectric vibration sensor, we noticed that even the mechanical contact of a physical button creates latency at the millisecond scale.
It is both due to the elasticity of the finger, and the mechanical actuation of the button switch.
-We made this measurement with an aluminium foil around a finger, tapping on a copper tape pasted on a mouse button.
+We made this measurement with an aluminum foil around a finger, tapping on a copper tape pasted on a mouse button.
The finger also had a piezoelectric sensor strapped around its tip.
-The \reffig{fig:piezotrigger} shows a measurement with three signals: the finger contact in blue, the button press in red and the piezo signal in green.
+The \reffig{fig:piezotrigger} shows a measurement with three signals: the finger contact in blue, the button press in red, and the piezo signal in green.
We notice it takes about $2ms$ for the button to switch.
The delay of the piezo trigger depends on the threshold, but it typically takes between $1ms$ and $3ms$.
-The paper contains more detail data about these measures~\cite{casiez17}.
+The paper contains more detailed data about these measures~\cite{casiez17}.
\input{figures/piezotrigger.tex}
The measurement of the screen change was more accurate.
The idea was to switch the screen between black and white so that we can detect the change with a simple luminosity sensor.
We used a photodiode because it is fast.
-The \reffig{fig:flicker} shows the signal when the screen turned white.
+\reffig{fig:flicker} shows the signal when the screen turned white.
Interestingly, the measure is so fast that we can observe the lines being drawn around the sensor.
-Thereforer the position of the photodiode on the screen has an impact on the latency we measure.
+Therefore the position of the photodiode on the screen has an impact on the latency we measure.
However, depending on the adjustment of the detection threshold, we can reliably measure the screen change in $4\mu s$.
-This is way sufficient for our purpose, and makes the actual position of the sensor less critical.
+This is way sufficient for our purpose and makes the actual position of the sensor less critical.
\begin{figure}[htb]
\includegraphics[width=\columnwidth]{figures/screenflicker}
Latency slicing requires having all measures with the same clock.
This was an issue because part of the process was done on the Arduino board, and another part on the host computer.
It means we needed a way to communicate in a fast and reliable way between the two.
-The problem is that typical operating systems (Windows, Linux and MacOS) cannot guarantee the execution of code with a $1ms$ precision.
+The problem is that typical operating systems (Windows, Linux, and MacOS) cannot guarantee the execution of code with a $1ms$ precision.
One solution was to use a Raspberry Pi\footnote{\href{https://www.raspberrypi.org/}{https://www.raspberrypi.org/}} as a host computer because it has GPIOs that can communicate with microcontrollers with low latency.
-However, measures with this platform would not necessarily be representative to the usage of other platforms.
-Thus we considered several communication channels between a microcontroller and a host computer such as ethernet or a parallel port with a specific extension card.
-Nevertheless, we opted for raw HID messages because its latency was sufficiently low and consistent.
+However, measures with this platform would not necessarily be representative of the usage of other platforms.
+Thus we considered several communication channels between a microcontroller and a host computer, such as ethernet or a parallel port with a specific extension card.
+Nevertheless, we opted for raw HID messages because their latency was sufficiently low and consistent.
The round-trip delay between the Arduino and the host computer was between $1s$ and $2s$, depending on the operating system and the processor load.
%Details are available in the paper.
Concerning the input, we compared different input frequencies with a Logitech G9 mouse that enables the adjustment of the mouse frequency.
%: $125Hz$, $250Hz$, $500Hz$ and $1000Hz$.
%The difference of latency is small.
-The host computer received the device HID packet after $3.3ms$ in average for $125Hz$ and under a millisecond for frequencies above $250Hz$.
+The host computer received the device HID packet after $3.3ms$ on average for $125Hz$ and under a millisecond for frequencies above $250Hz$.
The latency between the moment the device HID packet was received and the moment the application requested the repaint of the screen was about $3ms$.
-The end-to-end latency was aroung $60ms$, therefore the input part of the pipeline has little impact on latency.
+The end-to-end latency was around $60ms$, therefore the input part of the pipeline has little impact on latency.
%The system and toolkit part are fast as well.
Most of the latency is due to the output.
We observed several factors that impact latency: the operating system, graphic toolkit, and screen frequency.
We used a minimal application for our measures, but real applications most likely introduce more latency because of their normal operations.
-%note: minimal application, more latency with a real applicaiton
+%note: minimal application, more latency with a real application
%expected latency: hid packet at 1kHz, capacitive measure, event queue, display, repaint
%Precision of measure, precision of communication between arduino and host, need to have a common clock
\subsection{Flexible pens}
\label{sec:flexiblepens}
-Pen interaction is a good example for the exploration of new degrees of freedom.
-This is certainly due to the fact that pens are used in many context, in particular for artistic creation.
+Pen interaction is a good example of the exploration of new degrees of freedom.
+This is certainly because pens are used in many contexts, in particular for artistic creation.
Typical interactive pens sense the x-y position as well as proximity.
Research explored additional sensing such as pressure~\cite{hinckley13}, tilt~\cite{tian08} and roll~\cite{bi08}.
Not only this extended input vocabulary can be used to map brush parameters.
-But it also enables selecting commands and offer richer interactions, whether it is with combinations of pen and touch interactions~\cite{hinckley10} or by leveraging physical attributes of the pen~\cite{vogel11}.
-In this work we were interested in the bending of a flexible pen as additional degrees of freedom.
+But it also enables selecting commands and offers richer interactions, whether it is with combinations of pen and touch interactions~\cite{hinckley10} or by leveraging physical attributes of the pen~\cite{vogel11}.
+In this work, we were interested in the bending of a flexible pen as additional degrees of freedom.
\subsubsection{Prototypes}
The choice of the flexible part did not follow a systematic rationale, but rather general design considerations.
The idea was to have a diameter similar to the one of a drawing pen.
It had to be flexible enough to avoid muscle strain, but stiff enough so that users could write and draw conveniently.
-Full length flexible early prototypes shown to be unconvenient for precise manipulation.
-Therefore we chose to limit the flexible part to a few centimeters with a rigid part on both size to keep the benefits of both flexible and rigid pens.
-The rigid part between the tip and the flexible part is long enough to enable users gripping the stylus there.
+Full-length flexible early prototypes were shown to be inconvenient for precise manipulation.
+Therefore we chose to limit the flexible part to a few centimeters with a rigid part on both sides to keep the benefits of both flexible and rigid pens.
+The rigid part between the tip and the flexible part is long enough to enable users to grip the stylus there.
\begin{figure}[htb]
\centering
It provided more precise and reliable inputs.
However, the interesting new property of these prototypes was the ability to change the flexural stiffness with interchangeable components.
Both the rigid and flexible parts of the pen are 3D printed, and the flexible part is threaded so that it is screwed to the rigid parts.
-The end sensor slips inside the 3D printed stylus.
+The end sensor slips inside the 3D-printed stylus.
The FlexStylus prototypes had an orientation issue.
Users had no cues on where to hold the stylus.
Therefore the angles inputs were relative rather than absolute.
This is the same issue Buxton reports with the iMac Round Mouse\footnote{\href{https://www.microsoft.com/buxtoncollection/detail.aspx?id=109}{https://www.microsoft.com/buxtoncollection/detail.aspx?id=109}}.
-We addressed this issue with the HyperBrush prototypes by adding a fake button that users were encouraged to keep under their index finger.
-The relief of this button helped users keeping the stylus in a consistent orientation eyes-free.
+We addressed this issue with the HyperBrush prototypes by adding a fake button that users were encouraged to keep under their index fingers.
+The relief of this button helped users keep the stylus in a consistent orientation eyes-free.
\subsubsection{Pen grips}
-Studies in the literature show that there is a strong connection betweeen the way users hold a stylus and the way they use the stylus~\cite{song11,hinckley13}.
+Studies in the literature show that there is a strong connection between the way users hold a stylus and the way they use the stylus~\cite{song11,hinckley13}.
They noticed that participants in their user studies often changed the way they gripped the stylus depending on the task they were performing.
We took inspiration from this work, and we designed our prototypes for three kinds of grips.
\begin{itemize}
-\item \textbf{Tool grip}: users hold the stylus the same was they hold a pen.
+\item \textbf{Tool grip}: users hold the stylus the same way they hold a pen.
This is actually a family of grips, but the idea is that users hold the rigid part of the pen close to the tip.
They can press the pen with their thumb to bend it.
\item \textbf{Menu grip}: users hold the stylus from the rigid part on the other side of the flexible part.
The tip remains still on the surface thanks to friction.
This way, users can control an orientation vector around the contact point.
\item \textbf{In-air grip}: users hold the stylus in the air between their fingers.
- They can bend the stylus by squeezing their fingers, or roll the pen.
+ They can bend the stylus by squeezing their fingers, or rolling the pen.
\end{itemize}
When we evaluated our second series of prototypes some participants reported drawing with the menu grip, after the flexible part.
Drawing this way gave them the impression of painting with a brush, hence the name \emph{Hyperbrush}.
-%This kind of serendipitous behaviour is typically what we hope we will get with such a research project.
+%This kind of serendipitous behavior is typically what we hope we will get with such a research project.
This is a difficulty with this type of work.
-We need a design rationale to guide the implemention before we evaluate the device.
-We evaluate the interaction with the device to make sure that users are able to use it the way we wanted them to use it.
-However, we want to let such serendipitous behaviour happening because it makes interaction with the device richer.
+We need a design rationale to guide the implementation before we evaluate the device.
+We evaluate the interaction with the device to make sure that users can use it the way we wanted them to use it.
+However, we want to let such serendipitous behavior happen because it makes interaction with the device richer.
\begin{figure}[htb]
\includegraphics[height=5cm]{figures/flexstylus-pengrip}
\subsubsection{Flexural stiffness}
-We evaluated the users' ability to control the bending of FlexStylus, and compared it to pressure input~\cite{fellion17}.
+We evaluated the users' ability to control the bending of FlexStylus and compared it to pressure input~\cite{fellion17}.
We found out that the participants of our studies were more precise with the flexible pen than with pressure input.
-Pressure pens are isometric devices therefore it is unsurprising that they do not perform well with position control~\cite{zhai93,zhai93a}.
+Pressure pens are isometric devices, therefore it is unsurprising that they do not perform well with position control~\cite{zhai93,zhai93a}.
Flexstylus can be considered as an elastic device.
Therefore position control can be an issue.
However, the classification of a flexible pen between an isotonic or elastic device essentially depends on its flexural stiffness.
Therefore with the next generation of flexible styluses, HyperBrush, we wanted to compare different flexural stiffnesses.
We built several prototypes of different flexural stiffness (\reffig{fig:flexuralstiffnesses}).
-The flexural stuffness is adjusted with the thickness of the flexible parts inner walls.
+The flexural stiffness is adjusted with the thickness of the flexible parts inner walls.
%The thicker the harder
-The measure of flexural stiffness we used is the ratio of amount of force applied per unit of deflection.
+The measure of flexural stiffness we used is the ratio of the amount of force applied per unit of deflection.
We measured it with a hollowed cylinder cantilever beam test.
-We compared the performance and subjective preferences between these different flexural stiffnesses and a pressure pen for pie-menu item selection and for brush stroke precision.
-The results does not show any clear better configuration.
+We compared the performance and subjective preferences between these different flexural stiffnesses and a pressure pen for pie-menu item selection and brush stroke precision.
+The results do not show any clear better configuration.
We also collected subjective preferences with a drawing task.
-The results suggest that the best is to provide users several interactive pens that provide different kinds of input vocabularies and haptic feedback.
-Users will choose the one they feel appropriate for their current task.
+The results suggest that the best is to provide users with several interactive pens that provide different kinds of input vocabularies and haptic feedback.
+Users will choose the one they feel is appropriate for their current task.
%evaluation, preferences, recommendations
\input{figures/flexuralstiffnesses.tex}
-
-
\subsection{Finger identification}
\label{sec:fingeridentification}
-Hands and fingers have many degrees of freedom and most of people have enough skills and dexterity to perform gestures and manipulate objects.
+Hands and fingers have many degrees of freedom and most people have enough skills and dexterity to perform gestures and manipulate objects.
Therefore hands are widely used for interacting with interactive systems.
Yet, consumer electronics interactive devices only use a limited subset of gesture properties.
-Typically multi-touch devices essentially use the contacts coordinates and movement, number of contacts, and recently pressure.
+Typically multi-touch devices essentially use the coordinates and movement of the contacts, number of contacts, and recently pressure.
It already enables complex interaction techniques, in particular for command selection~\cite{bailly10}.
Further studies leverage the raw data of multi-touch surfaces, either from cameras or capacitive sensors.
-With this data we can detect contact blobs instead of single coordinates, which increases the input vocabulary \cite{roudaut09}.
+With this data, we can detect contact blobs instead of single coordinates, which increases the input vocabulary \cite{roudaut09}.
Other research investigated technologies that sense other gestures properties such as the part of the finger touching the surface \cite{harrison11}, make a distinction between contacts from different users \cite{dietz01}, or analyze the hand posture \cite{murugappan12}.
This has been an active research area in the last decades, therefore this is just a tiny overview of what has been done on this topic.
-In this work we were interested \defword{finger identification}.
+In this work, we were interested \defword{finger identification}.
It is an additional hand gesture property, which says which finger of which hand produced a given contact point.
-There is still no technology that sense this property directly in consumer electronic products.
+There is still no technology that senses this property directly in consumer electronic products.
A workaround with existing technologies is to ask users to press all fingers before releasing some of them \cite{lepinski10}.
-Other projects use a different processing of sensed data.
+Other projects use different processing of sensed data.
For example, the old generations tabletops used an infrared projector under a translucent surface, pointed towards the user.
An infrared camera sensed the reflection of the infrared light that created blobs at contact points.
The multi-touch coordinates were computed as the centroid of these blobs.
However, the whole hand is visible on this image and it is possible to identify fingers with image processing of this data \cite{ewerling12}.
-Several other studies in the literature use heuristics to detect detect finger chords.
-They detect the palm of the hand and deduce the position of fingers \cite{bailly08} or make assumptions based on the relative position of contact points \cite{ghomi13,wagner14}.
+Several other studies in the literature use heuristics to detect finger chords.
+They detect the hand palm and deduce the position of fingers \cite{bailly08} or make assumptions based on the relative position of contact points \cite{ghomi13,wagner14}.
Other research investigated finger identification with prototyping technologies: optical fibers under a touchpad that enable the capture of fingerprints \cite{holz13}, a glove with piezoelectric vibration sensors \cite{masson17}, or a glove with fiducial markers \cite{marquardt11}.
%Fiberio for example uses optical fibers under a touchpad that enable the capture of fingerprints, hence identifying fingers \cite{holz13}.
-%Whichfingers is another solutiuon that does works with any multi-touch interfaces, it uses a glove with piezoelectric vibration sensors on each finger \cite{masson17}.
-In the next section we will discuss the finger identification sensing technologies we used in our studies.
+%Whichfingers is another solution that does works with any multi-touch interfaces, it uses a glove with piezoelectric vibration sensors on each finger \cite{masson17}.
+In the next section, we will discuss the finger identification sensing technologies we used in our studies.
One of the simplest applications of finger identification is probably mapping a different command to every finger.
-Research showed that this is already sufficient to increase the input thoughput of touch interaction \cite{roy15}.
+Research showed that this is already sufficient to increase the input throughput of touch interaction \cite{roy15}.
This is particularly useful because multitouch applications typically cannot provide as many commands as desktop applications.
-For example, in 2014 Wagner \etal report that there were 648 commands in the menus of the desktop version of Adobe Photoshop compared to 35 commands on the tablet version.
-Interaction techniques that use finger identification is a solution.
+For example, in 2014 Wagner \etal reported that there were 648 commands in the menus of the desktop version of Adobe Photoshop compared to 35 commands on the tablet version.
+Interaction techniques that use finger identification are a solution.
However, the input space is huge: $2^{10}-1$ fingers chords, to which we can add gestures.
-We will discuss how we leverage this new input space in a systematic way, in particular how we reduce this input space.
-
+We will discuss how we systematically leverage this new input space, in particular how we reduce this input space.
\subsubsection{Prototyping apparatus}
-This project is a typical HCI project in which we investigate the benefits and limitations of an interaction paradigm before the technology is ready to implement it in consumer electronics products.
-The advantage is that research for such a technology will happen only if and when the benefits will balance the costs.
+This project is a typical HCI project in which we investigate the benefits and limitations of an interaction paradigm before the technology is ready to be implemented in consumer electronics products.
+The advantage is that research for such technology will happen only if and when the benefits will balance the costs.
Even if such technology is not ready yet, the HCI community regularly uses alternative technologies that necessarily make compromises.
-In some cases we just imagine the technology is there and works perfectly.
-The wizard of Oz technique consists in having an operator executing the actions on the users behalf, with any other technology.
+In some cases, we just imagine the technology is there and works perfectly.
+The wizard of Oz technique consists in having an operator executing the actions on the users' behalf, with any other technology.
This is not always even necessary.
In the case of finger identifications, the Glass+Skin study just indicated participants the finger they wanted them to use.
Other methods make assumptions on the inputs \cite{ghomi13,wagner14}, but it limits the resulting interaction techniques.
We can also use alternative technologies.
Either they only work for specific devices, like projection-based tabletops, or they use invasive methods like markers.
-We prefer the last method since it allowed us to build prototypes for tabletops, tablets and smartphones, without limitations on the input vocabulary.
+We prefer the last method since it allowed us to build prototypes for tabletops, tablets, and smartphones, without limitations on the input vocabulary.
\begin{figure}[htb]
\centering
The first one is the 2D coordinates of the contact points.
The second one is the 3D position of every finger that we sense with GameTraks\footnote{\href{https://en.wikipedia.org/wiki/Gametrak}{https://en.wikipedia.org/wiki/Gametrak}}.
GameTracks sense a 3D position by measuring the length and angles of a string attached to a joystick.
-This method requires a calibration and strings attached to every finger, but it is not sensible to occlusion for example.
+This method requires calibration and strings attached to every finger, but it is not sensible to occlusion for example.
The tablet and smartphone prototypes used color markers on every finger, as well as an external camera.
It requires calibration as well and it is sensible to occlusion.
However, having no strings attached to the fingers is more convenient for interacting on smaller devices.
These prototyping technologies are clearly not usable in consumer electronics products.
-But they are robust enough and add little constrains on multi-touch interaction, so they are convenient for exploring the input space of multi-touch interaction with finger identification.
+But they are robust enough and add little constraints on multi-touch interaction, so they are convenient for exploring the input space of multi-touch interaction with finger identification.
\subsubsection{Input vocabulary}
In this project we leveraged the input vocabulary of finger identification for command selection as well as the direct manipulation of the command parameters \cite{goguey14,goguey14a,goguey17}.
Without finger identification, the input space is just the number of contacts, between 1 and 10 for one user, and coordinates for each contact point.
-With finger identification, the theoretical input space much larger: each finger can either be pressed or not, which leads to $2^{10}-1$ possibilities.
+With finger identification, the theoretical input space is much larger: each finger can either be pressed or not, which leads to $2^{10}-1$ possibilities.
However, chords with too many fingers are less likely used.
-For a given number of fingers $k$ there are still $C_k^{10}$ possible combinations of two-hands chords.
+For a given number of fingers $k$, there are still $C_k^{10}$ possible combinations of two-hands chords.
This leads to 45 combinations of 2 fingers and 120 combinations of 3 fingers.
People cannot perform all chords efficiently because of biomechanical constraints~\cite{ghomi13}.
However, this input space remains still high therefore the design of a command selection technique requires a systematic approach.
Different combinations of modifier keys change the command activated with a given alphanumeric key.
Then, when the associated action is being executed the users can apply constraints with modifier keys.
The \reffig{fig:hotfingers} shows how we applied this principle to multi-touch interaction with finger identification.
-The users start with touching the surface with their left hand.
+The users start by touching the surface with their left hand.
Some of the possible chords enable the selection of commands with the right hand.
A crib sheet shows icons representing the five commands mapped to each of the five fingers of the right hand.
In this example pressing both the thumb and index finger maps drawing commands to fingers.
-The user draws a rectangle by using the command associated to the middle finger.
-The initial contact point of the middle finger of the right hand defines one corner of the rectangle, and the opposite corner is adjusted continuously with direct manipulation.
+The user draws a rectangle by using the command associated with the middle finger.
+The initial contact point of the middle finger of the right-hand defines one corner of the rectangle, and the opposite corner is adjusted continuously with direct manipulation.
During this step, the users can lift their left hand and continue to adjust the rectangle.
Now, they may want to apply constraints.
To do so, they touch the surface with the left hand.
-In this example the middle finger sets the initial contact point as the center of the rectangle rather than the first corner.
+In this example, the middle finger sets the initial contact point as the center of the rectangle rather than the first corner.
Another crib sheet shows an icon representing this constraint.
%users can lift the right hand during this process but also put it back to apply constraints, like perfect shape, etc.
One of the key issues of this kind of system is the discoverability of the available commands.
The \reffig{fig:hotfingers-helps} shows the visual feedforward items we use to promote the discoverability of commands.
On the left, when all fingers touch the surface, bubbles show all the possible chords associated with modifiers or commands.
-On the left, crib sheets represent the visual icons of available commands or or constraints for the current chord of the left hand.
-All this visual feedforward information is displayed around the corresponding contact points, and follow them continuously when they move.
+On the left, crib sheets represent the visual icons of available commands or constraints for the current chord of the left hand.
+All this visual feedforward information is displayed around the corresponding contact points and follows them continuously when they move.
The actual crib sheets are updated continuously as fingers touch or leave the surface.
Linear menus and toolbars in desktop applications show, in principle, all the available commands of an application.
In our case, the visual representation of commands is only available on demand.
-The command mapping and constraints change in real time as the users press or releases fingers.
+The command mapping and constraints change in real-time as the users press or release fingers.
It encourages users to explore the system and discover the available commands.
This exploration would be an issue if the explorable input space was too large.
However, we essentially use one-hand chords.
There are therefore $C_k^5$ possible chords with $k$ fingers, which gives 5 combinations of one or four fingers, and 10 combinations of 2 or 3 fingers.
Therefore there are $5+10+10+5=30$ modifier chords for a maximum of $150$ single-finger commands with the right hand.
-We assume this is a reasonnable size for an explorable input space.
+We assume this is a reasonable size for an explorable input space.
However, we did not conduct a user study to evaluate this aspect and validate this claim.
\begin{figure}[htb]
\caption[Discoverability features for multi-touch interaction with finger identification.]{Discoverability features for multi-touch interaction with finger identification. Users can see all the available commands when touching the surface with all their fingers. Crib sheets show the available commands and constraints when fingers of the left hand are touching the surface.}
\end{figure}
-Keyboard shortcuts with multiple modifier keys are sometimes difficult to execute because the keys position is fixed.
+Keyboard shortcuts with multiple modifier keys are sometimes difficult to execute because the position of the keys is fixed.
Therefore sometimes it requires users to stretch their fingers to reach all the keys, especially if they want to keep one hand on the mouse.
-In this context we studied the possibility to select modifiers on the mouse so that the left hand only has to select the alphanumeric key~\cite{pietrzak14}.
+In this context, we studied the possibility to select modifiers on the mouse so that the left hand only has to select the alphanumeric key~\cite{pietrzak14}.
The input space remained limited, especially because we only had two modifier buttons on the mouse.
-Il the case of our command selection technique for multi-touch interaction with finger identification, the advantage is that there is no actual button to press.
+In the case of our command selection technique for multi-touch interaction with finger identification, the advantage is that there is no actual button to press.
Therefore, the location of the contact point does not matter.
Hence, users can touch the surface with a comfortable posture.
Immersive virtual reality has been around since the beginning of personal computing \cite{sutherland65,sutherland68}.
Early prototypes already included a stereovision headset with head tracking to match the users' movements with their position in the virtual environment.
Yet, it took VR headsets much longer than personal computers to reach either households or professional environments.
-Even today that many different consumer electronics VR headsets are available at a reasonnable price, their usage remains limited.
+Even today that many different consumer electronics VR headsets are available at a reasonable price, their usage remains limited.
Similarly to every interactive technology, the objective is not to replace completely concurrent technologies, but rather find the particular application scenarios for which this technology is more suitable than others.
To do so, we need to know better how we interact in virtual environments.
%However, more research is still necessary to discover the benefits and best ways to interact in immersive virtual environments.
The studies I will describe in this section focus on input methods for immersive virtual reality.
We take immersive virtuality reality as a context that constrains the input methods we can use, in which users have to perform specific tasks.
-One of these constraints is that the users cannot see their own body because the virtual environment covers their entire field of view.
-Therefore, either a motion capture system sense the users' body movements, or they hold input devices in their hands.
+One of these constraints is that the users cannot see their own bodies because the virtual environment covers their entire field of view.
+Therefore, either a motion capture system senses the users' body movements, or they hold input devices in their hands.
Users interact with their environment mostly with gestures, but also with buttons, touchpads, and joysticks on handheld devices.
-As we discussed at the beginning of this chapter, the objective in virtual reality is not necessarily to reproduce what exists in the physical world.
+As we discussed at the beginning of this chapter, the objective in virtual reality is not always to reproduce what exists in the physical world.
We rather focus on either elementary or compound tasks and figure out ways to enable users to perform these tasks.
-For example selecting an object is an elementary task that is usually part of a more complex or compound tasks: manipulation, command selection, navigation, text entry, etc.
+For example, selecting an object is an elementary task that is usually part of more complex or compound tasks: manipulation, command selection, navigation, text entry, etc.
We describe below the design and evaluation of a new 3D pointing technique for immersive virtual reality.
% Should I keep this?
One of the main active research topics in this area is augmenting \defword{immersion}, \defword{presence}, and \defword{embodiment} \cite{witmer98,slater99,kilteni12}.
-In short, these three concepts are related to the users's sensation of being in the virtual environment.
+In short, these three concepts are related to the users' sensation of being in the virtual environment.
I will cover this topic in the next chapter, section \ref{sec:embodiment}.
%
Yet, a typical way to increase the users' sensation of embodiment is to provide them an \defword{avatar} that they can control.
The users can typically control their avatar's hands and head because it is easy to map their position to the VR headset and the controller’s position with inverse kinematics.
-The facial expression of the avatar is important for non verbal communication.
+The facial expression of the avatar is important for non-verbal communication.
However, it is more complex to control than the head and hands because it requires many degrees of freedom.
Therefore we describe below the design and evaluation of an interaction technique for the control of the facial expression of an avatar in immersive virtual environments.
\subsubsection{3D Pointing}
-In the physical world it is difficult to interact with objects remotely because of physical constraints.
-In virtual worlds such physical constraints do not exist.
-Users can inteteract remotely with objects, regardless of their size, shape, or weight.
+In the physical world, it is difficult to interact with objects remotely because of physical constraints.
+In virtual worlds, such physical constraints do not exist.
+Users can interact remotely with objects, regardless of their size, shape, or weight.
The most common technique for selecting objects is certainly \defword{raycasting} \cite{bowman04}.
-The users control a ray with their hand, and the first interested object can be selected with a validation action, such as a button press.
-It shares similarity with laser pointers, except that the light ray between the controller and the contact point is visible.
+The users control a ray with their hand, and the first intersected object can be selected with a validation action, such as a button press.
+It shares similarities with laser pointers, except that the light ray between the controller and the contact point is visible.
The simplicity of this technique has a cost.
-Occluded targets cannot be selected, or require users to get around it.
+Occluded targets cannot be selected, or require users to get around them.
Moreover, even if theoretically there is no limit to the distance and size of the object the users would like to select, in practice hand tremor and input noise create such a limit.
Therefore, there are many adaptations of this technique in the literature, well covered in \cite{argelaguet13}.
Most of these techniques use a disambiguation step to let the users select a target among all the interested targets \cite{grossman06,kopper11,delamare13}.
-Other techniques use a cursor on the ray that usrs can either manipulate by manipulating the ray \cite{grossman06}, or with another degree of freedom \cite{ro17}.
+Other techniques use a cursor on the ray that users can either manipulate by manipulating the ray \cite{grossman06}, or with another degree of freedom \cite{ro17}.
%Either the cursor has a fixed position that users control by manipulating the ray \cite{grossman06}.
-%Or the users can control the position of the cursor with an additional degree of fredom \cite{ro17}.
+%Or the users can control the position of the cursor with an additional degree of freedom \cite{ro17}.
-These solution fix the issue of occluded targets but not the issue of small and distant targets.
-To do so we propose RayCursor, an alternative of raycasting with a cursor that users can control that leverages proximity selection \cite{baloup19,baloup18,baloup19a}.
+These solutions fix the issue of occluded targets but not the issue of small and distant targets.
+To do so, we propose RayCursor, an alternative of raycasting with a cursor that users can control that leverages proximity selection \cite{baloup19,baloup18,baloup19a}.
Proximity selection consists in selecting the nearest target from the cursor instead of the intersected target \cite{grossman05}.
Initial studies were with 2D pointing.
However, it was also studied for 3D pointing with a virtual hand \cite{vanacken07}, that is to say pointing at targets the users can reach around them.
We combined this approach and raycasting with a cursor.
For the design of this technique, we performed three initial evaluations.
-The first one was about the most appropriate visual feedforward, and was inspired by a similar work for 2D proximity selection \cite{guillon15}. The second evaluation was about the transfer function for the movement of the cursor, and the last one was about the benefits of filtering the inputs that control the ray with a 1\euro~filter~\cite{casiez12}.
+The first one was about the most appropriate visual feedforward and was inspired by a similar work for 2D proximity selection \cite{guillon15}. The second evaluation was about the transfer function for the movement of the cursor, and the last one was about the benefits of filtering the inputs that control the ray with a 1\euro~filter~\cite{casiez12}.
After this, we designed a semi-automatic RayCursor that combines RayCursor and raycasting (\reffig{fig:raycursor}).
Finally, we compared the performance of the two versions of RayCursor, raycasting, and another technique of the literature \cite{ro17}.
%f)\hspace{-4mm}
\includegraphics[height=\fh]{raycursor_c}
\label{fig:raycursor}
- \caption[Illustration of Raycursor]{Two versions of Raycursor. The manual RayCursor (left) selects the nearest target from the cursor. The semi-automatic RayCursor acts like raycasting, a black cursor is position on the first intersected target. When the ray moves out of a target, it remains selected while it is the nearest target. The users can move the cursor manually with the touchpad, which turns red, to select another target. If the users lift their finger for more than 1s, the cursor switches back to its initial behavior.}
+ \caption[Illustration of Raycursor]{Two versions of Raycursor. The manual RayCursor (left) selects the nearest target from the cursor. The semi-automatic RayCursor acts like raycasting, a black cursor is positioned on the first intersected target. When the ray moves out of a target, it remains selected while it is the nearest target. The users can move the cursor manually with the touchpad, which turns red, to select another target. If the users lift their finger for more than 1s, the cursor switches back to its initial behavior.}
%Illustration of manual Raycursor:
%a)~the user controls a cursor along the ray using relative displacements of their thumb on the controller’s touchpad;
%b)~the target closest to the cursor is highlighted.
\subsubsection{Facial expression selection}
-Research on virtual reality and the possibility of creating immersive virtual environment inspired many science-fiction authors.
+Research on virtual reality and the possibility of creating immersive virtual environments inspired many science-fiction authors.
%Twenty years after Sutherland's work \cite{sutherland65}, the Neuromancer describes the \defword{cyberspace} as an alternate reality out of the physical world, created with machines \cite{gibson84}.
Decades after Sutherland's work \cite{sutherland65}, novels like Neuromancer \cite{gibson84}, Snow Crash \cite{stephenson92}, and more recently Ready Player One \cite{cline11} describe immersive virtual worlds as alternate realities in which people can socialize, play, or even work.
-The new brand of Facebook, Meta, is a reference to Snow Crash's \defword{metaverse} and shows the new focus of the company on immersive social network, with Spaces then Horizon\footnote{\href{https://web.archive.org/web/20191005002238/https://www.facebook.com/spaces}{https://www.facebook.com/spaces}, \href{https://www.oculus.com/facebookhorizon/}{https://www.oculus.com/facebookhorizon/}}.
-Similarly to other immersive social network like Mozilla Hubs\footnote{\href{https://hubs.mozilla.com/}{https://hubs.mozilla.com/}}, VRChat\footnote{\href{https://hello.vrchat.com/}{https://hello.vrchat.com/}}, and RecRoom\footnote{\href{https://recroom.com/}{https://recroom.com/}}, the objective is to enable people to get together in a virtual environment and interact with each other as if they were in the same room.
+The new brand of Facebook, Meta, is a reference to Snow Crash's \defword{metaverse} and shows the new focus of the company on an immersive social network, with Spaces then Horizon\footnote{\href{https://web.archive.org/web/20191005002238/https://www.facebook.com/spaces}{https://www.facebook.com/spaces}, \href{https://www.oculus.com/facebookhorizon/}{https://www.oculus.com/facebookhorizon/}}.
+Similarly to other immersive social networks like Mozilla Hubs\footnote{\href{https://hubs.mozilla.com/}{https://hubs.mozilla.com/}}, VRChat\footnote{\href{https://hello.vrchat.com/}{https://hello.vrchat.com/}}, and RecRoom\footnote{\href{https://recroom.com/}{https://recroom.com/}}, the objective is to enable people to get together in a virtual environment and interact with each other as if they were in the same room.
Such immersive virtual environments require easy and usable ways to perform atomic actions such as the study presented in the previous section.
%selecting and manipulating objects, or navigating the environment.
But above all, communication is certainly the most important aspect of social networks in general.
Text entry remains a difficult and tedious task in immersive virtual environments, and the current best solution is to simply use voice.
-However, non-verbal communication such as face expressions is also an essential aspect of communication, whether for speech in the physical world or by writing \cite{carter13}.
+However, non-verbal communication such as facial expressions is also an essential aspect of communication, whether for speech in the physical world or by writing \cite{carter13}.
%In this work, we focused on a particular type of non-verbal communication: facial expressions.
-One way to enable users to control the face expression of their avatar is to detect their own face expression, we call this isomorphic control.
+One way to enable users to control the facial expression of their avatar is to detect their own facial expression, we call this isomorphic control.
Vision-based techniques use either external depth cameras\cite{weise11,lugrin16} or cameras embedded in a VR headset \cite{li15,suzuki16}.
-However, with such techniques face expressions are limited to expressions users are able to perform.
+However, with such techniques facial expressions are limited to expressions users are able to perform.
Moreover, users cannot give their avatar a different expression than their own.
-Therefore we investigated the non-isomorphic control of face expressions, with interaction techniques \cite{baloup21}.
+Therefore we investigated the non-isomorphic control of facial expressions, with interaction techniques \cite{baloup21}.
-The fine control of face expressions requires many degrees of freedom.
+The fine control of facial expressions requires many degrees of freedom.
For example, the FACS standard defines 24 Action Units \cite{ekman78}, and the MPEG-4 proposes 68 Facial Animation Parameters\cite{pandzic03}.
-Therefore we propose to reduce the number of degrees of freedom by decomposing the selection of a face expression into several sub-tasks, similarly to Bowman's decomposition of 3D interaction tasks \cite{bowman04}.
-The sub-tasks are: selecting a face expression, its intensity, duration, and ending.
-The selection consists in choosing a face expression among a list of pre-defined expressions.
+Therefore we propose to reduce the number of degrees of freedom by decomposing the selection of a facial expression into several sub-tasks, similarly to Bowman's decomposition of 3D interaction tasks \cite{bowman04}.
+The sub-tasks are: selecting a facial expression, its intensity, duration, and ending.
+The selection consists in choosing a facial expression among a list of pre-defined expressions.
Each pre-defined expression is a configuration of FACS action units values.
-The \reffig{fig:faceexpressions} shows four of the face expression selection techniques we designed, the fifth one uses voice commands.
+The \reffig{fig:faceexpressions} shows four of the facial expression selection techniques we designed, the fifth one uses voice commands.
These techniques are essentially item selection techniques, similar to what we would use for command selection.
-Similarly to command selection, some of the face expressions share properties, which we leveraged to structure the layout of items in some of the techniques.
+Similar to command selection, some of the facial expressions share properties, which we leveraged to structure the layout of items in some of the techniques.
%Many of them represent emotions, therefore we leverage this information for the design of the selection techniques.
\begin{figure}[htb]
%d)
\includegraphics[height=\fh]{emoraye_rmw}
\label{fig:faceexpressions}
- \caption[Avatar face expression selection techniques in VR.]{Four of the face expression selection techniques: 2D circular menu arranged by emotion, raycasting 2D grid menu, touchpad gestures, raycasting on the 2D circular menu arranged by emotion.
+ \caption[Avatar facial expression selection techniques in VR.]{Four of the facial expression selection techniques: 2D circular menu arranged by emotion, raycasting 2D grid menu, touchpad gestures, raycasting on the 2D circular menu arranged by emotion.
%designed:
%a) menu presents a grid menu in front of the user with raycasting used to select an expression,
%b) touchpad presents a circular menu above the controller and selection is made using the controller's touchpad,
}
\end{figure}
-The visual representation of face expressions in our techniques use emojis.
+The visual representation of facial expressions in our techniques uses emojis.
We made this choice because they are frequently used in messaging and social networks.
-Also, people can identify them even small versions of them.
+Also, people can even identify small versions of them.
%Some of them represent emotions.
-If we restrict face expressions corresponding to emotions, we can leverage models of emotions such as PAD \cite{mehrabian96} or Plutchik's wheel of emotions \cite{plutchik01}.
+If we restrict facial expressions corresponding to emotions, we can leverage models of emotions such as PAD \cite{mehrabian96} or Plutchik's wheel of emotions \cite{plutchik01}.
The layout of our pie menu selection technique is adapted from Plutchik's wheel, which organizes emotions along 8 axes representing 8 base emotions.
The difference between Plutchik's wheel and our menu is that we mapped the maximum intensity to the edge of the circle rather than to the middle so that the center represents the neutral face.
-The grid menu allows the selection of any face expression, regardless if they represent an emotion or not.
-In particular users can select emojis with decorations such as hearts, tears, or glasses for example.
-In this case we render these decorations on top of the avatar's face.
-Transitions between two face expressions are made by interpolating the values of each FACS action unit.
-Our evaluations show that participants made more errors with the gesture and voice techniques, and they prefered the other techniques.
+The grid menu allows the selection of any facial expression, regardless of whether they represent an emotion or not.
+In particular, users can select emojis with decorations such as hearts, tears, or glasses for example.
+In this case, we render these decorations on top of the avatar's face.
+Transitions between two facial expressions are made by interpolating the values of each FACS action unit.
+Our evaluations show that participants made more errors with the gesture and voice techniques, and they preferred the other techniques.
-%The the layout of the circular menu is based on Plutchik's wheel of emotions \cite{plutchik01}.
+%The layout of the circular menu is based on Plutchik's wheel of emotions \cite{plutchik01}.
-We designed several techniques for controlling the intensity, duration, and ending of face expressions all at once.
-Users define the duration indirectly when they end the face expression.
+We designed several techniques for controlling the intensity, duration, and ending of facial expressions all at once.
+Users define the duration indirectly when they end the facial expression.
The first technique maps the intensity to the controller trigger.
-The face expression ends $1s$ after the user released the trigger to avoid incidental endings.
+The facial expression ends $1s$ after the user released the trigger to avoid incidental endings.
The other techniques below end when the user releases the selection button.
%This method is not convenient with the gesture and voice selection techniques because it adds another step, contrary to the other techniques that already use a validation action.
With the second technique, users have to shake the controller and the intensity is mapped to the shaking speed.
%Participants of our experiments had poor performance with these two methods.
The fourth technique creates a visible elastic band between the selected expression and the selection ray.
The intensity is mapped to the length of the elastic band.
-This technique requires a ray to which the elastic band is attached, therefore it cannot be used with a subset ot the face expression selection techniques.
-The last technique only works with pie menu techniques, and maps the intensity to the distance to the center of the menu.
-Our evaluations show a subjective preference of the elastic band, trigger, and orientation techniques over the shake technique.
+This technique requires a ray to which the elastic band is attached, therefore it cannot be used with a subset of the facial expression selection techniques.
+The last technique only works with pie menu techniques and maps the intensity to the distance to the center of the menu.
+Our evaluations show a subjective preference of the elastic band, trigger, and orientation techniques over the shaking technique.
-We evaluated the usability of the pie menu with a raycasting selection and an intensity control with the elastic band.
-The results show that controlling their avatar's face expression can disturb users while they are talking or listening to somebody else.
-However, the perceived interruptions were reasonnable.
+We evaluated the usability of the pie menu with a raycasting selection and intensity control with the elastic band.
+The results show that controlling their avatar's facial expression can disturb users while they are talking or listening to somebody else.
+However, the perceived interruptions were reasonable.
\subsubsection{Discussion}
More than fifty years after the first virtual reality headsets, the use of immersive virtual reality is still in its early stages.
-Social media companies started investing massively on this topic in the last decade with the idea to implement alternative realities that break the barriers of the physical world.
-One of the obvious barriers to break is distance, by allowing people to be in the same room even if they are physically far from each other.
-But when people evolve in such virtual environments, they are free of the constraints of the physical world.
+Social media companies recently started investing massively on this topic.
+%, with the idea to implement alternative realities that break the barriers of the physical world.
+%One of the obvious barriers to break is distance, by allowing people to be in the same room even if they are physically far from each other.
+%But when people evolve in such virtual environments, they are free of the constraints of the physical world.
+When people evolve in such virtual environments, they are free of the constraints of the physical world.
Therefore when we design interaction techniques to enable people to perform actions in such virtual environments, we also think about ways to perform actions they could not in the physical world.
-The two techniques we discussed in this section allow users to perform two distinct actions: select objects and show a face expressions.
-However they both have in common that they enable users to perform these actions in ways they could not in the physical world.
-In the physical world it is impossible to select an object remotely and show a face expression different from their actual face expression.
+The two techniques we discussed in this section allow users to perform two distinct actions: select objects and show facial expressions.
+However, they both have in common that they enable users to perform these actions in ways they could not in the physical world.
+In the physical world, it is impossible to select an object remotely and show a facial expression different from their actual facial expression.
The choice of using these techniques in virtual reality systems is indeed a matter of trade-offs.
Most of the benefits of Raycursor rely on proximity selection.
We identified issues with convex and oblong objects, as well as dense clusters of small objects surrounded by large objects.
Solutions remain to be investigated for such cases.
-The main argument for using a non-isomorphic technique for selecting a face expression is to enable users to show a face expression different to their actual face expression.
+The main argument for using a non-isomorphic technique for selecting a facial expression is to enable users to show a facial expression different from their actual facial expression.
This is typically what we do when we use emojis in messaging or social media apps.
-In particular it allows people to show a specific face expression even if they are not able to perform it at will.
+In particular, it allows people to show a specific facial expression even if they are not able to perform it at will.
Not everybody is an actor.
%What we wanted to evaluate in this work is the ability of people to select face expressions during a conversation without
Now, isomorphic techniques also have benefits.
-They do not require a conscious control and they do not disrupt or interrupt a primary task, like talking.
+They do not require conscious control and they do not disrupt or interrupt a primary task, like talking.
Moreover, new VR headsets include cameras for face tracking\footnote{\href{https://www.vive.com/eu/accessory/facial-tracker/}{https://www.vive.com/eu/accessory/facial-tracker/}}, which will make it easier to implement such interactions techniques.
-Therefore it would be interesting to investigare the combination of isomorphic and non-isometric techniques to get the best of both, and study the control/automation trade-off in the context of face expression selection.
+Therefore it would be interesting to investigate the combination of isomorphic and non-isometric techniques to get the best of both and study the control/automation trade-off in the context of facial expression selection.
\section{Conclusion}
The previous chapter was about the sense of touch, one of the five senses.
% humans use to get information from their environment.
I was however surprised by the difficulty for me to find an equivalent of the word “senses”, as the human input systems, for the human output systems.
-I decided to use the word \defword{ability}, that I refer to ways humans have to act on their environment, similarly to the way \defword{sense} enable them to get information from their environment.
+%I decided to use the word \defword{ability}, which I refer to the ways humans have to act on their environment, like the way \defword{sense} enables them to get information from their environment.
+I decided to use the word \defword{ability}, which refers to the ways humans have to act on their environment, the same way their \defwords{abilities}{ability} enable them to get information from their environment.
In fact, humans do not have many kinds of ways to act on their environment.
This is maybe the reason why I could not find the word I was searching for.
To the best of my knowledge, humans can do so with movements, voice, and fluid secretions.
This chapter focused on \emph{motor abilities}, which are abilities that leverage the human motor system.
The motor system enables people to touch and manipulate their environment, therefore it is the output part of the human haptic system.
I described on \reffig{fig:motorpath} the process between the moment a user plans a manipulation action and the moment the system produced information based on the effects of this manipulation it sensed.
-Similarly to the analogous figure in the previous chapter (\reffig{fig:hapticpath}), this description clarifies the software and hardware parts on both the human and system side, as well as the connection between humans and systems.
-This processed revealed general research questions that I addressed in some of my research projects.
+Similar to the analogous figure in the previous chapter (\reffig{fig:hapticpath}), this description clarifies the software and hardware parts on both the human and system side, as well as the connection between humans and systems.
+This process revealed general research questions that I addressed in some of my research projects.
-The first research questions was about the way we design and evaluate the sensing and interpretation of physical effects resulting from a human manipulation with an interactive system.
+The first research question was about the way we design and evaluate the sensing and interpretation of physical effects resulting from human manipulation with an interactive system.
This question is essentially related to the system.
-In particular I discussed a methodology and tool we designed and implemented to measure and slice the latency of interactive systems.
+In particular, I discussed a methodology and tool we designed and implemented to measure and slice the latency of interactive systems.
The second research question was about the interactive input vocabulary.
This is a whole research domain in itself.
I discussed first the design of alternative input methods with flexible pens.
Then I discussed the mapping and reduction of degrees of freedom with finger identification for multi-touch interaction.
-The third research question was about unnatural input, or ways to leverage the properties of virtual environments to perform actions that are difficult or impossible in the physical world.
+The third research question was about unnatural input, which are ways to leverage the properties of virtual environments to perform actions that are difficult or impossible in the physical world.
This research is essentially about interaction techniques, therefore it is both about the users and the system.
The contributions I describe are interaction techniques for immersive virtual reality.
The first one is a distant 3D pointing technique with proximity selection
%unnatural input
% vr
-Indeed, all the studies we discussed in this chapter focus on the hands dexterity and our capacity to touch and manipulate.
+Indeed, all the studies we discussed in this chapter focus on hands dexterity and our capacity to touch and manipulate.
However, the haptics as the sense of touch was barely used.
The latency measurement study assumed the output was visual.
-We performed measurements on Linux, MacOS and Windows with several graphics library, that send data to a graphic card connected to a monitor.
+We performed measurements on Linux, MacOS, and Windows with several graphics library, that send data to a graphic card connected to a monitor.
We showed that this part of the process was the major source of latency.
Haptic systems are quite different, and there is no clear standard.
There is typically a haptic look running around 1kHz that computes forces or vibrations faster than the human haptic sensitivity~\cite{salisbury04}.
Audio-haptic systems with physical simulations use loops up to $10kHz$ \cite{leonard15}.
-In the future, I would like to study the effect of the haptic loop on the perception of different kind of force models.
+In the future, I would like to study the effect of the haptic loop on the perception of different kinds of force models.
I will also measure the latency of several types of haptic systems, to compare them to visual systems.
-The flexible pens we designed provide passive force feedback with their flexural stiffness.
-We build several prototypes of different flexural stiffnesses and studied its effect on both objective and subjective measures.
-However, we could also provide active vibrotatile feedback with an actuator.
+The flexible pens we described in this chapter provide passive force feedback with their flexural stiffness.
+We build several prototypes of different flexural stiffnesses and studied their effect on both objective and subjective measures.
+However, we could also provide active vibrotactile feedback with an actuator.
With controllable haptic feedback, we could give users a haptic immediate response when the users are bending the stylus.
-My hypothesis is that such feedback, like haptic detents, would help users controlling the bend of the device with continuous gestures.
+I hypothesize that such feedback, like haptic detents, would help users control the bend of the device with continuous gestures.
We could also create discrete input with click sensations for activation gestures.
The overall idea would be to use haptic feedback to support direct manipulation.
We will discuss this concept in a different context with a haptic wristband in \refsec{sec:hapticdm} of \refchap{chap:loop}.
-Our multi-touch interaction paradigm with finger identification heavily rely on visual cues.
+Our multi-touch interaction paradigm with finger identification heavily relies on visual cues.
It uses visual feedforward, feedback, and uses additional visuals for promoting learnability and discoverability.
Tactile feedback on touch surfaces is usually poor.
-At best smartphones have a low quality tactile actuator.
+At best smartphones have a low-quality tactile actuator.
We started this project with tabletops, which are difficult to actuate.
%But above all, there is currently no convenient way to provide
We discussed the design and evaluation of several potential technologies for this in \refsec{sec:stimtac} and \refsec{sec:printgets}.
-It would be interesting to see how such haptic feedback can reduce visual clutter of our multi-touch interaction paradigm with finger identification.
+It would be interesting to see how such haptic feedback can reduce the visual clutter of our multi-touch interaction paradigm with finger identification.
Finally, we discussed the way that immersive virtual environments enable users to perform actions that are difficult or impossible to perform in the physical world.
-In particular, we discussed two contributions, the first one was a distant pointing technique with proximity selection, and the second one a facial expression selection technique.
+In particular, we discussed two contributions, the first one was a distant pointing technique with proximity selection and the second one was a facial expression selection technique.
These techniques use basic haptic feedback, with simple clicks upon selection activation for example.
However, it does not permit users to feel objects they touch in the environment.
Haptic devices for immersive Virtual Reality is an active research topic \cite{bouzbib21}.
projects / hdr.git / commitdiff