% Physical properties of objects also give haptic feedback\\
-%Shape changing~\cite{michelitsch04} Knobslider~\cite{kim16} Morphees~\cite{roudaut13a}
+%Shape-changing~\cite{michelitsch04} Knobslider~\cite{kim16} Morphees~\cite{roudaut13a}
%\subsection{Towards a haptic semiology}{}
%Lack of Bertin-like haptic semiology \cite{bertin83} \texttt{=>} diversity of sensations, stimulation mechanisms, haptic variables
In this work we focused on desktop interaction with the augmentation of desktop peripherals.
% peripherals: keyboards, mice and screens.
-Past research extended the capacities of computer peripherals, in particular keyboards.
+There are many examples of extensions of desktop peripherals in past research, in particular keyboards.
For example additional sensors enable contact sensing on the keys of a keyboard~\cite{rekimoto03}, gesture on the whole keyboard surface~\cite{block10,kato10,taylor14,zhang14}, or force sensing on keys~\cite{dietz09}.
-It is also possible to embed actuators in each key to make them harder to press~\cite{hoffmann09,savioz11}.
-The work we have done with Métamorphe explores the augmentation of keyboards further~\cite{bailly13}.
+In other works, actuators are embedded in each key to make them harder to press~\cite{hoffmann09,savioz11}.
+Mice werre also augmented, with shape-changing features to enable notifications or provide an elastic input device for continuous rate control~\cite{kim08}.
+More generally, the idea of changing the shape of a physical controler is to provide different affordances~\cite{michelitsch04,kim16}.
+The work we have done with Métamorphe explores the augmentation of keyboards for increasing its input vocabulary~\cite{bailly13}.
We embedded solenoids in keys such that they can either be raised or lowered.
In both positions they could be pressed though.
-Not only it changes the geometry of the keyboard, but it also gives users addess to the sides of the keys.
-We will discuss this in the next section, about the augmentation of desktop interaction at the device level.
+Not only it changes the geometry of the keyboard, but it also gives users access to the sides of the keys.
+We will discuss the augmentation of desktop interaction at the device level in the next section.
Beyond the augmentation of devices, the desktop itself can also be augmented.
-For example projection around the computer can give access to additional information and offer more interactive surface~\cite{bi11}.
-Every object in the environment can actually become a screen and enrich our interaction with interactive systems~\cite{gervais16}.
-
-%%%
-\fixme{
-Interesection of Tangible interaction and shape changing.
-
-The primary rationale of augmenting the desktop worksta- tion is to better blend it in office activities.
-Several works aimed at enabling simultaneous work with physical informa- tion such as paper documents and digital information \cite{arai95,wellner93}. The idea is to enhance paper documents that are cheap, famil- iar, easy to read, etc. with computation rather than replacing them with computers. For instance, the DigitalDesk \cite{wellner93} and InteractiveDesk \cite{arai95} recognize and scan paper documents with a camera and augment them with videoprojection.
-
-Other approaches make good use of everyday human skills to augment desktop workstation, typically designers who use
-Wacom tablets for drawing-like activities. In that respect, Hinckley et al. considers the ability of humans to use both hands in the real world and present a set of two-handed inter- action techniques for desktop \cite{hinckley98}.
-A final approach considers the furnitures surrounding the desktop workstation to extent peripheric interaction. For in- stance, office workers can use the desk as an input (touch or pen input for shortcuts) or output surface to extend the mon- itor display area with video-projection \cite{bi11,steimle11}[17]. The chair can also be used as an input device to capture emotional data, gesture input and support office activities (e.g. [35, 36]).
-Our approach also considers the desktop workstation as a whole that should be integrated in its environment, but our primary focus is on augmenting devices and their interaction.
-}
-
-\newpage
-
-In the early days of tangible interaction, Ullmer and Ishii described Tangible interaction this way: “TUIs will augment the real physical world by coupling digital information to everyday physical objects and environments.”\cite{ishii97}.
-The idea is to break the barrier between the physical and the digital world.
-With this paradigm, any object can either represent digital information or be a proxy for manipulating digital information.
-Similar to input and output devices, these objects are instrumented with sensors or actuators, to create links with the digital world.
-
-
-The question whether computer peripherals such as a mouse can be used as a TUI is subject to debate.
-On one hand it complies with Ullmer and Ishii's definition we provided at the beginning of this chapter.
-Since the introduction of this definition, computer peripherals actually became everyday objects.
-Augmenting them to couple them with digital information would make them TUIs.
-On the other hand, comxputer peripherals were specifically designed for interaction with digital information, and would not exist otherwise.
-Hence, considering them as TUIs would neglect the very concept of TUI.
-
-Now, consider the situation of users giving a talk with a slideshow.
-They holds the mouse in the hand and just use the buttons to move to the next slide, the same way they would do with a remote control.
-In this case they are not using the mouse as it was designed for.
-Is it sufficient to consider the computer mouse as a TUI in this specific scenario?
-
-Further, now imagine an actuated computer mouse so that it can move around on the table.
-This mouse moves around on the desk to give users notifications when they are not watching the screen.
-In this situation, the mouse is clearly not used as the mouse was designed to be used.
-In this work we explore actuation and motion as a way of interacting with computer peripherals in the way they were not designed for.
-%We discuss scenarios in which computer peripherals are tangible objects for interaction with digital information.
-
-\subsubsection{Related work}
-
-We describe below evolutions of the desktop interface, the use of motion as an output modality, and shape changing interfaces.
-
-\paragraph{Rethinking desktop interaction}
-
-The way we interact with computer peripherals has not changed much since their invention.
-Mice, keyboards and screens have not changed much on an interaction point of view.
-%The mouse wheel is an exception to this observation.
-%The touchpad replaced the mouse in many situations, but mostly for laptop interaction.
-%Besides, the touchpad is not an evolution of the mouse, but rather a new device itself.
-%Touchscreens are as evolutions of computer screens.
-%However touchscreens on a desktop setup hardly compete with indirect pointing devices such as a mouse or a touchpad.
-
-%We see two reasons why the devices hardly evolve.
-%The first one is that these devices became mainstream because they are well designed.
-%The second one is that users have years of experience with these devices.
-Studies showed that some design choices are questionable.
-For example Pietrzak \etal studied the impact of the mode delimiters for keyboard shortcuts by replicating the \textsc{Ctrl} and \textsc{Shift} on the thumb buttons of the mouse~\cite{pietrzak14}.
-They observed similar performance for keyboard shortcut entry than with the keyboard.
-This means it makes sense to revisit design choices made decades ago.
-
-%Research explored additional dimensions to extend the capacities of computer peripherals.
-%Rekimoto et al. added capacitive sensing to the keys of a keyboard~\cite{rekimoto03}.
-%It enables sensing whether the user touches a key or not.
-%They propose scenarios in which they use this information to display feedforward, and other scenarios which take advantage of this extended vocabulary to enhance interaction.
-
-%Beyond rethinking desktop devices, Bi et al. used the desk itself for interaction~\cite{bi11}.
-%They extend the peripherals capabilities with interaction with the desk, both for multi-touch input and a projected display.
-%At the opposite, Gervais et al. use everyday objects as viewports, which share or extend computer screens real estate~\cite{gervais16}.
-%These systems explore tangible properties of the desktop environment to extend interaction.
-
-\paragraph{Motion output}
-
-Motion is a property of the interaction with an object.
-It is commonly used as input values, but we are interested in motion of a physical object as an output modality.
-Motion as output produces visual and haptic feedback.
-
-Löffler et al. designed insect-like desk companions~\cite{loffler17}.
-These companions can move around on the desk to give the user notifications through the visual channel.
-Authors focused on their affective effect on the user.
-Interestingly, motion based interfaces can take advantage of both the visual and haptic aspects of movements.
-Zooids are small robots which cooperate to achieve a particular task~\cite{legoc16}.
-In some situations they represents points on a graph.
-In other situations they move an object on a table.
-%\thomas{trouver ref sur l'audio ?}
-
-Actuating an object enables dynamic force feedback when the user touches it.
-For example Roudaut et al. explored the idea of actuating a layer over a touchscreen to guide the finger touching the device~\cite{roudaut13}.
-It makes it possible to teach users gestures, such as gesture keyboard symbols.
-Other studies use motion to encode information.
-Either the system controls the movement~\cite{enriquez03,pietrzak05}, or only constrains the movements of the user~\cite{pietrzak05a}.
-Similarly to Tactons~\cite{brewster04-2}, information is coded by mapping pieces of information to signal parameters such as amplitude, size or shape.
-Below, we explain how we use motion to extend interaction with computer peripherals.
-
-\paragraph{Shape changing interfaces}
-
-Actuating objects also make it possible to change their shape, and therefore their affordances.
-Knobslider is an example of interactive object which is either a button or a slider, depending on its shape~\cite{kim16}.
-This object was specifically designed to behave this way.
-At the opposite, Kim et al. designed an inflatable mouse~\cite{kim08} which can either give notifications, or be used as an elastic input device for continuous rate control.
-
-\subsubsection{Actuated peripherals}
-
-In this project we use both the concepts of motion as output, and shape changing interfaces to redesign computer peripherals.
-We discuss design rationales on the device level, desktop level, and envision extending the concept to en entire room or a house.
-
-\paragraph{Device level}
-***
+The earlier works on the extension of the desktop combined the best of paper and computing~\cite{arai95,wellner93}.
+Later, other systems combined projection and touch interaction to give users access to additional information and offer them a larger interactive surface~\cite{bi11,steimle11}.
+%Further, Gervais \etal turned every object in the environment into a screen, to enrich our interaction with interactive systems~\cite{gervais16}.
+Actuation is another modality that augments desktop interaction.
+For example, a fleet of small robots can represent data dynamically~\cite{legoc16}.
+However the combination of ubiquitous displays with actuation brings another dimention that increases the interaction vocabulary~\cite{roudaut13a}
+Our approach with Living Desktop also considers the desktop workstation as a whole that should be integrated in its environment~\cite{bailly16}.
+However, our primary focus is on augmenting devices and their interaction by leveraging their physical properties.
+
+These two research projects are complementary, and they focus on different levels.
+Métamorphe focuses on the device level while Living Desktop focuses on the desktop level.
+In both cases we use actuation as a mechanish to provide new features, with two paradigms in mind.
+The first one is shape-changing interaction: the shape of an object is a signifier of its affordances.
+Hence changing the shape of a device is an interesting way of providing and advertising an extended interactive vocabulary.
+The second paradigm is tangible interaction, which “augments the real physical world by coupling digital information to everyday physical objects and environments.”\cite{ishii97}.
+Here we see desktop peripherals as everyday objects that we manipulate as such.
+
+\subsubsection{Device level}
Motion is an essential aspect of interaction with peripherals.
Pointing devices rely on movement measurements.
Keyboards use binary key positions as input data.
In the Métamorphe project~\cite{bailly13}, we actuated the keys so that they can either be up or down (\reffig{metamorphe}, left).
-Keys can still be pressed, whether the key is up or down.
+However, contrary to other augmented keyboards~\cite{hoffmann09,savioz11}, userrs can prress its keys, whether the key is up or down.
\begin{figure}[!htb]
\centering
\includegraphics[height=4.4cm]{metamorphe_raised}\hfill
\includegraphics[height=4.4cm]{metamorphe_pinch}
% \vspace*{-7mm}
- \caption{Métamorphe is a keyboard with actuated keys, which can either be up or down. Left: view of the keyboard with two keys up. Right: raised keys have new affordances. They can be pushed or pinched.}
+ \caption[Métamorphe keyboard]{Métamorphe is a keyboard with actuated keys, which can either be up or down. Left: view of the keyboard with two keys up. Right: raised keys have new affordances. They can be pushed or pinched.}
\label{metamorphe}
% \vspace*{-3mm}
\end{figure}
-This shape changing keyboard has new properties compared to regular keyboards.
-When a key is up, the use can push it in four directions, or even pinch it (\reffig{metamorphe}, right).
-With a touch sensor all around it, the key could be used as an isometric pointing device such as a trackpoint.
-
-Our previous studies showed that raising keys eases eyes-free interaction with the keyboard.
-Specifically we observed that users can easier locate raised keys and surrounding ones.
+This shape-changing keyboard provides new interactive properties compared to regular keyboards.
+First of all, it changes the haptic properties of the keyboard.
+When the users' hand scan the keyboard, they can distinguish raised keys.
+Not only it helps users to locate raised keys, but also the surrounding keys because the raised keys provide a reference point.
+This is the same phenomenon than the little notches on the \keys{F} and \keys{J} keys that help touch typists placing their hand on the keyboard, at a differrent scale and with more flexibility.
+We experimented this phenomenon with a user study~\cite{bailly13}.
+It is indeed an interesting property for eyes-free interaction.
+We can imagine for example that keys corresponding to a keyboard shortcut are raised when the \ctrl key is pressed.
+This was the original idea behind the CtrlMouse project~\cite{pietrzak14} that I will not discuss in this document.
+
+Beyond haptic properties, this new mechanism provides other benefits.
+When a key is up, users can push it in four directions, pinch it (\reffig{metamorphe}, right), or even rotate it.
+With a force sensor all around it, we can turn the key into an isometric pointing device such as a trackpoint.
+Previous work showed examples of interactions we can perform with an array of actuated rods~\cite{iwata01,leithinger10,follmer13}.
+Our prototype only actuated eight keys for technical reasons, and we kept the layout of traditional keyboards because our intention was to augment traditional keyboards.
+However, if we put technical limitations aside, we can envision a combination of these two concepts: a shape changing surface, and a new input vocabulary brought by controlers that pop out of the surface.
+
+\subsubsection{Desktop level}
-% \begin{figure}[!htb]
-% \centering
-% % \vspace{-3mm}
-% \includegraphics[width=\columnwidth]{metamorphe_pinch}
-% % \vspace*{-7mm}
-% \caption{Raised keys have new affordances. They can be pushed or pinched.}
-% \label{pinch}
-% % \vspace*{-3mm}
-% \end{figure}
-
-The possibilities of such a keyboard go beyond text typing and keyboard shortcuts. Similarly to Relief~\cite{leithinger10}, it is a shape changing device which can be used to display information.
-
-\paragraph{Desktop level}
-***
%Several augmented devices exist in the literature.
%They are essentially isolated devices, not a set of coherent devices which share the same behavior.
-We observed people when they use a desktop computer, and identified situations in which they move their peripherals besides interaction with the computer.
+In this project, we observed people when they use a desktop computer.
+We identified situations in which they move the peripherals but not for interacting with the computer.
For example we observed people turning their screen to avoid sun reflexions.
-Other users turned their screen either to show visual content to somebody, or to show something in the room in a video conference with the camera affixed to the screen.
-It is also frequent to move the mouse and keyboard to make space on the desk for something else.
+Other users turned their screen either to show visual content to somebody.
+%, or to show something in the room in a video conference with the camera affixed to the screen.
+%It is also frequent to move the mouse and keyboard to make space on the desk for something else, or share them with other users to give them the control of the computer.
+It is also frequent to give other people the mouse or keyboard to give them the control over the computer.
-In the Living Desktop project\cite{bailly16}, we actuated a mouse, keyboard and screen (\reffig{livingdesktop}):
-\begin{itemize}
-\item The mouse can translate in the $x,y$ plane directions.
-\item The keyboard can rotate, and translate in the $x,y$ plane directions.
-\item The screen can rotate, and translate in the $x$ axis direction.
-\end{itemize}
+In the Living Desktop project\cite{bailly16}, we actuated a mouse, keyboard and screen (\reffig{livingdesktop}).
+The mouse and the keyboard can translate in the $x,y$ plane directions.
+The keyboard can also rotate.
+The screen can rotate, and translate in the $x$ axis direction.
\begin{figure}[!htb]
\centering
\includegraphics[height=6cm]{livingdesktop_concept}\hfill
\includegraphics[height=6cm]{livingdesktop_poc}
% \vspace*{-7mm}
- \caption{The Living Desktop is a concept in which desktop peripherals can move around on the desk.}
+ \caption[Living Desktop]{The Living Desktop is a concept in which desktop peripherals can move around on the desk.}
\label{livingdesktop}
% \vspace*{-3mm}
\end{figure}
The interesting question here is the degree of control the user has over his devices.
There is a continuum between full control and full automation, in which we identify some particular degrees:
-\begin{itemize}
- \item Telekinesis: the user moves the devices with distant controls.
- \item Tele-operation: the user suggests movements, the device decides to which degree it complies.
- \item Constraint: the user defines the constraints of the devices movements.
- \item Insurrection: the user has no influence on the device movements.
-\end{itemize}
-\fixme{work this list a little}
+\paragraph{Telekinesis}
+User Full control
+
+the user moves the devices with distant controls.
+
+Video Conference
+
+\paragraph{Teleoperation}
+User control, with system constraints
+the user suggests movements, the device decides to which degree it complies.
+
+\paragraph{Constraint}
+System control, with user constraints
+the user defines the constraints of the devices movements.
+
+\paragraph{Insurrection}
+System full control
+the user has no influence on the device movements.
+
+Ergonomic coach
+
+
+\newpage
+
+
We implemented a couple of scenarios, which illustrate the concept.
The user has no control over the devices in this situation.
-\paragraph{Going further}
-***
+\subsubsection{Discussion and conclusion}
+
Looking at the office environment, there are many other objects involved.
They can be actuated to provide other interactive scenarios.
Probst et al. presented a prototype of chair they use for input\cite{probst14}.
The obvious application is to maintain sunlight at a specific location in the house.
But there may be many interesting interactive scenarios to study with such a building.
-\subsubsection{Conclusion}
-
The early studies about TUIs used to consider everyday objects for interaction.
Nowadays, computer peripherals became everyday objects.
As such, they can also be considered as TUIs as long as they are not used as the device they are designed to be.
projects / hdr.git / commitdiff