The physical data can be in the form of a report on the location and activity of the user, and the digital information can provide demographic comparisons between the user's profile and that of peers. Semantic web searches or explicit input by an expert can serve as alternative sources of data from the digital world.
The physical data acquired from the user has a quantitative nature and often comes in as samples obtained by a number of sensors, or in the case of using visual processing as a stream of events describing the user's location, pose, gesture, activity, or interactions. The expert information or demographic data often have a qualitative nature and provide a generic description of a behavioral condition or average values for certain behavioral parameters. This topic explores developing techniques for employing such generic descriptions to produce decision boundaries that can be applied to the real-time quantitative data from a user's observations for taking action in response to the observed user events.
Personal devices such as smartphones and laptop computers can serve as the interface to both collect the user's state (e.g. location) and interests (e.g. recent searches related to the application), and to deliver the digital content to the user. By being connected to the internet cloud, these devices can also interface with a server which creates and maintains the user's profile and provides recommendations based on an expert's rules or via demographic evaluation.
Example applications of a personal recommendation system for offering services at home include automation (user comfort and energy efficiency), social media (interactions and experience sharing with online peers), abnormal event detection (elderly care), and well-being (exercise, mobility, food habits). In the work environment, examples include ergonomic recommendations (posture, taking breaks), social interaction promotion (social events), and personal feedback to presenters and speakers. Examples in elderly care facilities include collecting statistics for caregivers and recommending healthy practices and social interactions to the inhabitants.
In an observation system based on visual input, processing functions transform the incoming streams of video to meta data which contain the events and attributes of the user's activities. Contextual information such as time of day and environmental factors can also be incorporated into the data that is delivered to the behavior modeling layer. Another set of input to the behavior modeling function is the information provided by an expert (which can be a human specialist or a digital agent that collects relevant data from the web). The expert information can define the set of events of interest to be recorded in the behavior model of the user, and it also provides guidelines for the range of interest in specific parameters related to the behavior. These values can be employed by the service layer of an application to trigger an action (e.g. intervene to remind the user of a task, alert a caregiver of the current condition), or to adjust the settings of the service parameters to better match the user's behavior or interests.
The occupancy map of a building can consist of reports of presence of people in different locations which are semantically defined on the building's floor schematic, duration of stay at a location, flow of people passing through corridors and gates, and occupancy pattern of offices, conference rooms and open areas. With contextual data like calendar schedules for conference rooms or event areas, the occupancy map can also contain information about the distribution of occupants during specific events such as meetings, seminars, social gatherings, poster sessions, and exhibitions.
A network of cameras can provide the most informative detail about the occupancy (number and distribution of people) as well as the dynamics of the occupancy behavior in different contexts. The occupancy map of the building can also serve other important applications: resource and space planning, service automation, and emergency management. The building's usage schematics and space partitioning can be planned based on the occupancy model, enabling factors such as better distribution of personnel across space, or reduction of noise and interruptions in work areas based on actual usage patterns. A long-term behavior model for the occupancy of a building, when correlated with contextual data such as calendar information and seasonal factors, can provide a model for energy expenditure planning and automation and enables the adaptation of automatic services to the occupancy behavior. It can also allow for measuring the effectiveness of any adopted energy saving functions against actual occupancy of the building. An extended set of applications such as planning of space usage, effective partitioning, promoting and measuring social interaction between occupants, and assisting with emergency evacuation operations can also be enabled though observing and modeling the occupancy behavior.
Smart cameras have been proposed and evaluated in a number of settings. They operate based on the notion of processing the acquired video frames in the camera unit and subsequently deleting the raw video while only preserving and reporting meta data which results from the processing operation. However, since a track record of successful applications based on these devices does not exist in the market, nor a trust metric is in general established to demonstrate the security of the acquired visual data, the progress in adopting smart cameras has been slow.
The information that can be obtained via the use of one or multiple cameras in an environment is very rich relative to that of any other types of sensors. This richness can obviously be employed to serve the user in a context-aware fashion. Inferences of the user's presence, location, track, pose, posture, gesture, action, behavior, and interactions with others or with objects can all be in principle measured by a camera. This richness however is accompanied with a need to address the user's concerns of privacy. To propose a camera-based service, a clear value proposition based on solving an existing need of the user which cannot be addressed with other means should hence be defined.
In the home environment, applications in elderly care such as fall detection and abnormal event detection have been discussed in the past. Besides the need to address the privacy issues these applications face the challenge of having to offer a reliable and persistent accuracy. While an application like visual communication between the elderly and others through the use of a webcam and TV may be considered as an ordinary and acceptable camera-based application by the user, developing a service based on analyzing the content of the acquired video has to consider a number of demanding requirements such as those listed above as well as offering the user full control of the application's function.
Many personal computers are equipped with cameras. It is conceivable that applications running on a user's personal computer can use the data from the camera to provide personalized services to the user. A simple application which adjusts the screen's brightness based on the sensed light intensity in the camera view has existed for some time. More complex applications based on the presence of the user, the posture, duration of working on the computer, facial expressions and emotions can be developed to offer the user recommendations on better ergonomics or taking breaks. A helpful application could be to analyze the performance of the user during a seminar and offering recommendations on better body language and use of gestural habits in future presentations.
User acceptance issues need to be considered as an integral part of the design effort when developing applications based on sensors and cameras. Such consideration can offer an insight into the practical aspects of the design such as methods of privacy and data security, user control, and the intelligibility of the system's operation for the user, which altogether determine the chances that the application be adopted by the intended target users.
Earlier Topics: Research concepts developed as physical embodiments at A I R Lab include: