In a multi-sensory environment, supported with embedded computer technology, the system can capture and interpret what the users are doing and assist or collaborate with the users in real-time. Such an environment should be aware of users intentions, tasks and feelings, and allow people to interact with the environment in a natural way: by moving, pointing and gesturing. In this paper we propose an architecture for such a smart environment consisting of three modules.