A Framework for Dynamically Adaptable Augmented Reality Systems Thomas Reicher Preface The history of the DWARF project The starting point of developing an Augmented Reality system at the Technische Universit¨at M¨unchen was the course “Erweiterte Realit¨at: Bildbasierte Modellierung und Tragbare Computer”. It was part of the 1999 summer school organized by the Technische Universit¨at M¨unchen and the Friedrichs- Alexander Universit¨at Erlangen-N¨urnberg. Prof. Bernd Br¨ugge and Prof. Gudrun Klinker had the vision that software engineering and Augmented Reality would have a stimulating influence on each other. There, the idea was born to develop an own Augmented Reality system based on modern software engineering concepts. The project name was DWARF for Distributed Wearable Augmented Reality Framework. The DWARF project started in earnest in spring 2000 and resulted in the first DWARF application, DWARF Pathfinder. It was demonstrated successfully in fall 2000. After that, the DWARF framework became the basis for several other Augmented Reality student projects at the Technische Universit¨at. But now DWARF has become a complex framework. It is the result of applying software engineering to develop a reusable framework for adaptable mobile Augmented Reality systems in ubiquitous computing environments. Since the first version of DWARF, a lot of development work and improvement has beeb done by several people on each of the different layers of the framework. There are several remaining research issues which are theses of their own. However, the basic concept of DWARF as distributed service-based system is still the same, and has proven valid for Augmented Reality system design. Acknowledgements I would like to thank all people that supported me, and apologize to all which I had not enough time for. The people I want to thank first are the original members of the DWARF team (the “seven dwarfs”): Martin Bauer, Asa MacWilliams, Florian Michahelles, Stefan Riß, Christian Sandor, Martin Wagner, and Bernhard Zaun. Without their efforts and talent, DWARF never would have been as successful as it appears to be now. The collaboration with them in the DWARF team has always been very enjoyable. In particular I want to thank Asa, who is the one who shares my interests and visions in respect to software architectures for Augmented Reality. Next, I want to thank Prof. Bernd Br¨ugge and Prof. Gudrun Klinker who had the initial idea to implement a mobile Augmented Reality system as part of an intelligent building. Their concepts for intelligent buildings developed within the OWL project at the Carnegie Mellon University, Pittsburgh and the Technische Universit¨at M¨unchen, and the vision of Augmented-Reality-ready buildings, developed into the idea of a completely distributed peer-to-peer Augmented Reality system. This concept has shown to be very flexible and open for future enhancements. For the reading and the many valuable comments on previous versions of this work, I want to thank Prof. Bernd Br¨ugge, Prof. Gudrun Klinker, and the former dwarfs and now colleagues Asa, the two Martins, and Christian. I want to thank all my friends and my family who encouraged and supported me during the last years. There was not much time for them for too long. And finally, I want to thank my two little women Gabi and Luisa for their endless patience with me during the creation of my dissertation. No one has supported me more on my way than they did. Thank you! Overview 1 Introduction 1 An introduction into the research area and the technical problem, the identification of four abstraction layers, goals, hypotheses, approach, and contributions of this dissertation. 2 Exploration of the Design Space 19 A motivating scenario, design space analysis, requirements analysis, design goals, and related work. 3 Reference Architecture and Design Patterns for Augmented Reality . 47 A reference architecture and design patterns for Augmented Reality systems. 4 The DWARF Contract-based Peer-to-Peer Architectural Style . 73 A contract-based peer-to-peer architectural style, a supporting middleware, and a graphical notation for distributed AR systems. 5 A Case Study for the DWARF Framework 119 The M3ARF sub framework of DWARF for mobile AR maintenance systems and the DWARF Pathfinder application. 6 Conclusion 145 Results and future work. A Design Patterns for Augmented Reality Systems 151 Architectural Patterns for the Application, Interaction, Presentation, Tracking, Context, and World Model subsystems. B Details on the Pathfinder Services 165 Reference documentation for DWARF Services used in Pathfinder: Pathfinder Application, CAP Router, Bluetooth Communication Service, Taskflow Engine, Tracking Manager, Optical Tracker, User Interface Engine Contents 1 Introduction 1 1.1 What is Augmented Reality? 2 1.2 Augmented Reality and Contributing Research Fields 4 1.3 Enabling Technologies 5 1.4 Goals of this Dissertation 7 1.5 Hypothesis 9 1.6 Approach . 9 1.7 Software Abstraction Layers . 10 1.7.1 Architecture Layers . 10 1.7.2 Framework Layers 12 1.7.3 The DWARF Framework 12 1.8 Contributions of this Dissertation 15 1.9 Outline 17 2 Exploration of the Design Space. 19 2.1 Maintenance of Complex Systems 19 2.2 The Design Space for Augmented Reality Systems 23 2.2.1 User Device 23 2.2.2 Device Mobility 24 2.2.3 Network Access 25 2.2.4 Component Coupling 26 2.2.5 Location Awareness . 27 2.2.6 User Interface . 27 2.3 Non-functional Requirements 28 2.4 Design Goals . 32 2.5 Related Work . 33 2.5.1 Augmented Reality Systems . 34 2.5.2 Wearable Computing Systems 42 2.5.3 Ubiquitous Computing Systems . 44 2.6 Conclusion 46 3 Reference Architecture and Design Patterns for Augmented Reality. 47 3.1 An Augmented Reality Reference Model 49 3.1.1 Overview . 51 3.1.2 Application Subsystem 53 3.1.3 Interaction Subsystem 54 3.1.4 Presentation Subsystem . 54 3.1.5 Tracking Subsystem . 58 3.1.6 Context Subsystem 60 3.1.7 World Model Subsystem . 60 3.1.8 Mapping of the ARVIKA System onto the Reference Architecture 62 3.2 Architectural Patterns for Augmented Reality Systems 66 3.2.1 A Catalogue of Patterns for Augmented Reality Systems . 66 3.2.2 A Scheme for the Description of Patterns . 68 3.2.3 A System of Patterns 70 3.3 Conclusion 72 4 The DWARF Contract-based Peer-to-Peer Architectural Style 73 4.1 A Contract-based Peer-to-Peer Architectural Style 74 4.2 The DWARF Peer-to-Peer Middleware . 80 4.2.1 Use Cases . 80 4.2.2 Functional Requirements 86 4.2.3 Non-functional Requirements 88 4.2.4 Object Models 89 4.2.5 System Design 95 4.2.6 Hardware/ Software Mapping 107 4.2.7 Persistent Data Management 108 4.3 A Graphical Notation for DWARF Systems 112 4.3.1 DWARF Service Modelling . 112 4.3.2 System Modelling 114 4.4 An Example for a Customized DWARF Service 116 4.5 Conclusion 117 5 A Case Study for the DWARF Framework 119 5.1 The Pathfinder Scenario . 120 5.2 The Minimal Mobile Maintenance Augmented Reality Framework . 122 5.2.1 DWARF Services for M3ARF 123 5.2.2 Classifying the Services into the DWARF Framework . 125 5.2.3 Mapping M3ARF to the Reference Architecture 125 5.3 The Pathfinder DWARF Services 129 5.3.1 The Connection View of Pathfinder 129 5.3.2 Pathfinder Application 129 5.3.3 Taskflow Engine . 132 5.3.4 User Interface Engine 134 5.3.5 Tracking Manager and Position Trackers 136 5.3.6 Optical Feature Tracker . 138 5.3.7 World Model . 139 5.4 Service Deployment . 141 5.5 Conclusion 144 6 Conclusion 145 A Design Patterns for Augmented Reality Systems 151 B Details on the Pathfinder Services 165 B.1 Pathfinder Application 165 B.2 Bluetooth Communication Service . 166 B.3 CAP Router 167 B.4 Taskflow Engine . 167 B.5 User Interface Engine 176 B.6 Tracking Manager and Position Trackers 184 B.7 Optical Tracker 192 B.8 World Model . 196 Bibliography 203 Acronyms 217 Index 219 List of Figures 1.1 Example for Augmented Reality 2 1.2 Head-Mounted Displays . 3 1.3 Informal Model of Augmented Reality . 4 1.4 Network types 6 1.5 Relations between architecture types and between framework types . 11 1.6 The DWARF framework . 14 2.1 User in a ubiquitous computing environment . 22 2.2 Design space for Augmented Reality systems . 31 2.3 Architecture of the ARVIKA stationary solution 35 2.4 Deployment of the ARVIKA web-based system 36 2.5 STAR system architecture 39 2.6 UbiCom system 40 2.7 Hardware architecture of the UbiCom mobile terminal 41 2.8 In the traditional deployment hardware and software are separated 43 3.1 Chapter solution domain layer . 47 3.2 MVC pattern plus Augmented Reality specific extensions 48 3.3 Subsystem decomposition of the reference model 50 3.4 Abstract Augmented Reality architecture . 52 3.5 Application subsystem 53 3.6 Interaction subsystem 55 3.7 Presentation pipeline . 56 3.8 Presentation subsystem . 57 3.9 Tracking subsystem 59 3.10 Context subsystem 61 3.11 World model subsystem . 63 3.12 ARVIKA architecture mapped onto reference architecture 65 3.13 System of AR patterns 71 4.1 Architectural style layer chapter 73 4.2 Concept of contract-based peer-to-peer style 75 4.3 Connection layers . 76 4.4 Peer-to-Peer style . 77 4.5 Peer-to-Peer architectural style model . 78 4.6 Use cases for the DWARF peer-to-peer middleware 81 4.7 Object model of a Service with two Needs and one Ability 90 4.8 Example dependency graph between Services . 91 4.9 Model of Service, Need, Ability, and Connector Descriptions . 92 4.10 Communication resources 93 4.11 DWARF Connectors . 93 4.12 Communication between Services using Connectors 94 4.13 Subsystem decomposition of the DWARF middleware 96 4.14 Communication subsystem 98 4.15 Location subsystem 99 4.16 Service Manager subsystem . 101 4.17 Display service/middleware interaction . 103 4.18 Tracker service/middleware interaction . 104 4.19 Dynamic model of a DWARF Service (UML state diagram) 105 4.20 Setting up the Service dependency graph . 106 4.21 DWARF subsystems deployment 111 4.22 Model of an Optical Tracker Service in UML 2.0 . 113 4.23 Compact model of an Optical Tracker Service in UML 2.0 113 4.24 Model of an Optical Tracker Service with DWARF UML extensions 114 4.25 Compact model of an Optical Tracker Service with DWARF UML extensions 115 4.26 The integration of the Optical Tracker Service via Needs and Abilities . 115 4.27 Customized DWARF Services: Example Optical Tracker Service 116 4.28 Simplified Service creation 117 5.1 The abstraction layers of the case study 119 5.2 The use cases of the Pathfinder application 122 5.3 The Pathfinder DWARF Services 127 5.4 The Pathfinder architecture . 128 5.5 Pathfinder connection view . 130 5.6 General idea of the Taskflow Engine Service 132 5.7 Several I/O channels for multi-modal interfaces 134 5.8 Activity diagram of data cooking of the Tracking Manager 138 5.9 Basic concept of the Optical Tracker Service 140 5.10 Prototype wearable computer 142 5.11 Service deployment 143 B.1 TaskflowEngine and -Editor boundaries 168 B.2 MVC architecture of the Taskflow Engine enhanced by a facade . 174 B.3 Taskflow Engine Service specification 175 B.4 Use cases of the User Interface Engine . 177 B.5 Conceptual view of the User Interface Engine . 181 B.6 User Interface Engine Service specification . 183 B.7 Use cases of the Tracking Manager 186 B.8 Architecture of the Tracking subsystem 189 B.9 Tracking Manager and the Position Tracker Service specifications 191 B.10 Use case of the Optical Tracker Service . 192 B.11 Optical Tracking Service specification . 195 B.12 Use cases describing the World Model’s behaviour 197 B.13 Subsystem Decomposition of the World Model Service 200 B.14 World Model Service specification 201