Cultural tourism is regarded as a key contributor for economic growth in various destinations. While increasing awareness and interest of different cultures play a key role, technology has made information and interaction at these sites more accessible and engaging. An increasing amount of research is being conducted around the potential and implementation of AR and VR technology to enhance the visitor experience. However, limited theoretical knowledge has been developed on how these need to be designed to facilitate forming memorable experiences at cultural tourism sites. This study discusses elements affecting the visitor experience and discusses how AR and VR should be designed to contribute to enhancing the experience and making it memorable from a theoretical perspective. Further research recommendations are outlined that suggest the use of complementing research methodologies to better understand the nature of experiences in order to design AR and VR application more purpose-specifically.