The world is undergoing multiple crises that require resilience to withstand them. The built environment can significantly enhance or weaken society’s (and individuals’) resilience. However, understanding of resilience in the built environment is scattered and manifold–whilst the design of buildings primarily focuses on the restoration of buildings’ physical characteristics, urban policies centre on the recovery of society. Scholars highlight the need for a holistic approach where different resilience concepts merge to improve the resilience of people and communities. For this, understanding the relationship between people and places is crucial. Thus, the aim of this paper is to deepen the understanding of the social resilience concept in relation to the built environment and how the built environment can enhance it. This is achieved through an extensive literature review, concept mapping and panel discussion. The built environment characteristics affecting individual and/or community resilience are identified, and a conceptual model is provided, attempting to visualize the relationship between the constructs. The paper’s novelty lies in its multidisciplinary approach and integration of various social science knowledge in the context of the built environment. Furthermore, it emphasizes the built environment’s role in supporting social resilience, which has been often overlooked previously.