An activity-friendly environment may increase physical activity (PA) levels and decrease sedentary behavior (SB). This study investigated associations between socio-demographic characteristics, health-related quality of life (HRQoL), perceived environment and objectively measured PA outcomes. Socio-demographic characteristics were assessed using a questionnaire and HRQoL was measured using the EQ-5D. The Neighborhood Environment Walkability Scale (NEWS-A) was used to assess the perceived environment. SB, light PA (LPA) and moderate-to-vigorous PA (MVPA) were measured using the Actigraph GT3X+. Data from 622 Dutch adults were used in multivariate linear regression analyses to investigate associations between NEWS-A and PA outcomes. Analyses were controlled for socio-demographic characteristics and HRQoL. The presence of attractive buildings was associated with less SB ( = ?0.086, p < 0.01) and more MVPA ( = 0.118, p < 0.01). Presence of destinations within walking distance was also positively associated with MVPA ( = 0.106, p < 0.01). Less crime was associated with less MVPA ( = 0.092, p < 0.05). Interactions between personal and environmental characteristics showed that the absence of PA-hindering characteristics (e.g., heavy traffic) was associated with less SB and more MVPA, but only for residents with problems regarding pain and usual activities. The presence of PA-facilitating characteristics (e.g., aesthetics and destinations) was associated with less SB, more LPA and more MVPA but only for the more advantaged people in society. Results suggest that to reduce health inequalities, it would be more helpful to remove barriers rather than introduce PA facilitating characteristics.