This study explored associations between perceived neighborhood walkability and neighborhood-based physical activity (NB-PA) and assessed possible moderation effects of the amount of time spent in the home neighborhood and individual characteristics (i.e., educational level and health-related problems). In 2016 to 2017, 509 Dutch adults, living in the South Limburg area, were included. Context-specific PA levels were measured using the Actigraph GT3X+ accelerometer and the Qstarz BTQ1000XT GPS-logger. Perceived neighborhood walkability, level of education, work status, and health-related quality of life were measured with validated self-report instruments. Results showed that individuals with a lower level of education or health-related problems spent more time in the home neighborhood. The perceived neighborhood walkability only affected NB-PA for individuals spending a relatively large amount of time in their home neighborhood. PA-facilitating features in the home neighborhood, for example, aesthetics, were only associated with more NB-PA for individuals without health-related problems or with a higher level of education.