Given the complexity of the poverty problem, efforts and policies aiming at reducing poverty should be tailored to local conditions, including cultural, economic, social, and geographic aspects. Taking the Sichuan Province of China as the study area, this paper explores the impact of physical geographic factors on poverty using the Ordinary Least Squares (OLS) Regression and the Geographically Weighted Regression (GWR) models at the county level. In total, 28 factors classified in seven groups were considered as variables, including terrain (relief degree of the land surface, altitude, and slope); vegetation (forest coverage rate); water (drainage network density); climate (temperature, annual average rainfall); and natural disaster (landslide, debris flow, and torrential flood). The 28 variables were then tested using correlations and regressions. A total of six physical variables remained significant for the OLS and GWR models. The results showed that the local GWR model was superior to the OLS regression model and, hence, more suitable for explaining the associations between the poverty rate and physical geographic features in Sichuan. © 2020 by the authors. Licensee MDPI, Basel, Switzerland.