试图抓取网址

Question

所以我试图从 Steam 上的免费游戏网站获取所有网址，但它一直返回空。我不知道我在这里做错了什么，下图显示了路径result = requests.get("https://steamdb.info/upcoming/free/")src = result.contentsoup = BeautifulSoup(src, 'lxml')urls = []for td_tag in soup.find_all('td'):    a_tag = td_tag.find('a')    urls.append(a_tag.attrs['href'])print(urls)

Answer 1

您必须使用标头User-Agent，它不能很短Mozilla/5.0，但是来自真实网络浏览器的完整字符串

import requests

from bs4 import BeautifulSoup

headers = {

    "User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0",

}

result = requests.get("https://steamdb.info/upcoming/free/", headers=headers)

soup = BeautifulSoup(result.content, 'lxml')

#print(result.content)

urls = []

for td_tag in soup.find_all('td'):

    a_tag = td_tag.find('a')

    if a_tag:

        urls.append(a_tag.attrs['href'])

print(urls)