使用 BeautifulSoup 抓取 shopee.co.id 时得到一个空列表

Question

我试图使用 BeautifulSoup 包在https://shopee.co.id/top_products上抓取顶级产品的名称和销售额。但无论我尝试什么，它总是返回一个空的产品列表。你能解释一下幕后发生的事情吗？这是我的代码：import requestsfrom bs4 import BeautifulSoup as bsheaders = {    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'}shopee_url = 'https://shopee.co.id/top_products'response = requests.get(shopee_url, headers=headers)soup = bs(response.text, "html.parser")products = soup.find_all("._3S8sOC _2QfAXF")print(type(products))print(products)感谢您阅读到这里。我读过一些类似的答案，但它们似乎并不适用于此。

Answer 1

如果您愿意使用 selenium，您可以使用 selenium 提出请求：

from selenium import webdriver

from bs4 import BeautifulSoup as bs

headers = {

    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',

    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'

}

driver = webdriver.Firefox(executable_path='c:/program/geckodriver.exe')

shopee_url = 'https://shopee.co.id/top_products'

driver.get(shopee_url)

driver.implicitly_wait(15)

response = driver.page_source

driver.close()

soup = bs(response, "html.parser")

products = soup.select(".shopee-item-card__lower-padding")

#print(products)

for product in products:

    name = product.select_one('.shopee-item-card__text-name span')

    print(name.get_text(strip=True))

    price = product.select_one('.shopee-item-card__current-price')

    print(price.text)

请注意，您需要安装 Firefox、geckodriver 和 selenium。

输出：

Kuota Telkomsel Cek Dadakan

Rp300

Aigo 1GB 5Hari

Rp7.375

Paket Data Tri Mix Small 5GB

Rp4.500

Paket Data Tri Mix Small 2.75GB

Rp9.500

*PROMO MURAH* SURPRISE DEAL PAKET DATA INTERNET KUOTA TELKOMSEL SIMPATI AS LOOP OMG 20GB 30GB 50GB

Rp1.000

Kuota Telkomsel Cek Dadakan

Rp500

Paket Data Telkomsel 12GB

Rp98.000

Paket Data Tri AON Unlimited + 6GB

Rp58.000

KUOTA INTERNET PAKET DATA INTERNET TELKOMSEL KUOTA DATA FLASH Kartu Simpati As Loop (KUOTA MURAH)

Rp1.000

[MURAH] PAKET DATA TELKOMSEL ∣ KUOTA INTERNET ∣ PAKET DATA ∣ SIMPATI  ∣ AS  ∣ LOOOP

Rp1.000

Paket Data Tri AddOn 12GB

Rp48.000

Paket Data  Indosat Freedom U 7GB+20GB Apps / 30 Hari

Rp66.500

PAKET DATA TELKOMSEL ISI ULANG SIMPATI 14GB/27GB/37GB - AS 6,5GB/10GB/17GB/32GB - LOOP 3GB/8GB/15GB

Rp38.000

Paket Data Indosat Freedom Internet 4GB / 30 Hari

Rp24.000

Paket Data Indosat Freedom U 1GB+4.5GB Apps / 30 Hari

Rp24.000

KUOTA TELKOMSEL paket kuota data internet telkomsel termurah 52gb injek injeck kuota  promo

Rp1.000

Kuota Telkomsel Simpati 52GB 50GB 24jam Langsung Masuk

Rp1.000

Paket Data Telkomsel 50GB

Rp195.000

Paket Data Indosat Freedom Internet 10GB /30 Hari

Rp48.000

KUOTA TELKOMSEL paket data internet telkomsel kuota internet TELKOMSEL  termurah simpati as loop

Rp1.000

Paket Data Telkomsel Data 25.000

Rp23.500

VOCHER VOUCHER KUOTA AXIS AIGO 1GB 5HARI NASIONAL

Rp7.390

PAKET DATA KUOTA INDOSAT [DISKON 15%] TERMURAH

Rp1.000

VOUCHER Axis 1gb 5 hari 24 Jam KUOTA 1 GB

Rp7.500

Paket Data Indosat Freedom Combo 14GB, 30 Hari

Rp48.500

[PROMO] KUOTA BESAR PAKET OMG TELKOMSEL MURAH

Rp1.000

Paket Data Indosat Freedom Combo 8GB/ 30 Hari

Rp33.500

TELKOMSEL PEMBAYARAN RESELLER/ AGEN atau Penambahan Selisih Harga Produk ThalhahCell

Rp1.000

Aigo 3GB 15Hari

Rp18.500

Paket Data Indosat Freedom U 3GB+15GB Apps / 30 Hari

Rp49.900