Python批量抓取网页标题列表：并发请求与结果汇总

讲解用requests配合BeautifulSoup批量抓取多个网页的标题，线程池并发提速，把结果存成CSV清单，适合竞品监控、文章列表收集、网页索引整理。

场景痛点

手里有一堆URL，可能是从Excel导出来的、从别的页面爬出来的，需要知道每个链接对应的网页标题是什么。手动一个个打开浏览器看标题，几十个链接要点半天。写个脚本把URL丢进去，自动并发请求、自动解析title、存成清单，几分钟搞定。

用到的库

pip install requests beautifulsoup4 lxml

完整代码

# 导入库
import csv
import requests
from bs4 import BeautifulSoup
from concurrent.futures import ThreadPoolExecutor, as_completed

HEADERS = {
"User-Agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 Chrome/124.0.0.0 Safari/537.36"),
}

def fetch_title(url):
"""抓取单个URL的标题"""
try:
resp = requests.get(url, headers=HEADERS, timeout=10)
resp.encoding = resp.apparent_encoding
soup = BeautifulSoup(resp.text, "lxml")
title = soup.title.get_text(strip=True) if soup.title else ""
return (url, title, resp.status_code)
except Exception as e:
return (url, f"ERROR:{e}", 0)

def main():
# 待抓取的URL列表，实际项目里可以从CSV读
urls = [
"https://example.com",
"https://www.python.org",
"https://httpbin.org",
"https://docs.python.org",
"https://pypi.org",
]

results = []
# 5个线程并发
with ThreadPoolExecutor(max_workers=5) as pool:
futures = {pool.submit(fetch_title, u): u for u in urls}
for fut in as_completed(futures):
url, title, code = fut.result()
results.append((url, title, code))
print(f"[{code}] {title[:50]}  <-  {url}")

# 存成CSV
with open("titles.csv", "w", newline="", encoding="utf-8-sig") as f:
writer = csv.writer(f)
writer.writerow(["URL", "标题", "状态码"])
writer.writerows(results)
print(f"已保存 {len(results)} 条到 titles.csv")

if __name__ == "__main__":
main()

代码讲解

fetch_title 函数封装单个URL的抓取+解析：请求网页→解析HTML→取 <title> 标签文字。任何一步出错都返回ERROR，不会影响其他URL。
ThreadPoolExecutor(max_workers=5) 并发执行，5个URL同时请求，比串行快5倍左右。
as_completed 按完成顺序收集结果，谁先回来谁先处理，不用等最慢的那个。
结果统一存成CSV，三列：URL、标题、状态码，方便后续筛选和导入Excel。
每个请求都带统一的User-Agent，避免被反爬识别成脚本。