基于WebMagic的互联网热门榜单信息采集方案
- 爬取b站每日热门、知乎热榜、豆瓣豆列等页面榜单并进行落库；
- 集成Python3WebSpider/ProxyPool代理池
- 优化框架爬取逻辑，基于网络情况动态进行代理切换
-
Notifications
You must be signed in to change notification settings - Fork 1
FelixMundial/simple-crawler
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
Crawler on Zhihu/Bilibili/Weibo/Baidu/Douban trending items, powered by WebMagic
Topics
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published