目录

前言 1.1

PySpider简介 1.2

PySpider安装与基本用法 1.3

PySpider安装 1.3.1

安装和启动的常见问题 1.3.1.1

PySpider基本用法 1.3.2

项目操作举例 1.3.2.1

查找提取元素 1.3.2.2

PySpider的高级用法 1.4

self.crawl 1.4.1

config.json 1.4.2

data目录 1.4.3

phantomjs 1.4.4

PySpider经验与心得 1.5

PySpider的心得 1.5.1

删除项目 1.5.1.1

PySpider常见的坑 1.5.2

PySpider案例 1.6

汽车之家的品牌等数据 1.6.1

汽车之家的车型详细数据 1.6.2

过程 1.6.2.1

代码 1.6.2.2

结果 1.6.2.3

百度热榜 1.6.3

附录 1.7

参考资料 1.7.1

前言

Python爬虫框架：PySpider

最新版本：v2.0

更新时间：20210908

简介

PySpider是一个简单易用且强大的Python主流爬虫框架。此处总结PySpider的安装和基本的使用，以及安装和启动时常见问题，并且给出查找定位元素的PyQuery的基本用法举例，以及详细描述了项目中从新建到调试的每一步的细节如何操作，以及一些高级用法，比如self.craw、config.json、data目录、phantomjs，和一些心得，比如删除项目，以及常见的坑，并且给出一些实际的例子供参考，包括汽车之家的品牌车型车系数据、汽车之家的车型详细数据、百度首页热榜列表等。

源码+浏览+下载

本书的各种源码、在线浏览地址、多种格式文件下载如下：

Gitbook源码

crifan/python_spider_pyspider: Python爬虫框架：PySpider

如何使用此Gitbook源码去生成发布为电子书

详见：crifan/gitbook_template: demo how to use crifan gitbook template and demo

在线浏览

Python爬虫框架：PySpider book.crifan.com

Python爬虫框架：PySpider crifan.github.io

离线下载阅读

Python爬虫框架：PySpider PDF

Python爬虫框架：PySpider ePub

Python爬虫框架：PySpider Mobi

版权说明

此电子书教程的全部内容，如无特别说明，均为本人原创和整理。其中部分内容参考自网络，均已备注了出处。如有发现侵犯您版权，请通过邮箱联系我 admin 艾特 crifan.com，我会尽快删除。谢谢合作。

鸣谢

感谢我的老婆陈雪的包容理解和悉心照料，才使得我crifan有更多精力去专注技术专研和整理归纳出这些电子书和技术教程，特此鸣谢。

更多其他电子书

本人crifan还写了其他100+本电子书教程，感兴趣可移步至：

crifan/crifan_ebook_readme: Crifan的电子书的使用说明

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-09-08 13:59:23

PySpider简介

PySpider简介

PySpider的基本信息：

是个Python的爬虫框架

最大特点：

带图形界面WebUI的调试

简单易用

同时功能也很强大

GitHub

https://github.com/binux/pyspider

文档：

官方，英文：http://docs.pyspider.org/

非官方，中文：http://www.pyspider.cn/index.html

作者

网名：Binux

别名：足叉兆虫

博客：Binuxの杂货铺

Github: binux (Roy Binux)

PySpider对比Scrapy

对于两个流行的Python的爬虫框架，PySpider和Scrapy，常常会被人拿来对比。

对此，之前简单总结如下：

PySpider：简单易上手，带图形界面（基于浏览器页面）

一图胜千言：在WebUI中调试爬虫代码

Scrapy：可以高级定制化实现更加复杂的控制

一图胜千言：Scrapy一般是在命令行界面中调试页面返回数据：

详见： 【整理】pyspider vs scrapy

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2020-07-30 14:00:04

PySpider安装与基本用法

PySpider安装与基本用法

概述：

安装

  pip install pyspider

启动

  pyspider

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-04-14 22:39:25

PySpider安装

PySpider安装

下面介绍，如何安装PySpider。

Mac中安装PySpider

此处以Mac中为例，去介绍如何安装PySpider。

如果只是简单的直接安装的话，则可以去：

pip install pyspider

安装phantomjs

如果后续用到网页中要允许js的代码，则需要用到无头浏览器phantomjs，就需要先去安装phantomjs

brew cask install phantomjs

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-01-16 21:34:45

安装和启动的常见问题

安装和启动的常见问题

此处整理在安装和启动pyspider期间，经常遇到的问题和其原因及解决办法。

启动报错：async=True SyntaxError: invalid syntax

运行pyspider报错：

  File "/Users/limao/.pyenv/versions/3.8.0/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyspider/run.py", line 231
    async=True, get_object=False, no_input=False):
    ^
SyntaxError: invalid syntax

原因：PySpider（很久没继续维护了）最新支持版本是Python 3.6，其把async作为普通函数参数，是没问题的。

但是Python 3.7之后把async改为了系统保留字，表示异步，所以asyc不能再作为普通函数参数名

所以当前环境是Python 3.7+时，就会报语法错误了

解决办法：2种思路：

（自己）把代码（中这类的语法错误）都改掉

把当前Python版本换成旧版本：Python 3.6

此处选择换旧版本Python 3.6

步骤：

Mac中换Python为3.6

此处以pyenv为例：

先去安装Python 3.6的某个版本：

pyenv install 3.6.8

再去设置使用Python 3.6：

本地

pyenv local 3.6.8

或全局

pyenv global 3.6.8

然后重新安装：

pip install pyspider

之后即可正常运行

pyspider

详见：

【已解决】Mac中给Python3安装PySpider

推荐用虚拟环境

在Python的开发中，为了避免不同开发环境的互相影响，一般都会用虚拟环境工具，比如pipenv、virtualenv。

注意：此处即使用Python虚拟环境，由于前面提到的问题，PySpider也还是要用Python 3.6的

pipenv中安装PySpider

此处是pipenv，为了指定使用Python 3.6则，可以在Pipfile中加上：

[requires]
python_version = "3.6"

贴出完整配置：

[[source]]
#url = "https://pypi.python.org/simple"
url = "https://pypi.tuna.tsinghua.edu.cn/simple"
verify_ssl = true
name = "pypi"

[packages]
pymysql = "*"

[dev-packages]

[requires]
python_version = "3.6"

创建并安装PySpider：

pipenv install pyspider

或：

先创建虚拟环境，再安装PySpider

pipenv install
pipenv shell
pip install pyspider

virtualenv中安装PySpider

关于virtualenv：

创建虚拟环境：virtualenv venv

激活激活环境并进入：

Mac/Linux：source venv/bin/activate

或：

. venv/bin/activate

Win：venv\Scripts\activate.bat

退出虚拟环境：deactivate

在virtualenv中安装PySpider：

pip install pyspider

启动报错：pycurl报ImportError错 或 Curl报ConfigurationError错

Mac中，如果安装期间出错：

__main__.ConfigurationError: Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using

或运行时报错：

ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)

原因：此处导入pycurl时，发现libcurl运行时所依赖的ssl的底层是openssl，和当时编译时的版本不匹配

解决办法：重新编译安装，使得版本一致

步骤：

pip uninstall -y pycurl
export PYCURL_SSL_LIBRARY=openssl
export LDFLAGS=-L/usr/local/opt/openssl/lib;export CPPFLAGS=-I/usr/local/opt/openssl/include;pip install pycurl --compile --no-cache-dir

附上，进入虚拟环境后再操作的例子：

➜  ChildQuPeiYinApp_downloadDemo pipenv shell
Launching subshell in virtual environment…
 . /Users/crifan/.local/share/virtualenvs/ChildQuPeiYinApp_downloadDemo-IRhFV7xw/bin/activate
➜  ChildQuPeiYinApp_downloadDemo  . /Users/crifan/.local/share/virtualenvs/ChildQuPeiYinApp_downloadDemo-IRhFV7xw/bin/activate
➜  ChildQuPeiYinApp_downloadDemo pip uninstall pycurl
Skipping pycurl as it is not installed.
➜  ChildQuPeiYinApp_downloadDemo export PYCURL_SSL_LIBRARY=openssl
➜  ChildQuPeiYinApp_downloadDemo export LDFLAGS=-L/usr/local/opt/openssl/lib;export CPPFLAGS=-I/usr/local/opt/openssl/include;pip install pycurl --compile --no-cache-dir
Collecting pycurl
  Downloading https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz (214kB)
    100% |████████████████████████████████| 215kB 198kB/s
Installing collected packages: pycurl
  Running setup.py install for pycurl ... done
Successfully installed pycurl-7.43.0.2

注意：上述的：

/usr/local/opt/openssl是你的openssl安装路径

如果你的不是这个路径， 要换成你的实际路径

对应的

/usr/local/opt/openssl/lib是lib库的路径

/usr/local/opt/openssl/include是include头文件的路径

详见：

【记录】Mac中安装和运行pyspider

【已解决】pipenv虚拟环境中用pip安装pyspider出错：、__main__.ConfigurationError: Curl is configured to use SSL, but we have not been able to determine which SSL backend it is using

【已解决】pyspider运行出错：ImportError pycurl libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)

启动报错：fatal error openssl/ssl.h file not found

如果上面步骤：

export LDFLAGS=-L/usr/local/opt/openssl/lib;export CPPFLAGS=-I/usr/local/opt/openssl/include;pip install pycurl --compile --no-cache-dir

报错：

    In file included from src/docstrings.c:4:
    src/pycurl.h:165:13: fatal error: 'openssl/ssl.h' file not found
    #   include <openssl/ssl.h>
    1 error generated.
    error: command 'gcc' failed with exit status 1

直接原因：找不到openssl/ssl.h

多种可能

之前没安装过openssl

解决办法：Mac中去安装：

brew install openssl

然后再重试即可

Mac中已安装过openssl

所以此处Mac中是有openssl/ssl.h的，只是传入的路径不对

解决办法：找到已安装的openssl的实际路径，传入正确的路径。

步骤：

找到已安装的openssl的实际安装路径

brew info openssl

可以看到有：

/usr/local/Cellar/openssl@1.1/1.1.1d (7,983 files, 17.9MB)

其中的：

/usr/local/Cellar/openssl@1.1/1.1.1d

就是我们要的，此处openssl的实际安装路径

通过

open /usr/local/Cellar/openssl@1.1/1.1.1d

确认是有对应的：

/usr/local/Cellar/openssl@1.1/1.1.1d/include/openssl/ssl.h

这个文件的。所以传入路径应该改为：

/usr/local/Cellar/openssl@1.1/1.1.1d/include

完整命令是：

export LDFLAGS=-L/usr/local/opt/openssl/lib;export CPPFLAGS=-I/usr/local/Cellar/openssl@1.1/1.1.1d/include;pip install pycurl --compile --no-cache-dir

详见：

【已解决】Mac中pip安装pycurl报错：fatal error openssl/ssl.h file not found

启动报错：Error Could not create web server listening on port 25555

现象：运行pyspider时能看到有错误

Error: Could not create web server listening on port 25555

原因：之前已启动过pyspider，其内部会默认启动phantomjs，而虽然之前虽然已关闭掉pyspider，但是没有杀掉phantomjs的进程，导致端口25555被占用，而报错

解决办法：杀掉端口是25555的phantomjs进程即可

步骤：

找phantomjs进程ID：

ps aux | grep 25555

杀掉对应进程

kill -9 xxx

举例

✘ limao@xxx  ~/dev/crifan/python/demo_spider  ps aux | grep 25555
limao            35620   0.0  0.0  4277272    820 s002  R+   10:27上午   0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn 25555
limao            33983   0.0  0.4  6130968  34128 s002  S    10:17上午   0:30.45 phantomjs --ssl-protocol=any --disk-cache=true /Users/limao/.pyenv/versions/3.6.8/lib/python3.6/site-packages/pyspider/fetcher/phantomjs_fetcher.js 25555
 limao@xxx  ~/dev/crifan/python/demo_spider  kill -9 33983

之后即可正常启动pyspider，且能看到phantomjs可以正常启动了：

phantomjs fetcher running on port 25555

启动报错：Deprecated option domaincontroller use http_authenticator.domain_controller instead

启动报错：

  File "/Users/limao/.pyenv/versions/3.6.8/lib/python3.6/site-packages/wsgidav/wsgidav_app.py", line 118, in _check_config
    raise ValueError("Invalid configuration:\n  - " + "\n  - ".join(errors))
ValueError: Invalid configuration:
  - Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead.

原因：wsgidav版本兼容问题

解决办法：换兼容的没问题的旧版本2.4.1

步骤：

pip install wsgidav==2.4.1

启动报错：ImportError cannot import name DispatcherMiddleware

启动报错：

  File "/Users/limao/.pyenv/versions/3.6.8/lib/python3.6/site-packages/pyspider/webui/app.py", line 64, in run
    from werkzeug.wsgi import DispatcherMiddleware
ImportError: cannot import name 'DispatcherMiddleware'

原因：werkzeug版本兼容问题

解决办法：换兼容的没问题的旧版本0.16.1

步骤：

pip install werkzeug==0.16.1

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-04-14 23:05:53

PySpider基本用法

PySpider基本用法

使用PySpider的基本步骤

下面来介绍一下PySpider的使用的步骤和操作：

运行PySpider

在某个目录下的终端命令行中输入

pyspider

即可启动运行，输出举例：

 pyspider
phantomjs fetcher running on port 25555
[I 200731 10:28:35 result_worker:49] result_worker starting...
[I 200731 10:28:35 processor:211] processor starting...
[I 200731 10:28:35 tornado_fetcher:638] fetcher starting...
[I 200731 10:28:35 scheduler:647] scheduler starting...
[I 200731 10:28:35 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 200731 10:28:35 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 200731 10:28:35 app:84] webui exiting...

注：

如果是用虚拟环境安装的PySpider，记得先进入虚拟环境后再运行PySpider

比如用的pipenv，则是

pipenv shell
pyspider

pyspider等价于pyspider all

进入WebUI

然后去用浏览器打开：

http://0.0.0.0:5000/

即可进入爬虫的管理界面=WebUI

新建爬虫项目

点击Create，去新建一个爬虫项目

输入：

爬虫名称：

入口地址：自动生成的代码中，会作为起始要抓取的url

也可以不填

后续可以在代码中修改

然后再点击新建的爬虫项目，进入调试页面

新建出来的项目，默认状态是TODO

点击新建出来的项目名，直接进入调试界面

然后右边是编写代码的区域

左边是调试的区域，用于执行代码，显示输出信息等用途

调试爬虫代码

编写代码，调试输出信息，保存代码

调试代码期间，对于想要返回上一级：

先说之前不熟悉的时候的操作：

之前调试运行时，不知道还有回到上一级，在想要返回上一级时，都直接是点击左上角的项目名字

返回项目列表：

然后重新进去，重新点击Run，直到跑到对应的层级，去继续调试。

再说后来知道了PySpider内置支持这种逻辑操作：

PySpider对在调试期间所需要在上一个连接和下一个连接之间切换的操作，支持的很好：

点击 < | > 的 < 或 >，则可以 返回上一级 或 进入下一级

实际效果演示：

想要返回上一级的爬取函数的话，点击 左箭头

然后再点击Run：

然后就可以返回上一级了。

然后也才注意到，每行的follow的左边开始显示的是：callback函数名

此处的是detail_page

而对应的上一级的结果中，也是上一级的callback：

运行爬虫去爬取数据

调试完毕后，返回项目，status改为DEBUG或RUNNING，点击Run

想要暂停运行：status改为STOP

保存已爬取的数据

当爬取完毕数据，需要保存下来时，可以有多种保存方式：

mysql数据库

MongoDB数据库

CSV或Excel文件

保存到csv或Excel文件

基本思路：确保自己代码中，最后return返回的字段是你要的字段

如何得到CSV文件：在任务运行期间或完毕后，去Results-》点击下载CSV，即可得到你要的csv格式的数据文件。

结果：PySpider会自动在已有字段中加上额外的url字段

用VSCode编辑csv文件

如果想要去除多余的不需要的url字段，则可以通过文本编辑器，比如VSCode去列编辑模式，批量删除，或者查找和替换，都可以实现

最后会多余一列，标题是 …，内容全是,{}，所以直接用编辑器比如VCScode去替换为空以清空，即可

详见：

【已解决】PySpider如何把json结果数据保存到csv或excel文件中 – 在路上

Excel去打开CSV文件结果乱码

csv文件编码默认为UTF8（是好事，通用的），但是如果用（不论是Mac还是Win中的）excel去打开，结果（估计对于中文系统，都是）会默认以GKB（或GB18030）打开，所以会乱码

解决办法：【已解决】Mac或Win中用Excel打开UTF8编码的csv文件显示乱码

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-04-14 20:26:27

项目操作举例

项目操作举例

此处就以一个简单的项目为例来说明，从头到尾是如何运行和操作的：

首次运行pyspider，会提示是否运行使用网络，点击允许

输出：

➜  ChildQuPeiYinApp_downloadDemo pyspider
phantomjs fetcher running on port 25555
[I 180925 11:25:51 result_worker:49] result_worker starting...
[I 180925 11:25:51 processor:211] processor starting...
[I 180925 11:25:51 tornado_fetcher:638] fetcher starting...
[I 180925 11:25:52 scheduler:647] scheduler starting...
[I 180925 11:25:52 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 180925 11:25:52 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 180925 11:25:52 app:76] webui running on 0.0.0.0:5000

打开页面：

http://0.0.0.0:5000

去Create New Project新建项目

进入项目调试界面：

编写代码，或者已写好代码后去粘贴代码，再点击保存：

接着点击Run，开始运行。

会出现Follow，点击Follow

其中Follow后的3，指的是有产生了3条请求链接，可供后续继续访问

点击第一个的三个点，去展开：

可以看到当前请求的详细参数

点击右箭头=>

可以看到输出

然后再点击后续的链接，去运行：

点击 左箭头=< 返回上一级：

再点击Run

回到上一级的输出了：

如此，即可，根据需要去，反复的：

点击某个请求的Run，进入下一级

点击<返回上一级

去调试，直到得到你需要的结果，即可完成。

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-04-14 23:02:46

查找提取元素

PySpider中查找提取元素

PySpider中内置的用于查找和定位html网页中元素的库是：PyQuery

PyQuery算是一个css选择器，模拟JS领域的jQuery，所以叫做PyQuery。

具体细节是，PySpider针对html的响应response，默认提供了一个doc属性，其内置了PyQuery解析后结果，所以你可以用response.doc("your_css_selector")去选择你要的html中的内容了。

而具体的your_css_selector的写法，则就变成PyQuery的写法了。

举例：提取汽车之家车型车系相关数据

下面通过想象的例子来解释，PyQuery的常见用法：

比如想要提取：

<ul class="rank-list-ul" 0="">
  <li id="s3170">
...
    <div>
      ...
      <a id="atk_3170" href="//car.autohome.com.cn/pic/series/3170.html#pvareaid=103448">图库</a>
    ...
    </div>
  </li>
...
</ul>

中的href的值

语言描述可以是：class是rank-list-ul的ul元素，下面的li，下面的div，下面的a，且href值是包含/pic/series的

转换成PyQuery的语法是

.rank-list-ul li div a[href*="/pic/series"]

也可以换成另外的写法：

ul[class='rank-list-ul'] li div a[href*="/pic/series"]

ul[class='rank-list-ul'] a[href*="/pic/series"]

如果你确定此规则不会误匹配其他元素，也可以省略中间的节点的查找

对应PySpider的代码是：

获取匹配到的第一个元素

  firstMatchADoc = response.doc('.rank-list-ul li div a[href*="/pic/series"]')

获取到所有匹配到的元素

  for eachADoc in response.doc('.rank-list-ul li div a[href*="/pic/series"]').items():
    print("eachADoc=%s" % eachADoc)

以及对于：

<dl id="33" olr="6">
  <dt><a href="//car.autohome.com.cn/price/brand-33.html#pvareaid=2042362"><img width="50" height="50"
        src="//car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png"></a>
    <div><a href="//car.autohome.com.cn/price/brand-33.html#pvareaid=2042362">奥迪</a></div>
  </dt>
  <dd>

    <div class="h3-tit"><a href="//car.autohome.com.cn/price/brand-33-9.html#pvareaid=2042363">一汽-大众奥迪</a></div>
    <ul class="rank-list-ul" 0>

      <li id="s3170">
        <h4><a href="//www.autohome.com.cn/3170/#levelsource=000000000_0&pvareaid=101594">奥迪A3</a></h4>
        <div>指导价：<a class="red" href="//www.autohome.com.cn/3170/price.html#pvareaid=101446">19.32-23.46万</a></div>
        <div><a href="//car.autohome.com.cn/price/series-3170.html#pvareaid=103446">报价</a> <a id="atk_3170"
            href="//car.autohome.com.cn/pic/series/3170.html#pvareaid=103448">图库</a> <a data-value="3170"
            class="js-che168link" href="//www.che168.com/china/series3170/">二手车</a> <a
            href="//club.autohome.com.cn/bbs/forum-c-3170-1.html#pvareaid=103447">论坛</a> <a
            href="//k.autohome.com.cn/3170/#pvareaid=103459">口碑</a></div>

      </li>

      <li id="s692">
        <h4><a href="//www.autohome.com.cn/692/#levelsource=000000000_0&pvareaid=101594">奥迪A4L</a></h4>
        <div>指导价：<a class="red" href="//www.autohome.com.cn/692/price.html#pvareaid=101446">30.58-39.68万</a></div>
        <div><a href="//car.autohome.com.cn/price/series-692.html#pvareaid=103446">报价</a> <a id="atk_692"
            href="//car.autohome.com.cn/pic/series/692.html#pvareaid=103448">图库</a> <a data-value="692"
            class="js-che168link" href="//www.che168.com/china/series692/">二手车</a> <a
            href="//club.autohome.com.cn/bbs/forum-c-692-1.html#pvareaid=103447">论坛</a> <a
            href="//k.autohome.com.cn/692/#pvareaid=103459">口碑</a></div>

      </li>
...
...
...

对应的代码是：

想要从

<a href="//car.autohome.com.cn/price/brand-33.html#pvareaid=2042362"><img width="50" height="50" src="//car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png"></a>

获取brand的logo的img的代码：

brandDoc = response.doc('dl dt')
brandLogoDoc = brandDoc.find('a img')
brandLogoUrl = brandLogoDoc.attr["src"]

从：

<div><a href="//car.autohome.com.cn/price/brand-33.html#pvareaid=2042362">奥迪</a></div>

中获取brand的name的a的代码：

brandNameDoc = brandDoc.find('div a')
brandName = brandNameDoc.text()

从：

<div class="h3-tit"><a href="//car.autohome.com.cn/price/brand-33-9.html#pvareaid=2042363">一汽-大众奥迪</a></div>

获取merchant的所有的a的代码：

merchantDocGenerator = response.doc("dd div[class='h3-tit'] a").items()
merchantDocList = list(merchantDocGenerator)
merchantDocLen = len(merchantDocList)

注意：.items()返回的是generator，想要得到list，需要用list(yourGenerator)去转换得到

从：

<ul class="rank-list-ul" 0>
...

获取rank-list-ul的class的dd下面的ul的merchant的代码：

merchantRankDocGenerator = response.doc("dd ul[class='rank-list-ul']")
merchantRankDocList = list(merchantRankDocGenerator)
merchantRankDocListLen = len(merchantRankDocList)

以及获取每个元素：

属性值：用attr

类型是：dict

字符串值：用text()

类型是：str

举例：

for curIdx, merchantItem  in enumerate(merchantDocList):
    merchantName = merchantItem.text()
    merchantItemAttr = merchantItem.attr
    merchantUrl = merchantItemAttr["href"]

PyQuery资料

response.doc返回后的PyQuery对象，之后可以继续用PyQuery去操作

此处列出PyQuery的一些典型的操作函数：

PyQuery.filter(selector)

PyQuery.find(selector)

PyQuery.items(selector=None)

PyQuery.siblings(selector=None)

另外，常见的一些属性来说：

PyQuery.text(value=<NoDefault>)：当前节点的text文本值

PyQuery.html(value=<NoDefault>, **kwargs)：当前节点的html值

详见：

官网文档

pyquery – PyQuery complete API — pyquery 1.2.4 documentation

Traversing — pyquery 1.2.4 documentation

Attributes — pyquery 1.2.4 documentation

CSS — pyquery 1.2.4 documentation

独立教程

HTML解析库Python版jQuery：PyQuery

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-01-16 21:34:45

PySpider的高级用法

PySpider的高级用法

下面介绍PySpider中，除了基本用法之外的，可以算作是高级，稍微更加复杂一些的用法。

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2020-07-30 14:00:04

self.crawl

self.crawl详解

官方文档

此处要介绍的PySpider的获取网络请求的函数是：self.crawl，功能很强大。

具体详细的解释，可以参考：

官网的英文文档：

比如： crawl参数 params

某热心网友整理的 中文文档：

self.crawl - pyspider中文文档 - pyspider中文网

给GET的请求添加查询参数

给self.crawl中给params传递对应字典变量，PySpider内部会自动把字典编码为url的查询参数 query string.

官方实例：

self.crawl('http://httpbin.org/get', callback=self.callback, params={'a': 123, 'b': 'c'})

等价于：

self.crawl('http://httpbin.org/get?a=123&b=c', callback=self.callback)

自己之前用的例子有：

  topSignTopParam = {
      "start": 0,
      "rows": 20
  }
  self.crawl(TopSignTopUrl,
      callback=self.getMoreUserCallback,
      params=topSignTopParam,
      save={
          "baseUrl": TopSignTopUrl,
          "isNeedCheckNextPage": True,
          "curPageParam": topSignTopParam
      }
  )

给callback函数加上额外的参数

使用self.crawl的save参数即可，然后callback中用response.save获取传入的值

举例：

    def getUserDetail(self, userId):
        self.crawl(UserDetailUrl,
            callback=self.userDetailCallback,
            params={"member_id": userId },
            save=userId
        )

    def userDetailCallback(self, response):
        userId = response.save
        print("userId=%s" % userId)

当请求出错时也执行callback回调函数

需要给callback回调函数加上修饰符@catch_status_code_error

举例：

    def picSeriesPage(self, response):
      ...
      self.crawl(curSerieDict["url"], callback=self.carModelSpecPage, save=curSerieDict)

    @catch_status_code_error
    def carModelSpecPage(self, response):
        curSerieDict = response.save
        print("curSerieDict=%s", curSerieDict)
        ...
        return curSerieDict

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2020-07-30 14:00:04

config.json

独立的配置文件: config.json

如果需要用到复杂一点的配置，比如result worker，则是需要单独写配置文件。

PySpider的配置文件一般叫做config.json

比如用如下内容：

{
  "taskdb":     "mysql+taskdb://root:crifan_mysql@127.0.0.1:3306/AutohomeTaskdb",
  "projectdb":  "mysql+projectdb://root:crifan_mysql@127.0.0.1:3306/AutohomeProjectdb",
  "resultdb":   "mysql+resultdb://root:crifan_mysql@127.0.0.1:3306/AutohomeResultdb",
  "result_worker":{
      "result_cls": "AutohomeResultWorker.AutohomeResultWorker"
   }
}

将config.json保存在pyspider命令运行所在的当前目录下：

然后去-c指定配置文件：

pyspider -c config.json

举例

一个配置更简单的config.json

{
    "webui": {
        "port": 5000,
        "need-auth": false
    },
    "scheduler": {
        "delete_time": 30
    }
}

运行调用效果：

 pyspider -c use_python_spider_framework/config.json
[I 200731 14:48:06 result_worker:49] result_worker starting...
phantomjs fetcher running on port 25555
[I 200731 14:48:06 processor:211] processor starting...
[I 200731 14:48:06 scheduler:647] scheduler starting...
[I 200731 14:48:06 tornado_fetcher:638] fetcher starting...
[I 200731 14:48:06 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333
[I 200731 14:48:06 scheduler:126] project crawlBaiduHotList_PySpider_0731_1436 updated, status:DEBUG, paused:False, 0 tasks
[I 200731 14:48:06 scheduler:965] select crawlBaiduHotList_PySpider_0731_1436:_on_get_info data:,_on_get_info
[I 200731 14:48:06 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0
[I 200731 14:48:06 tornado_fetcher:188] [200] crawlBaiduHotList_PySpider_0731_1436:_on_get_info data:,_on_get_info 0s
[D 200731 14:48:06 project_module:145] project: crawlBaiduHotList_PySpider_0731_1436 updated.
[I 200731 14:48:06 processor:202] process crawlBaiduHotList_PySpider_0731_1436:_on_get_info data:,_on_get_info -> [200] len:12 -> result:None fol:0 msg:0 err:None
[I 200731 14:48:06 scheduler:360] crawlBaiduHotList_PySpider_0731_1436 on_get_info {'min_tick': 0, 'retry_delay': {}, 'crawl_config': {}}
[I 200731 14:48:06 app:76] webui running on 0.0.0.0:5000

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-04-14 20:55:10

data目录

PySpider所在目录下的data目录

在你运行pyspider后，自动会在命令执行路径下生成data文件夹，其中包含几个（SQLite）文件：

project.db：保存了用户的爬虫项目相关信息，包括项目的Python代码

比如用（SQlite）工具去查看，可以看到详细数据

比如Mac中的DB Browser for SQLite查看的效果：

Python代码：

对应数据库结构字段：

result.db：项目运行的结果数据

task.db：项目相关的任务信息

其中如果开始运行爬虫，还会出现相关的调度信息：

scheduler.all, scheduler.1d, scheduler.1h：保存了任务执行后所有，1天，1小时内相关的信息，和WebUI中的progress中的all, 1d, 1h对应：

指定data目录

用data-path参数

方法1：配置config.json中的data-path

{
  "data-path": "/root/xxx/crawler/pyspider/data",
  "webui": {
    "port": 7700,
    "username": "admin",
    "password": "yourPassword",
    "need-auth": true
  },
  "scheduler": {
    "delete_time": 30
  }
}

方式2：命令行传递参数--data-path

--data-path="your_data_folder_path"

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-04-14 20:51:12

phantomjs

Phantomjs

如何解决部分页面内部不显示，无法抓取的问题？

折腾【已解决】PySpider中页面部分内容不显示 – 在路上，遇到个问题：

页面中的部分内容不显示，所以无法抓取。

经过研究发现，其实是：

这部分不显示的内容，是原网页中通过后续调用js去生成和获取的，所以可以通过：

给self.crawl添加

fetch_type='js'

会使得内部调用phantomjs，模拟js，渲染生成页面内容。

从而，此处PySpider，在这种需要显示js加载的页面内容时，可以利用phantomjs。

用了phantomjs后又出错：FETCH_ERROR HTTP 599 Connection timed out after milliseconds

后续继续运行，加了fetch_type='js'的代码，去爬取页面数据，结果遇到了：

【未解决】pyspider运行出错：FETCH_ERROR HTTP 599 Connection timed out after milliseconds

尝试了多种办法，都无法解决此问题。

所以目前的情况是：

如果加了phantomjs，结果在大量爬取页面期间，又会导致出错FETCH_ERROR HTTP 599 Connection timed out after milliseconds，而暂时找不到解决办法。

给Phantomjs添加额外参数

之前折腾过：

【未解决】pyspider中如何给phantomjs传递额外参数 – 在路上

基本上没有实现想要的效果。但是可供参考。

phantomjs中的proxy是什么意思

对于pyspider来说，phantomjs-proxy参数指的是：

你另外所运行的phantomjs的实例 = host:port

比如：

在一个终端中运行：

pyspider phantomjs --port 23450 --auto-restart true

然后去另外一个终端中运行pyspider：

pyspider -c config.json all

其中config.json包含了：

"phantomjs-proxy": "127.0.0.1:23450"

就可以使得此处的pyspider在启动时不另外启动phantomjs了。

而是去在需要用到phantomjs时，去连接本地电脑127.0.0.1的23450端口中的phantomjs去处理，去加载页面了。

而对于phantomjs本身来说：

proxy，指的是代理，比如翻墙的代理，等等。

具体相关设置，可以参考：

Command Line Interface | PhantomJS

中的：

–proxy=address:port specifies the proxy server to use (e.g. –proxy=192.168.1.42:8080)

–proxy-type=[http|socks5|none] specifies the type of the proxy server (default is http).

–proxy-auth specifies the authentication information for the proxy, e.g. –proxy-auth=username:password)

详见：【已解决】pyspider中phantomjs中的proxy是什么意思 – 在路上

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2020-07-30 14:00:04

PySpider经验与心得

PySpider经验与心得

折腾了一些PySpider项目，有些经验和心得，整理如下：

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2020-07-30 14:00:04

PySpider的心得

PySpider的心得

对于加载更多内容，除了想办法找js或api，也可以换个其他的思路

问题：想要获取单个页面的更多的内容，一般页面都是向下滚动，加载更多。内部往往是js实现，调用额外的api获取更多数据，加载更多数据。

思路：所以一般往往会去研究和抓包，搞清楚调用的api。但是其实有思路多去看看网页中与之相关的其他内容，往往可以通过其他途径，比如另外有个单独的页面，可以获取我所需要的所有的车型车系的数据。就可以避免非要去研究和抓包api了。

详见：【已解决】pyspider中如何加载汽车之家页面中的更多内容

调试界面中的enable css selector helper

点击web后可以看到html页面内容

再点击enable css selector helper后

之后点击某个页面元素，则可以直接显示出对应的css的selector

不过话说我个人调试页面期间，很少用到。

都是直接去Chrome浏览器中调试页面，查看html源码，寻找合适的css selector。

发送POST请求且传递格式为application/x-www-form-urlencoded的form data参数

代码：

    @config(age=10 * 24 * 60 * 60)
    def index_page(self, response):
        # &lt;ul class="list-user list-user-1" id="list-user-1"&gt;
        for each in response.doc('ul[id^="list-user"] li  a[href^="http"]').items():
            self.crawl(each.attr.href, callback=self.detail_page)

        maxPageNum = 10
        for curPageIdx in range(maxPageNum):
            curPageNum = curPageIdx + 1
            print("curPageNum=%s" % curPageNum)
            getShowsUrl = "http://xxx/index.php?m=home&amp;c=match_new&amp;a=get_shows"
            headerDict = {
                "Content-Type": "application/x-www-form-urlencoded"
            }
            dataDict = {
                "counter": curPageNum,
                "order": 1,
                "match_type": 2,
                "match_name": "",
                "act_id": 3
            }
            self.crawl(
                getShowsUrl,
                method="POST",
                headers=headerDict,
                data=dataDict,
                cookies=response.cookies,
                callback=self.parseGetShowsCallback
            )

    def parseGetShowsCallback(self, response):
        print("parseGetShowsCallback: self=%s, response=%s"%(self, response))
        respJson = response.json
        print("respJson=%s" % (respJson))

实现了：

发送POST

传递header

"Content-Type": "application/x-www-form-urlencoded"

传递data

一个dict，包含对应的key和value

顺带传递了cookie

cookies=response.cookies

获得返回的JSON

callback中用response.json

无法继续爬取时，注意是否是重复url导致的

当发现没有继续爬取后续数据时，记得想想是不是重复url导致的。

比如此处的：

POST /selfReadingBookQuery2
{ "offset": 0, "limit":10}

和：

POST /selfReadingBookQuery2
{ "offset": 10, "limit":10}

虽然（json参数）变化了，但是url没变

-> 导致不（重复）爬取

解决办法：让每次的url不同

实现方式：比如给url后面加上#hash值

举例说明

timestampStr = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
curUrlWithHash = curUrl + "#" + timestampStr

self.crawl(curUrlWithHash,
...

的：

/selfReadingBookQuery2#20190409_162018_413205

/selfReadingBookQuery2#20190409_162117_711811

即可实现，每次请求url都不同，就可以继续爬取了。

如果还是不行，或者说，为了更加保险，可以再去加上itag，比如：

# add hash value for url to force re-crawl when POST url not changed
timestampStr = datetime.now().strftime("%Y%m%d_%H%M%S_%f")
curUrlWithHash = curUrl + "#" + timestampStr

fakeItagForceRecrawl = "%s_%s_%s" % (timestampStr, offset, limit)

self.crawl(curUrlWithHash,
    itag=fakeItagForceRecrawl, # To force re-crawl for next page
    method="POST",

当连续多个请求都出现599超时连接后，且尝试retry也都全部失败后，会自动暂停

之前遇到过多次，类似这种：

[I 180922 11:25:42 scheduler:959] task retry 0/3 ChildQupeiyinApp:c3e0a65a42199256898652ce1e737321 https://childapi.qupeiyin.com/show/detail?show_id=129410670
[E 180922 11:25:42 tornado_fetcher:212] [599] ChildQupeiyinApp:80040ee81a217bc05a877ff41ee74d05 https://childapi.qupeiyin.com/show/detail?show_id=130095443, HTTP 599: Connection timed out after 20000 milliseconds 20.00s
[E 180922 11:25:42 processor:202] process ChildQupeiyinApp:80040ee81a217bc05a877ff41ee74d05 https://childapi.qupeiyin.com/show/detail?show_id=130095443 -> [599] len:0 -> result:None fol:0 msg:0 err:Exception('HTTP 599: Connection timed out after 20000 milliseconds',)
[I 180922 11:25:42 scheduler:959] task retry 0/3 ChildQupeiyinApp:80040ee81a217bc05a877ff41ee74d05 https://childapi.qupeiyin.com/show/detail?show_id=130095443
[E 180922 11:25:42 tornado_fetcher:212] [599] ChildQupeiyinApp:3783982a707a6c82b8c30d619a6933d7 https://childapi.qupeiyin.com/show/detail?show_id=130096232, HTTP 599: Connection timed out after 20000 milliseconds 20.00s
[E 180922 11:25:42 processor:202] process ChildQupeiyinApp:3783982a707a6c82b8c30d619a6933d7 https://childapi.qupeiyin.com/show/detail?show_id=130096232 -> [599] len:0 -> result:None fol:0 msg:0 err:Exception('HTTP 599: Connection timed out after 20000 milliseconds',)
[I 180922 11:25:43 scheduler:959] task retry 0/3 ChildQupeiyinApp:3783982a707a6c82b8c30d619a6933d7 https://childapi.qupeiyin.com/show/detail?show_id=130096232
[I 180922 11:26:08 scheduler:126] project ChildQupeiyinApp updated, status:STOP, paused:True, 1667087 tasks
^C[I 180922 11:26:13 scheduler:663] scheduler exiting...
[I 180922 11:26:13 tornado_fetcher:671] fetcher exiting...
[I 180922 11:26:13 processor:229] processor exiting...
[I 180922 11:26:13 result_worker:66] result_worker exiting...

上述 status:STOP, paused:True 就是表示暂停了。

对应着界面上status自动变成PAUSED

-> 估计是内部逻辑发现多次是599的错误，就自动暂停重试了。避免了后续无效的请求

-> 还是很智能的，因为此处实际上是网络断了，导致无法请求的。

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-04-14 23:09:48

删除项目

删除项目

如何清除之前的或正在运行的任务

对于一个写好的爬虫，且已经点击Run运行，或者运行了一段时间后，主动停止了。

接着想要去删除之前下载的数据，则：

官网的解释是：

设置group为delete，以及status为STOP后，过了(默认)24小时后，会自动删除该项目所有信息。

但是往往没法满足我们需求：

我不想要等待，只想现在就去：删除掉所有的信息，包括之前已经爬取的数据，之前的调度的任务等等数据。

经过一番研究后，发现了解决方案：

先去停止项目

WebUI中设置status为STOP

终端中用Control+C强制停止pyspider的运行

再去删除文件：result.db和task.db

如果还有任务相关的 scheduler.all，scheduler.1d，scheduler.1h，则一并删除

不要轻易在没备份代码情况下删除project.db

注意不要删除，保存了项目（配置和）代码的：project.db，否则代码就没了。（我最开始就这么干过，😂）

之后去重新运行pyspider，再去刷新WebUI界面：

http://0.0.0.0:5000/

即可看到干净的项目，没有了之前的任务和数据了。

指定多久之后删除项目，即指定项目删除等待时间

PySpider中的项目，想要删除：

默认逻辑是，status设置为STOP（如果有group，那么group的status也要设置为delete），再等24小时后，才会自动删除

但是往往我们不想要等待那么久

想要指定删除的时间，则有2种方式去设置参数。

举例说明，比如想要30秒后删除，则可以：

文件：config.json：设置scheduler的delete_time参数

  "scheduler": {
      "delete_time": 30
  }

命令行传入：scheduler.DELETE_TIME

  pyspider -c config.json scheduler --delete-time=30

然后WebUI中设置

status是STOP

group设置为delete

然后过了30秒后，去刷新，该项目就被删除了，看不到了：

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-04-14 20:49:26

PySpider常见的坑

PySpider常见的坑

关于折腾PySpider期间，遇到很多或大或小的坑，常见和具体细节相关的坑，已记录到对应部分中了。

此处再继续整理出，其他的一些常见的坑。

HTTP 599 Operation timed out after milliseconds with out of bytes received

类似现象：

after 120001 milliseconds with 0 bytes receive
after 120000 milliseconds with 1723300 out of 2343850 bytes received 120.00s

解释：

-> 意思是：超时了（超过设置的最大超时时间了），但是只下载了总共数据的其中一部分

-> 重点是后半句，意思是可以下载到数据的，只是直到超时都还没下载完全

-> 这种情况的最大可能原因就是：网速太慢

-》所以

解决办法

根本办法：换个更快的网络

比如，我公司是1MB/s的网络，家里是10MB/s的网络，换到家里下载，就不会出现这个问题了

临时的规避的缓解的办法：增大延迟timeout （ +增大其他容错参数connect_timeout, retries）

给单个self.crawl增大参数

  self.crawl(urlToDownload,
      callback=self.downloadFileCallback,
      connect_timeout=100,
      timeout=600,
      retries=15,
      save=fileInfo)

或：增大全局参数

  class Handler(BaseHandler):
      crawl_config = {
          "connect_timeout": 100,
          "timeout": 600,
          "retries": 15,
      }

参数含义解释详见官网：

self.crawl - pyspider

css的选择器不工作

背景：网页中的源码本来是：

<a href="//car.autohome.com.cn/pic/series/3170.html#pvareaid=103448">

或者类似的：

href="/pic/series-t/3170.html"

然后去写css选择器：

a[href^="//car.autohome.com.cn/pic/series/"]

但是却无法匹配

原因：PySpider内部的css选择器用的是PyQuery，其默认把href的路径，加上了对应的host，所以此时获取到的html实际上变成了：

<a href="https://car.autohome.com.cn/pic/series/3170.html#pvareaid=103448">

详见：

response.doc

Reponse.doc() 返回的就是一个PyQuery的对象 Links have made as absolute by default

猜测：估计是为了方便小白用户，所以默认加上了host，但是坑了其他人啊。

解决办法：此处被逼的css选择器写法只能改为：

a[href*="pic/series/"]

或类似的代码：

fnRightPicSeries = response.doc('.search-pic-tbar .fn-right a[href*="/pic/series"]')
fullPicSeriesUrl = fnRightPicSeries.attr.href

已经得到的是，加了host/domain的绝对路径了：

fullPicSeriesUrl= https://car.autohome.com.cn/pic/series-t/3170.html

详见：【已解决】pyspider中的css选择器不工作 – 在路上

Error Could not create web server listening on port 25555

原因：对应的25555端口被占用了

根本原因：之前的PySpider没有正常的彻底的被关闭，所以残留了。

解决办法：彻底kill干掉之前的PySpider的进程即可。

举例：

普通Linux类系统，用：

找到占了25555端口的进程的id：ps aux | grep 25555

再去杀掉进程：kill process_id -9

即可。

如果是Mac中，则用lsof

➜  AutocarData lsof -i:25555
COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
phantomjs 46971 crifan   12u  IPv4 0xe4d24cdcaf5e481f      0t0  TCP *:25555 (LISTEN)
➜  AutocarData kill  46971

PAUSED后无法立刻继续运行

当PySpider在爬取期间发现太多的url都是retry重试，则会变成PAUSED

猜测：其内部有比较智能的判断，推测是断网或者网络异常了，所以暂停下载

通过直接把PAUSED改为STOP

或改为了RUNNING

但是刷新一下页面，就还是显示PAUSED

-> 不能立刻开始继续下载

-> 往往要等很长时间之后才能继续下载

而如果自己想要立刻继续下载，经过研究，可以：

先改为STOP

停止PySpider

Control+C停止

重新运行PySpider

再改为RUNNING，点击Run

即可立刻继续运行了

post时data传递dict有时候不行

比如

crifan/PySpiderChinaProvinceCity

getCityUrl = "http://www.dianping.com/ajax/citylist/getDomesticCityByProvince"
for eachProvince in provinceList:
    print("eachProvince=%s" % eachProvince)
    provinceIdInt = eachProvince["provinceId"]
    paramDict = {
        "provinceId": provinceIdInt,
    }
    paramDictStr = json.dumps(paramDict)
...
    self.crawl(
        # getCityUrl,
        urlWithHash,
        # itag=fakeItag,
        method="POST",
        # data=paramDict,
        data=paramDictStr,
        callback=self.getCityCallback,
        save=provinceDict,
    )

data要传递json变量paramDict去json.dumps后的字符串paramDictStr才可以：

如果换成dict：

getCityUrl = "http://www.dianping.com/ajax/citylist/getDomesticCityByProvince"
for eachProvince in provinceList:
    print("eachProvince=%s" % eachProvince)
    provinceIdInt = eachProvince["provinceId"]
    paramDict = {
        "provinceId": provinceIdInt,
    }
    paramDictStr = json.dumps(paramDict)
...
    self.crawl(
        # getCityUrl,
        urlWithHash,
        # itag=fakeItag,
        method="POST",
        data=paramDict,
        # data=paramDictStr,
        callback=self.getCityCallback,
        save=provinceDict,
    )

且已经指定了全局配置crawl_config中的headers的Content-Type是application/json

-> 以为PySpider的self.crawl会自动把dict类型的data去json dump成字符串，结果却是：

只保留了key和value：

"data": "provinceId=10",

导致报错：

    requests.exceptions.HTTPError: HTTP 400: Bad Request

之前还遇到过一个类似的例子：

SelfReadingUrl = "http://www.xxxxxxxxx.cn:83/Reading.svc/selfReadingBookQuery2"

    def on_start(self):
        jValueTemplateSelfReading = "{\"userId\":\"%s\",\"fieldName\":\"\",\"fieldValue\":\"全部类别\",\"grades\":\"\",\"levels\":\"\",\"supportingResources\":\"有音频\",\"offset\":%d,\"limit\":%d}"

        paramDictSelfReading = {
            "curUrl": SelfReadingUrl,
            "offset": 0,
            "limit": DefaultPageSize,
            "jValueTemplate": jValueTemplateSelfReading
        }
        self.getBookQuery2(paramDictSelfReading)

    def getBookQuery2(self, curParamDict):
...
        jValueStr = jValueTemplate % (gUserId, offset, limit)
        jcJsonDict = {
            "J": jValueStr,
            "C": 0
        }
        jcJsonDictStr = json.dumps(jcJsonDict)
        ......
        self.crawl(curUrlWithHash,
            itag=fakeItagForceRecrawl, # To force re-crawl for next page
            method="POST",
            # data=jcJsonDict,
            data= jcJsonDictStr,
            # callback=curCallback,
            callback=self.getBookQuery2Callback,
            headers=curHeaders,
            save=curParamDict
        )

其中也是：

给data参数用了json去dump后的字符串变量：jcJsonDictStr

而不是json的dict变量：jcJsonDict

才最终正确获取到数据的。

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-09-08 11:13:04

PySpider案例

PySpider案例

下面把一些之前写过的PySpider的爬虫分享出来，供参考。

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-04-14 20:12:48

汽车之家的品牌等数据

汽车之家的品牌等数据

需求

想要抓取

汽车之家

https://www.autohome.com.cn/car/

的所有的品牌的车系和车型数据

核心过程

虽然前面汽车之家的主页是

https://www.autohome.com.cn/car/

但是对于：方便爬虫去找所有品牌的入口

经过研究是：

self.crawl("https://www.autohome.com.cn/grade/carhtml/%s.html" % eachLetter, callback=self.gradCarHtmlPage)

其中：%s支持从a到z

比如对于a是：

https://www.autohome.com.cn/grade/carhtml/a.html

其中，我们要去抓取的是某个品牌的某个系列，比如：奥迪A4L的图库：

https://car.autohome.com.cn/pic/series/692.html#pvareaid=103448

然后再去picSeriesPage中：

        fnRightPicSeries = response.doc('.search-pic-tbar .fn-right a[href*="/pic/series"]')

找到：search-pic-tbar下面的href*="/pic/series"：

然后，单独再去找到车型详情页spec页面，比如

modelSpecUrlTemplate = "https://www.autohome.com.cn/spec/%s/#pvareaid=2042128"

对应的地址，比如：

https://www.autohome.com.cn/spec/32708/#pvareaid=2042128

对应页面：

其顶部，有很多相关的字段，比如：品牌（=车厂=厂商）、子品牌、厂商指导价等等

以及页面下面有，全系（不同配置的）车型的数据：

然后用carModelSpecPage，再去解析不同字段

最终即可获取到，所有从车品牌的所有车系的所有车型的基本信息了。

详见：

【已解决】写Python爬虫爬取汽车之家品牌车系车型数据 – 在路上

PySpider

核心文件：autohomeBrandData.py

#!/usr/bin/env python
# -*- encoding: utf-8 -*-
# Created on 2018-04-27 21:53:02
# Project: autohomeBrandData

from pyspider.libs.base_handler import *
import string
import re

class Handler(BaseHandler):
    crawl_config = {
    }

    # @every(minutes=24 * 60)
    def on_start(self):
        for eachLetter in list(string.ascii_lowercase):
            self.crawl("https://www.autohome.com.cn/grade/carhtml/%s.html" % eachLetter, callback=self.gradCarHtmlPage)

    @catch_status_code_error
    def gradCarHtmlPage(self, response):
        print("gradCarHtmlPage: response=", response)
        picSeriesItemList = response.doc('.rank-list-ul li div a[href*="/pic/series"]').items()
        print("picSeriesItemList=", picSeriesItemList)
        # print("len(picSeriesItemList)=%s"%(len(picSeriesItemList)))
        for each in picSeriesItemList:
            self.crawl(each.attr.href, callback=self.picSeriesPage)

    @config(priority=2)
    def picSeriesPage(self, response):
        # <a href="/pic/series-t/66.html">查看停产车型&nbsp;&gt;</a>
        # <a class="ckmore" href="/pic/series/588.html">查看在售车型&nbsp;&gt;</a>
        # <span class="fn-right">&nbsp;</span>
        fnRightPicSeries = response.doc('.search-pic-tbar .fn-right a[href*="/pic/series"]')
        print("fnRightPicSeries=", fnRightPicSeries)
        if fnRightPicSeries:
            # hrefValue = fnRightPicSeries.attr.href
            # print("hrefValue=", hrefValue)
            # fullPicSeriesUrl = "https://car.autohome.com.cn" + hrefValue
            fullPicSeriesUrl = fnRightPicSeries.attr.href
            print("fullPicSeriesUrl=", fullPicSeriesUrl)
            self.crawl(fullPicSeriesUrl, callback=self.picSeriesPage)

        # contine parse brand data
        aDictList = []
        # for eachA in response.doc('.breadnav a[href^="/"]').items():
        for eachA in response.doc('.breadnav a[href*="/pic/"]').items():
            eachADict = {
                "text": eachA.text(),
                "href": eachA.attr.href
            }
            print("eachADict=", eachADict)
            aDictList.append(eachADict)

        print("aDictList=", aDictList)

        mainBrandDict = aDictList[-3]
        subBrandDict = aDictList[-2]
        brandSerieDict = aDictList[-1]
        print("mainBrandDict=%s, subBrandDict=%s, brandSerieDict=%s" % (mainBrandDict, subBrandDict, brandSerieDict))

        dtTextList = []
        for eachDt in response.doc("dl.search-pic-cardl dt").items():
            dtTextList.append(eachDt.text())

        print("dtTextList=", dtTextList)

        groupCount = len(dtTextList)
        print("groupCount=", groupCount)

        for eachDt in response.doc("dl.search-pic-cardl dt").items():
            dtTextList.append(eachDt.text())

        ddUlEltList = []
        for eachDdUlElt in response.doc("dl.search-pic-cardl dd ul").items():
            ddUlEltList.append(eachDdUlElt)

        print("ddUlEltList=", ddUlEltList)

        modelDetailDictList = []

        for curIdx in range(groupCount):
            curGroupTitle = dtTextList[curIdx]
            print("------[%d] %s" % (curIdx, curGroupTitle))

            for eachLiAElt in ddUlEltList[curIdx].items("li a"):
                # 1. model name
                # curModelName = eachLiAElt.text()
                curModelName = eachLiAElt.contents()[0]
                curModelName = curModelName.strip()
                print("curModelName=", curModelName)
                curFullModelName = curGroupTitle + " " + curModelName
                print("curFullModelName=", curFullModelName)

                # 2. model id + carSeriesId + spec url
                curModelId = ""
                curSeriesId = ""
                curModelSpecUrl = ""
                modelSpecUrlTemplate = "https://www.autohome.com.cn/spec/%s/#pvareaid=2042128"
                curModelPicUrl = eachLiAElt.attr.href
                print("curModelPicUrl=", curModelPicUrl)
                # https://car.autohome.com.cn/pic/series-s32708/3457.html#pvareaid=2042220
                foundModelSeriesId = re.search("pic/series-s(?P<curModelId>\d+)/(?P<curSeriesId>\d+)\.html",
                                               curModelPicUrl)
                print("foundModelSeriesId=", foundModelSeriesId)
                if foundModelSeriesId:
                    curModelId = foundModelSeriesId.group("curModelId")
                    curSeriesId = foundModelSeriesId.group("curSeriesId")
                    print("curModelId=%s, curSeriesId=%s", curModelId, curSeriesId)
                    curModelSpecUrl = (modelSpecUrlTemplate) % (curModelId)
                    print("curModelSpecUrl=", curModelSpecUrl)

                # 3. model status
                modelStatus = "在售"
                foundStopSale = eachLiAElt.find('i[class*="icon-stopsale"]')
                if foundStopSale:
                    modelStatus = "停售"
                else:
                    foundWseason = eachLiAElt.find('i[class*="icon-wseason"]')
                    if foundWseason:
                        modelStatus = "未上市"

                modelDetailDictList.append({
                    "url": curModelSpecUrl,
                    # "车系ID": curSeriesId,
                    # "车型ID": curModelId,
                    # "车型": curFullModelName,
                    # "状态": modelStatus
                    "brandSerieId": curSeriesId,
                    "modelId": curModelId,
                    "model": curFullModelName,
                    "modelStatus": modelStatus
                })

        print("modelDetailDictList=", modelDetailDictList)

        allSerieDictList = []
        for curIdx, eachModelDetailDict in enumerate(modelDetailDictList):
            """
            defined in mysql

                CREATE TABLE `tbl_autohome_car_info` (
                  `id` int(11) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增，主键',
                  `cityDealerPrice` int(11) unsigned NOT NULL DEFAULT '0' COMMENT '经销商参考价',
                  `msrpPrice` int(11) unsigned NOT NULL DEFAULT '0' COMMENT '厂商指导价',
                  `mainBrand` char(20) NOT NULL DEFAULT '' COMMENT '品牌',
                  `subBrand` varchar(20) NOT NULL DEFAULT '' COMMENT '子品牌',
                  `brandSerie` varchar(20) NOT NULL DEFAULT '' COMMENT '车系',
                  `brandSerieId` varchar(15) NOT NULL DEFAULT '' COMMENT '车系ID',
                  `model` varchar(50) NOT NULL DEFAULT '' COMMENT '车型',
                  `modelId` varchar(15) NOT NULL DEFAULT '' COMMENT '车型ID',
                  `modelStatus` char(5) NOT NULL DEFAULT '' COMMENT '车型状态',
                  `url` varchar(200) NOT NULL DEFAULT '' COMMENT '车型url',
                  PRIMARY KEY (`id`)
                ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
            """
            curSerieDict = {
                "url": eachModelDetailDict["url"],
                # "品牌": mainBrandDict["text"],
                # "子品牌": subBrandDict["text"],
                # "车系": brandSerieDict["text"],
                # "车系ID": eachModelDetailDict["车系ID"],
                # "车型": eachModelDetailDict["车型"],
                # "车型ID": eachModelDetailDict["车型ID"],
                # "状态": eachModelDetailDict["状态"]
                "mainBrand": mainBrandDict["text"],
                "subBrand": subBrandDict["text"],
                "brandSerie": brandSerieDict["text"],
                "brandSerieId": eachModelDetailDict["brandSerieId"],
                "model": eachModelDetailDict["model"],
                "modelId": eachModelDetailDict["modelId"],
                "modelStatus": eachModelDetailDict["modelStatus"]
            }
            allSerieDictList.append(curSerieDict)
            # print("before send_message: [%d] curSerieDict=%s" % (curIdx, curSerieDict))
            # self.send_message(self.project_name, curSerieDict, url=eachModelDetailDict["url"])
            print("[%d] curSerieDict=%s" % (curIdx, curSerieDict))
            self.crawl(eachModelDetailDict["url"],
                       callback=self.carModelSpecPage,
                       fetch_type='js',
                       retries=5,
                       connect_timeout=50,
                       timeout=300,
                       save=curSerieDict)

        # print("allSerieDictList=", allSerieDictList)
        # return allSerieDictList

    # def on_message(self, project, msg):
    #    print("on_message: msg=", msg)
    #    return msg

    @catch_status_code_error
    def carModelSpecPage(self, response):
        print("carModelSpecPage: response=", response)
        # https://www.autohome.com.cn/spec/32708/#pvareaid=2042128
        curSerieDict = response.save
        print("curSerieDict", curSerieDict)

        cityDealerPriceInt = 0
        cityDealerPriceElt = response.doc('.cardetail-infor-price #cityDealerPrice span span[class*="price"]')
        print("cityDealerPriceElt=%s" % cityDealerPriceElt)
        if cityDealerPriceElt:
            cityDealerPriceFloatStr = cityDealerPriceElt.text()
            print("cityDealerPriceFloatStr=", cityDealerPriceFloatStr)
            cityDealerPriceFloat = float(cityDealerPriceFloatStr)
            print("cityDealerPriceFloat=", cityDealerPriceFloat)
            cityDealerPriceInt = int(cityDealerPriceFloat * 10000)
            print("cityDealerPriceInt=", cityDealerPriceInt)

        msrpPriceInt = 0
        # body > div.content > div.row > div.column.grid-16 > div.cardetail.fn-clear > div.cardetail-infor > div.cardetail-infor-price.fn-clear > ul > li.li-price.fn-clear > span
        # 厂商指导价=厂商建议零售价格=MSRP=Manufacturer's suggested retail price
        msrpPriceElt = response.doc('.cardetail-infor-price li[class*="li-price"] span[data-price]')
        print("msrpPriceElt=", msrpPriceElt)
        if msrpPriceElt:
            msrpPriceStr = msrpPriceElt.attr("data-price")
            print("msrpPriceStr=", msrpPriceStr)
            foundMsrpPrice = re.search("(?P<msrpPrice>[\d\.]+)万元", msrpPriceStr)
            print("foundMsrpPrice=", foundMsrpPrice)
            if foundMsrpPrice:
                msrpPrice = foundMsrpPrice.group("msrpPrice")
                print("msrpPrice=", msrpPrice)
                msrpPriceFloat = float(msrpPrice)
                print("msrpPriceFloat=", msrpPriceFloat)
                msrpPriceInt = int(msrpPriceFloat * 10000)
                print("msrpPriceInt=", msrpPriceInt)

        # curSerieDict["经销商参考价"] = cityDealerPriceInt
        # curSerieDict["厂商指导价"] = msrpPriceInt
        curSerieDict["cityDealerPrice"] = cityDealerPriceInt
        curSerieDict["msrpPrice"] = msrpPriceInt

        return curSerieDict

worker文件：AutohomeResultWorker.py

#!/usr/bin/env python
# -*- encoding: utf-8 -*-
# Project: autohomeBrandData
# Function: implement custom result worker for autohome car data
# Author: Crifan Li
# Date: 20180512
# Note:
#   If you want to modify to your mysql and table, you need:
#   (1) change change MysqlDb config to your mysql config
#   (2) change CurrentTableName to your table name
#   (3) change CreateTableSqlTemplate to your sql to create new mysql table fields
#   (4) before use this ResultWorker, run py file to execute testMysqlDb, to init db and create table
#   (5) if your table field contain more type, edit insert to add more type for "TODO: add more type formatting if necessary"

import pymysql
import pymysql.cursors
from pyspider.result import ResultWorker

CurrentTableName = "tbl_autohome_car_info"
CreateTableSqlTemplate = """CREATE TABLE IF NOT EXISTS `%s` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增，主键',
  `cityDealerPrice` int(11) unsigned NOT NULL DEFAULT '0' COMMENT '经销商参考价',
  `msrpPrice` int(11) unsigned NOT NULL DEFAULT '0' COMMENT '厂商指导价',
  `mainBrand` char(20) NOT NULL DEFAULT '' COMMENT '品牌',
  `subBrand` varchar(20) NOT NULL DEFAULT '' COMMENT '子品牌',
  `brandSerie` varchar(20) NOT NULL DEFAULT '' COMMENT '车系',
  `brandSerieId` varchar(15) NOT NULL DEFAULT '' COMMENT '车系ID',
  `model` varchar(50) NOT NULL DEFAULT '' COMMENT '车型',
  `modelId` varchar(15) NOT NULL DEFAULT '' COMMENT '车型ID',
  `modelStatus` char(5) NOT NULL DEFAULT '' COMMENT '车型状态',
  `url` varchar(200) NOT NULL DEFAULT '' COMMENT '车型url',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;"""

class AutohomeResultWorker(ResultWorker):

    def __init__(self, resultdb, inqueue):
        """init mysql db"""
        print("AutohomeResultWorker init: resultdb=%s, inqueue=%s" % (resultdb, inqueue))
        ResultWorker.__init__(self, resultdb, inqueue)

        self.mysqlDb = MysqlDb()
        print("self.mysqlDb=%s" % self.mysqlDb)

    def on_result(self, task, result):
        """override pyspider on_result to save data into mysql"""
        # assert task['taskid']
        # assert task['project']
        # assert task['url']
        # assert result
        print("AutohomeResultWorker on_result: task=%s, result=%s" % (task, result))
        insertOk = self.mysqlDb.insert(result)
        print("insertOk=%s" % insertOk)

class MysqlDb:
    config = {
        'host': '127.0.0.1',
        'port': 3306,
        'user': 'root',
        'password': 'crifan_mysql',
        'database': 'AutohomeResultdb',
        'charset': "utf8"
    }

    defaultTableName = CurrentTableName
    connection = None

    def __init__(self):
        """init mysql"""
        # 1. connect db first
        if self.connection is None:
            isConnected = self.connect()
            print("Connect mysql return %s" % isConnected)

        # 2. create table for db
        createTableOk = self.createTable(self.defaultTableName)
        print("Create table %s return %s" %(self.defaultTableName, createTableOk))

    def connect(self):
        try:
            self.connection = pymysql.connect(**self.config, cursorclass=pymysql.cursors.DictCursor)
            print("connect mysql ok, self.connection=", self.connection)
            return True
        except pymysql.Error as err:
            print("Connect mysql with config=", self.config, " error=", err)
            return False

    def quoteIdentifier(self, identifier):
        """
            for mysql, it better to quote identifier xxx using backticks to `xxx`
            in case, identifier:
                contain special char, such as space
                or same with system reserved words, like select
        """
        quotedIdentifier = "`%s`" % identifier
        # print("quotedIdentifier=", quotedIdentifier)
        return quotedIdentifier

    def executeSql(self, sqlStr, actionDescription=""):
        print("executeSql: sqlStr=%s, actionDescription=%s" % (sqlStr, actionDescription))

        if self.connection is None:
            print("Please connect mysql first before %s" % actionDescription)
            return False

        cursor = self.connection.cursor()
        print("cursor=", cursor)

        try:
            cursor.execute(sqlStr)
            self.connection.commit()
            print("+++ Ok to execute sql %s for %s" % (sqlStr, actionDescription))
            return True
        except pymysql.Error as err:
            print("!!! %s when execute sql %s for %s" % (err, sqlStr, actionDescription))
            return False

    def createTable(self, newTablename):
        print("createTable: newTablename=", newTablename)

        createTableSql = CreateTableSqlTemplate % (newTablename)
        print("createTableSql=", createTableSql)

        return self.executeSql(sqlStr=createTableSql, actionDescription=("Create table %s" % newTablename))

    def dropTable(self, existedTablename):
        print("dropTable: existedTablename=", existedTablename)

        dropTableSql = "DROP TABLE IF EXISTS %s" % (existedTablename)
        print("dropTableSql=", dropTableSql)

        return self.executeSql(sqlStr=dropTableSql, actionDescription=("Drop table %s" % existedTablename))

    # def insert(self, **valueDict):
    def insert(self, valueDict, tablename=defaultTableName):
        """
            inset dict value into mysql table
            makesure the value is dict, and its keys is the key in the table
        """
        print("insert: valueDict=%s, tablename=%s" % (valueDict, tablename))

        dictKeyList = valueDict.keys()
        dictValueList = valueDict.values()
        print("dictKeyList=", dictKeyList, "dictValueList=", dictValueList)

        keyListSql = ", ".join(self.quoteIdentifier(eachKey) for eachKey in dictKeyList)
        print("keyListSql=", keyListSql)
        # valueListSql = ", ".join(eachValue for eachValue in dictValueList)
        valueListSql = ""
        formattedDictValueList = []
        for eachValue in dictValueList:
            # print("eachValue=", eachValue)
            eachValueInSql = ""
            valueType = type(eachValue)
            # print("valueType=", valueType)
            if valueType is str:
                eachValueInSql = '"%s"' % eachValue
            elif valueType is int:
                eachValueInSql = '%d' % eachValue
            # TODO: add more type formatting if necessary
            print("eachValueInSql=", eachValueInSql)
            formattedDictValueList.append(eachValueInSql)

        valueListSql = ", ".join(eachValue for eachValue in formattedDictValueList)
        print("valueListSql=", valueListSql)

        insertSql = """INSERT INTO %s (%s) VALUES (%s)""" % (tablename, keyListSql, valueListSql)
        print("insertSql=", insertSql)
        # INSERT INTO tbl_car_info_test (`url`, `mainBrand`, `subBrand`, `brandSerie`, `brandSerieId`, `model`, `modelId`, `modelStatus`, `cityDealerPrice`, `msrpPrice`) VALUES ("https://www.autohome.com.cn/spec/5872/#pvareaid=2042128", "宝马", "华晨宝马", "宝马3系", "66", "2010款 320i 豪华型", "5872", "停售", 325000, 375000)

        return self.executeSql(sqlStr=insertSql, actionDescription=("Insert value to table %s" % tablename))

    def delete(self, modelId, tablename=defaultTableName):
        """
            delete item from car model id for existing table of autohome car info
        """
        print("delete: modelId=%s, tablename=%s" % (modelId, tablename))

        deleteSql = """DELETE FROM %s WHERE modelId = %s""" % (tablename, modelId)
        print("deleteSql=", deleteSql)

        return self.executeSql(sqlStr=deleteSql, actionDescription=("Delete value from table %s by model id %s" % (tablename, modelId)))

def testMysqlDb():
    """test mysql"""

    testDropTable = True
    testCreateTable = True
    testInsertValue = True
    testDeleteValue = True

    # 1.test connect mysql
    mysqlObj = MysqlDb()
    print("mysqlObj=", mysqlObj)

    # testTablename = "autohome_car_info"
    # testTablename = "tbl_car_info_test"
    testTablename = CurrentTableName
    print("testTablename=", testTablename)

    if testDropTable:
        # 2. test drop table
        dropTableOk = mysqlObj.dropTable(testTablename)
        print("dropTable", testTablename, "return", dropTableOk)

    if testCreateTable:
        # 3. test create table
        createTableOk = mysqlObj.createTable(testTablename)
        print("createTable", testTablename, "return", createTableOk)

    if testInsertValue:
        # 4. test insert value dict
        valueDict = {
            "url": "https://www.autohome.com.cn/spec/5872/#pvareaid=2042128", #车型url
            "mainBrand": "宝马", #品牌
            "subBrand": "华晨宝马", #子品牌
            "brandSerie": "宝马3系", #车系
            "brandSerieId": "66", #车系ID
            "model": "2010款 320i 豪华型", #车型
            "modelId": "5872", #车型ID
            "modelStatus": "停售", #车型状态
            "cityDealerPrice": 325000, #经销商参考价
            "msrpPrice": 375000 # 厂商指导价
        }
        print("valueDict=", valueDict)
        insertOk = mysqlObj.insert(valueDict=valueDict, tablename=testTablename)
        print("insertOk=", insertOk)

    if testDeleteValue:
        toDeleteModelId = "5872"
        deleteOk = mysqlObj.delete(modelId=toDeleteModelId, tablename=testTablename)
        print("deleteOk=", deleteOk)

def testAutohomeResultWorker():
    """just test for create mysql db is ok or not"""
    autohomeResultWorker = AutohomeResultWorker(None, None)
    print("autohomeResultWorker=%s" % autohomeResultWorker)

if __name__ == '__main__':
    testMysqlDb()
    # testAutohomeResultWorker()

配置文件：config.json

{
  "resultdb":   "mysql+resultdb://root:crifan_mysql@127.0.0.1:3306/AutohomeResultdb",
  "result_worker":{
      "result_cls": "AutohomeResultWorker.AutohomeResultWorker"
   },
  "phantomjs-proxy": "127.0.0.1:23450",
  "phantomjs" : {
    "port": 23450,
    "auto-restart": true,
    "load-images": false,
    "debug": true
  }
}

相关内容

相关页面

汽车之家的入口页面：

对应的PySpider调试页面

调试PySpider

期间会涉及到PySpider的调试

比如：

用了debug模式去调试：

遇到一个出错的：

点击进去可以看到（出错的子任务的）详情信息：

此处明显是：

Document is empty

导出结果

PySpider运行完毕（或运行期间）可以去results页面去导出结果：

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-09-08 13:09:24

汽车之家的车型详细数据

汽车之家的车型详细数据

需求

爬取

汽车之家

https://www.autohome.com.cn/car/

的汽车品牌的车型车型的详细数据

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-09-08 13:55:04

过程

过程

本身汽车之家入口是

https://www.autohome.com.cn/car/

但是经研究，更利于爬虫抓取的是：

从：

https://www.autohome.com.cn/grade/carhtml/a.html

到：

https://www.autohome.com.cn/grade/carhtml/z.html

其中可以找到车系的数据：

对应的值分别是：

carBrandName：奥迪

carBrandLogoUrl：https://car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png

carMerchantName：一汽-大众奥迪

carMerchantUrl：https://car.autohome.com.cn/price/brand-33-9.html#pvareaid=2042363

carSeriesName：奥迪A3

carSeriesUrl：https://www.autohome.com.cn/3170/#levelsource=000000000_0&pvareaid=101594

carSeriesMsrp：19.32-23.46万

carSeriesMsrpUrl：https://www.autohome.com.cn/3170/price.html#pvareaid=101446

然后进去车系主页=车系详情页

【奥迪A3】奥迪奥迪A3报价奥迪A3图片_汽车之家

https://www.autohome.com.cn/3170/#levelsource=000000000_0&pvareaid=101594

可以抓取到：

汽车品牌Logo图片

汽车主品牌=主机厂=（汽车）厂商

车系

厂商指导价

汽车级别

以及页面下方的，全部的车系的数据：

可以抓取到：

carModelGroupName：1.5升 涡轮增压 169马力 国VI

carModelYear：2020款

carModelEmissionStandards：国VI

carModelPower：1.5T

carModelGearBox：7挡双离合

如果历史车型很多，分多个年份，则还要切换年份后再抓取：

之后，还要抓取每个车型的详细配置参数：

【奥迪Q2L e-tron 2019款 Q2L e-tron 纯电智酷型参数配置表】价格单奥迪汽车之家

https://car.autohome.com.cn/config/spec/42875.html#pvareaid=3454541

期间，有些内容的处理，还是很复杂的。

具体细节详见：

【未解决】用Python爬取汽车之家的车型车系详细数据

【已解决】Mac中恢复或重新搭建PySpider开发环境

【已解决】PySpider访问汽车之家报错：requests.exceptions.HTTPError HTTP 403 Forbidden

【已解决】PySpider中获取PyQuery获取到节点的子元素

【已解决】PySpider中PyQuery如何匹配某个元素中多个属性值

【已解决】汽车之家车型车系数据：支持新版车系页面

【已解决】汽车之家车型车系数据：支持旧版车系页面

【已解决】汽车之家车型车系数据：抓取车型的详细参数配置

【未解决】汽车之家车型车系数据：想办法破解js和css加密的文字显示

【已解决】汽车之家车型车系数据：解决部分车型基本参数字段个数不一致问题

【已解决】汽车之家车型车系数据：优化去掉js加速抓取车型参数配置

【规避解决】汽车之家配置页面无法直接获取到文字数据

【已解决】汽车之家车型车系数据：carBrandId是空

【未解决】汽车之家车型车系数据：能源类型是空白的车型

【已解决】汽车之家车型车系数据：车身结构的值包含span标签

【未解决】汽车之家车型车系数据：有些参数字段的值包含span特殊标签

【已解决】汽车之家车型车系数据：抓取车型的详细参数配置

【已解决】车型车系数据缺失如红旗H5等部分车型数据

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-09-08 13:52:05

代码

PySpider代码

核心文件：autohome_20200902.py

直接下载：autohome_20200902.py

贴出如下

#!/usr/bin/env python
# -*- encoding: utf-8 -*-
# Created on 2020-09-02 21:22:43
# Project: autohome_20200902

import string
import re
import copy
import json

from lxml import etree

# from bs4 import BeautifulSoup

from pyspider.libs.base_handler import *

AutohomeHost = "https://www.autohome.com.cn"
CarSpecPrefix = "%s/spec" % AutohomeHost # "https://www.autohome.com.cn/spec/%s/"

class Handler(BaseHandler):
    UserAgent_Mac_Chrome = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
    crawl_config = {
        "headers": {
            "User-Agent": UserAgent_Mac_Chrome,
        }
    }

    def genSpecUrl(self, specId):
        # return "%s/%s" % (CarSpecPrefix, specId)
        return "%s/%s/" % (CarSpecPrefix, specId)

    def genConfigSpecUrl(self, specId):
        configSpecTemplate = "https://car.autohome.com.cn/config/spec/%s.html"
        # https://car.autohome.com.cn/config/spec/43593.html
        return configSpecTemplate % specId

    def to10KPrice(self, originPrice):
        tenKPrice = ""
        # 19.08 / '19.08' -> '19.08万'
        if isinstance(originPrice, str):
            tenKPrice = "%s万" % originPrice
        elif isinstance(originPrice, float):
            tenKPrice = "%.2f万" % originPrice
        elif isinstance(originPrice, int):
            tenKPrice = "%s.00万" % originPrice

        return tenKPrice

    def extractSpecId(self, specUrl):
        carSpedId = ""
        # https://www.autohome.com.cn/spec/41511/#pvareaid=3454492
        # https://www.autohome.com.cn/spec/2304/
        foundSpecId = re.search("spec/(?P<specId>\d+)", specUrl)
        print("foundSpecId=%s" % foundSpecId)
        if foundSpecId:
            carSpedId = foundSpecId.group("specId")
            print("carSpedId=%s" % carSpedId)
        return carSpedId

    # @every(minutes=24 * 60)
    def on_start(self):
        # autohomeEntryUrl = "https://www.autohome.com.cn/car/"
        # self.crawl(autohomeEntryUrl, callback=self.carBrandListCallback)
        for eachLetter in list(string.ascii_lowercase):
            letterUpper = eachLetter.upper()
            # # for debug
            # letterUpper = "H"
            print("letterUpper=%s" % letterUpper)
            self.crawl("https://www.autohome.com.cn/grade/carhtml/%s.html" % eachLetter,
                save={"initials": letterUpper},
                callback=self.gradCarHtmlPage)

    @catch_status_code_error
    def gradCarHtmlPage(self, response):
        print("gradCarHtmlPage: response=", response)

        # picSeriesItemList = response.doc('.rank-list-ul li div a[href*="/pic/series"]').items()
        # print("picSeriesItemList=", picSeriesItemList)
        # print("len(picSeriesItemList)=%s"%(len(picSeriesItemList)))
        # for each in picSeriesItemList:
        #     self.crawl(each.attr.href, callback=self.picSeriesPage)

        saveDict = response.save
        print("saveDict=", saveDict)
        initials = saveDict["initials"]
        print("initials=", initials)
        respText = response.text
        # print("respText=", respText)

        """
        <dl id="33" olr="6">
            <dt><a href="//car.autohome.com.cn/price/brand-33.html#pvareaid=2042362"><img width="50" height="50"
                    src="//car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png"></a>
                <div><a href="//car.autohome.com.cn/price/brand-33.html#pvareaid=2042362">奥迪</a></div>
            </dt>
        """
        # brandDoc = response.doc('dl dt')
        # print("brandDoc=%s" % brandDoc)
        # brandListDoc = response.doc('dl[id and orl] dt')
        # dlListDoc = response.doc('dl[id and orl]').items()
        # dlListDoc = response.doc("dl[id*=''][orl*='']").items()
        # dlListDoc = response.doc("dl[orl*='']").items()
        # dlListDoc = response.doc("dl").items()
        # dlListDoc = response.doc("dl:regex(id, \d+)").items()
        # dlListDoc = response.doc("dl:regex(id,[0-9])").items()
        # dlListDoc = response.doc("dl[id]").items()
        dlListDoc = response.doc("dl[olr]").items()
        print("type(dlListDoc)=%s" % type(dlListDoc))
        dlList = list(dlListDoc)
        print("len(dlList)=%s" % len(dlList))
        print("dlList=%s" % dlList)
        for curBrandIdx, eachDlDoc in enumerate(dlList):
            print("%s [%d] %s" % ('#'*30, curBrandIdx, '#'*30))

            dtDoc = eachDlDoc.find("dt")
            # print("dtDoc=%s" % dtDoc)
            # <a href="//car.autohome.com.cn/price/brand-33.html#pvareaid=2042362"><img width="50" height="50" src="//car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png"></a>
            brandLogoDoc = dtDoc.find('a img')
            # print("brandLogoDoc=%s" % brandLogoDoc)
            carBrandLogoUrl = brandLogoDoc.attr["src"]
            print("carBrandLogoUrl=%s" % carBrandLogoUrl)
            # <div><a href="//car.autohome.com.cn/price/brand-33.html#pvareaid=2042362">奥迪</a></div>
            brandADoc = dtDoc.find('div a')
            print("brandADoc=%s" % brandADoc)
            # <a href="https://car.autohome.com.cn/price/brand-63.html#pvareaid=2042362">日产</a>
            carBrandName = brandADoc.text()
            print("carBrandName=%s" % carBrandName)
            carBrandUrl = brandADoc.attr["href"]
            print("carBrandUrl=%s" % carBrandUrl)
            carBrandId = ""
            foundBrandId = re.search("brand-(?P<carBrandId>\d+)\.html", carBrandUrl)
            # print("foundBrandId=%s" % foundBrandId)
            if foundBrandId:
                carBrandId = foundBrandId.group("carBrandId")
            print("carBrandId=%s" % carBrandId) # 63

            # <div class="h3-tit"><a href="//car.autohome.com.cn/price/brand-33-9.html#pvareaid=2042363">一汽-大众奥迪</a></div>
            # merchantDocGenerator = response.doc("dd div[class='h3-tit'] a").items()
            # ddDoc = eachDlDoc.find("dd")
            ddDoc = eachDlDoc.find("dd")
            # print("ddDoc=%s" % ddDoc)

            merchantDocGenerator = ddDoc.items("div[class='h3-tit'] a")
            merchantDocList = list(merchantDocGenerator)
            # print("merchantDocList=%s" % merchantDocList)
            merchantDocLen = len(merchantDocList)
            print("merchantDocLen=%s" % merchantDocLen)

            # <ul class="rank-list-ul" 0>
            # merchantRankDocGenerator = response.doc("dd ul[class='rank-list-ul']")
            # merchantRankDocGenerator = response.doc("dd ul[class='rank-list-ul']").items()
            merchantRankDocGenerator = ddDoc.items("ul[class='rank-list-ul']")
            merchantRankDocList = list(merchantRankDocGenerator)
            # print("merchantRankDocList=%s" % merchantRankDocList)
            merchantRankDocListLen = len(merchantRankDocList)
            print("merchantRankDocListLen=%s" % merchantRankDocListLen)

            for curIdx, merchantItem  in enumerate(merchantDocList):
            # for curIdx, merchantItem  in enumerate(merchantDocGenerator):
                # print("%s" % "="*80)
                print("%s [%d] %s" % ('='*30, curIdx, '='*30))
                # print("type(merchantItem)=%s" % type(merchantItem))
                # print("[%d] merchantItem=%s" % (curIdx, merchantItem))
                # print("[%d] merchantItem=%s" % (curIdx, merchantItem))
                carMerchantName = merchantItem.text()
                print("carMerchantName=%s" % carMerchantName)
                merchantItemAttr = merchantItem.attr
                # print("merchantItemAttr=%s" % merchantItemAttr)
                carMerchantUrl = merchantItemAttr["href"]
                print("carMerchantUrl=%s" % carMerchantUrl)

                # curSubBrandDict = {
                #     "brandName": brandName,
                #     "carBrandLogoUrl": carBrandLogoUrl,
                #     "carMerchantName": carMerchantName,
                #     "carMerchantUrl": carMerchantUrl,
                # }
                # self.send_message(self.project_name, curSubBrandDict, url=carMerchantUrl)

                merchantRankDoc = merchantRankDocList[curIdx]
                # print("merchantRankDoc=%s" % merchantRankDoc)
                # print("type(merchantRankDoc)=%s" % type(merchantRankDoc))

                # type(merchantRankDoc)=<class 'lxml.html.HtmlElement'>
                # merchantRankHtml = etree.tostring(merchantRankDoc)

                # type(merchantRankDoc)=<class 'pyquery.pyquery.PyQuery'>
                # merchantRankHtml = merchantRankDoc.html()

                # print("merchantRankHtml=%s" % merchantRankHtml)

                # <li id="s3170">
                # carSeriesDocGenerator = merchantRankDoc.find("li")
                # carSeriesDocGenerator = merchantRankDoc.find("li[id*='s']")
                carSeriesDocGenerator = merchantRankDoc.items("li[id*='s']")
                # print("type(carSeriesDocGenerator)=%s" % type(carSeriesDocGenerator))
                carSeriesDocList = list(carSeriesDocGenerator)
                # print("type(carSeriesDocList)=%s" % type(carSeriesDocList))
                # print("carSeriesDocList=%s" % carSeriesDocList)
                carSeriesDocListLen = len(carSeriesDocList)
                # print("carSeriesDocListLen=%s" % carSeriesDocListLen)

                for curSeriesIdx, eachCarSeriesDoc in enumerate(carSeriesDocList):
                    print("%s [%d] %s" % ('-'*30, curSeriesIdx, '-'*30))
                    # print("[%d] eachCarSeriesDoc=%s" % (curSeriesIdx, eachCarSeriesDoc))
                    # print("type(eachCarSeriesDoc)=%s" % type(eachCarSeriesDoc)) # type(eachCarSeriesDoc)=<class 'lxml.html.HtmlElement'>
                    # <h4><a href="//www.autohome.com.cn/3170/#levelsource=000000000_0&pvareaid=101594">奥迪A3</a></h4>
                    carSeriesInfoDoc = eachCarSeriesDoc.find("h4 a")
                    # print("type(carSeriesInfoDoc)=%s" % type(carSeriesInfoDoc))
                    # print("carSeriesInfoDoc=%s" % carSeriesInfoDoc)
                    carSeriesName = carSeriesInfoDoc.text()
                    print("carSeriesName=%s" % carSeriesName)
                    carSeriesUrl = carSeriesInfoDoc.attr.href
                    print("carSeriesUrl=%s" % carSeriesUrl)

                    # <div>指导价：<a class="red" href="//www.autohome.com.cn/3170/price.html#pvareaid=101446">19.32-23.46万</a></div>
                    # 厂商指导价=厂商建议零售价格=MSRP=Manufacturer's Suggested Retail Price
                    # carSeriesMsrpDoc = eachCarSeriesDoc.find("div a")
                    carSeriesMsrpDoc = eachCarSeriesDoc.find("div a[class='red']")
                    # print("carSeriesMsrpDoc=%s" % carSeriesMsrpDoc)
                    carSeriesMsrp = carSeriesMsrpDoc.text()
                    print("carSeriesMsrp=%s" % carSeriesMsrp)
                    carSeriesMsrpUrl = carSeriesMsrpDoc.attr.href
                    print("carSeriesMsrpUrl=%s" % carSeriesMsrpUrl)

                    carSeriesDict = {
                        "carBrandName": carBrandName,
                        "carBrandId": carBrandId,
                        "carBrandLogoUrl": carBrandLogoUrl,
                        "carMerchantName": carMerchantName,
                        "carMerchantUrl": carMerchantUrl,
                        "carSeriesName": carSeriesName,
                        "carSeriesUrl": carSeriesUrl,
                        "carSeriesMsrp": carSeriesMsrp,
                        "carSeriesMsrpUrl": carSeriesMsrpUrl,
                    }
                    # self.send_message(self.project_name, carSeriesDict, url=carSeriesUrl)
                    self.crawl(carSeriesUrl,
                        callback=self.carSeriesDetailPage,
                        save=carSeriesDict,
                    )

    @catch_status_code_error
    def carSeriesDetailPage(self, response):
        print("in carSeriesDetailPage")
        carSeriesDict = response.save
        print("carSeriesDict=%s" % carSeriesDict)

        carModelDict = copy.deepcopy(carSeriesDict)

        carSeriesUrl = response.url
        print("carSeriesUrl=%s" % carSeriesUrl)

        carSeriesMainImgUrl = ""
        # carSeriesId = ""
        carSeriesLevelId = ""
        carSeriesMsrp = ""
        carSeriesMinPrice = ""
        carSeriesMaxPrice = ""

        # carSeriesUrl=https://www.autohome.com.cn/2123/#levelsource=000000000_0&pvareaid=101594
        foundSeriesId = re.search("www\.autohome\.com\.cn/(?P<seriesId>\d+)/", carSeriesUrl)
        carSeriesId = foundSeriesId.group("seriesId")
        # carSeriesId = int(carSeriesId)
        print("carSeriesId=%s" % carSeriesId) # 2123
        carModelDict["carSeriesId"] = carSeriesId

        carSeriesHtml = response.text
        print("type(carSeriesHtml)=%s" % type(carSeriesHtml)) # <class 'str'>
        # print("carSeriesHtml=%s" % carSeriesHtml)

        foundLevelId = re.search("var\s+levelid\s+=", carSeriesHtml)
        print("foundLevelId=%s" % foundLevelId)
        isNewLayoutHtml = bool(foundLevelId)
        print("isNewLayoutHtml=%s" % isNewLayoutHtml)
        foundShowCityId = re.search("var\s+showCityId\s+=", carSeriesHtml)
        print("foundShowCityId=%s" % foundShowCityId)
        isOldLayoutHtml = bool(foundShowCityId)
        print("isOldLayoutHtml=%s" % isOldLayoutHtml)

        if isOldLayoutHtml:
            # Q开头
            # https://www.autohome.com.cn/grade/carhtml/q.html
            # ->
            # 东风悦达起亚-千里马
            # https://www.autohome.com.cn/142/#levelsource=000000000_0&pvareaid=101594
            # 其他：
            #
            # 一汽丰田-花冠
            # https://www.autohome.com.cn/109/#levelsource=000000000_0&pvareaid=101594
            #
            # 昶洧-昶洧 SUV
            # https://www.autohome.com.cn/4550/#levelsource=000000000_0&pvareaid=101594

            """
            <div class="car_detail " id="tab1-2">
                <div class="models">
                <!--年代-->
                    <div class="header">
                        <div class="car_price">
                            <span class="years">2005款</span>
                            <span class="price">指导价（停售）：<strong class="red">6.28万-9.18万</strong></span>
                            <span class="price">二手车价格：<strong class="red"><a class='cd60000' href='//www.che168.com/china/qiya/qianlima/a0_0msdgscncgpiltocsp1exs276/?pvareaid=103693'>0.39万-1.30万</a></strong></span>
            。。。
            <div class="car_detail current" id="tab1-1">
                <div class="models">
                    <!--年代-->
                    <div class="header">
                        <div class="car_price">
                            <span class="years">2006款</span>
                            <span class="price">指导价（停售）：<strong class="red">7.28万-8.58万</strong></span>
            。。。
            """
            carDetailDivGenerator = response.doc("div[class^='car_detail']").items()
            print("carDetailDivGenerator=%s" % carDetailDivGenerator)
            carDetailDivList = list(carDetailDivGenerator)
            print("carDetailDivList=%s" % carDetailDivList)
            for curDivIdx, eachCarDetailDoc in enumerate(carDetailDivList):
                print("%s [%d] %s" % ('#'*30, curDivIdx, '#'*30))

                if curDivIdx == 0:
                    # use first car model as series: main img, msrp, ...
                    """
                    <div class="models_info">
                        <dl class='models_pics'>
                            <dt><a href='//car.autohome.com.cn/photolist/series/2305/23796.html?pvareaid=101468'><img
                                src='https://car0.autoimg.cn/upload/spec/1344/t_1344388912334.jpg' width='240'
                                height='180' /></a></dt>
                    """
                    # modelMainImgDocListGenerator = response.doc("div[class='models_info'] dl[class='models_pics'] dt a img").items()
                    # modelMainImgDocList = list(modelMainImgDocListGenerator)
                    # firstModelMainImgDoc = modelMainImgDocList[0]
                    firstModelMainImgDoc = eachCarDetailDoc.find("div[class='models_info'] dl[class='models_pics'] dt a img")
                    firstModelMainImgUrl = firstModelMainImgDoc.attr["src"]
                    print("firstModelMainImgUrl=%s" % firstModelMainImgUrl)
                    carSeriesMainImgUrl = firstModelMainImgUrl
                    print("carSeriesMainImgUrl=%s" % carSeriesMainImgUrl)

                    carModelDict["carSeriesMainImgUrl"] = carSeriesMainImgUrl

                    # <div class="car_price">
                    #   <span class="price">指导价（停售）：<strong class="red">7.28万-8.58万</strong></span>
                    carPriceStrongDocGenerator = eachCarDetailDoc.items("div[class='car_price'] span[class='price'] strong[class='red']")
                    print("carPriceStrongDocGenerator=%s" % carPriceStrongDocGenerator)
                    if carPriceStrongDocGenerator:
                        carPriceStrongDocList = list(carPriceStrongDocGenerator)
                        print("carPriceStrongDocList=%s" % carPriceStrongDocList)
                        carPriceStrongDoc = carPriceStrongDocList[0]
                        print("carPriceStrongDoc=%s" % carPriceStrongDoc)
                        carPriceMinMax = carPriceStrongDoc.text()
                        print("carPriceMinMax=%s" % carPriceMinMax)
                        if carPriceMinMax:
                            foundMinMax = re.search("(?P<minPrice>[\d\.]+)万-(?P<maxPrice>[\d\.]+)万", carPriceMinMax)
                            print("foundMinMax=%s" % foundMinMax)
                            if foundMinMax:
                                minPrice = foundMinMax.group("minPrice")
                                print("minPrice=%s" % minPrice)
                                minPriceFloat = float(minPrice)
                                print("minPriceFloat=%s" % minPriceFloat)
                                maxPrice = foundMinMax.group("maxPrice")
                                print("maxPrice=%s" % maxPrice)
                                maxPriceFloat = float(maxPrice)
                                print("maxPriceFloat=%s" % maxPriceFloat)
                                averageMsrcPrice = (minPriceFloat + maxPriceFloat) / 2.0
                                print("averageMsrcPrice=%s" % averageMsrcPrice)

                                # carSeriesMsrp = "%.2f万" % averageMsrcPrice
                                carSeriesMsrp = self.to10KPrice(averageMsrcPrice)
                                print("carSeriesMsrp=%s" % carSeriesMsrp)
                                # carSeriesMinPrice = "%.2f万" % minPriceFloat
                                carSeriesMinPrice = self.to10KPrice(minPriceFloat)
                                print("carSeriesMinPrice=%s" % carSeriesMinPrice)
                                # carSeriesMaxPrice = "%.2f万" % maxPriceFloat
                                carSeriesMaxPrice = self.to10KPrice(maxPriceFloat)
                                print("carSeriesMaxPrice=%s" % carSeriesMaxPrice)

                                carModelDict["carSeriesMsrp"] = carSeriesMsrp
                                carModelDict["carSeriesMinPrice"] = carSeriesMinPrice
                                carModelDict["carSeriesMaxPrice"] = carSeriesMaxPrice
                    print("")

                self.processSingleCarDetailDiv(carModelDict, eachCarDetailDoc)

        elif isNewLayoutHtml:
            # https://www.autohome.com.cn/3170/#levelsource=000000000_0&pvareaid=101594

            """
            <div class="information-pic">
                <div class="pic-main">
                。。。
                        <picture>
                            。。。
                            <img sizes="380px" width="380" height="285"
                                src="//car2.autoimg.cn/cardfs/product/g1/M04/0B/F0/380x285_0_q87_autohomecar__ChwFqV8YG-aACch8AAkAdoJoSYM874.jpg"
                                srcset="//car2.autoimg.cn/cardfs/product/g1/M04/0B/F0/380x285_0_q87_autohomecar__ChwFqV8YG-aACch8AAkAdoJoSYM874.jpg 380w, //car2.autoimg.cn/cardfs/product/g1/M04/0B/F0/760x570_0_q87_autohomecar__ChwFqV8YG-aACch8AAkAdoJoSYM874.jpg 760w">
                        </picture>
            """
            mainImgDoc = response.doc("div[class='information-pic'] div[class='pic-main'] picture img")
            print("mainImgDoc=%s" % mainImgDoc)
            carSeriesMainImgUrl = mainImgDoc.attr["src"]
            print("carSeriesMainImgUrl=%s" % carSeriesMainImgUrl)
            carModelDict["carSeriesMainImgUrl"] = carSeriesMainImgUrl

            """
            <script type="text/javascript">
                。。。
                var seriesid = '2123';
                var seriesname='哈弗H6';
                var yearid = '0';
                var brandid = '181';
                var levelid = '17';
                var levelname='紧凑型SUV';
                var fctid = '4';
                var SeriesMinPrice='9.80';
                var SeriesMaxPrice='14.10';
            """

            infoKeyList = [
                "seriesid",
                # "seriesname", # has got
                # "yearid", # no need
                "brandid",
                "levelid",
                "levelname",
                # "fctid", # unknown meaning
                "SeriesMinPrice",
                "SeriesMaxPrice",
            ]
            InfoDict = {}
            for eachInfoKey in infoKeyList:
                curPattern = "var\s+%s\s*=\s*'(?P<infoValue>[^']+)'\s*;" % eachInfoKey
                print("curPattern=%s" % curPattern)
                foundInfo = re.search(curPattern, carSeriesHtml)
                print("foundInfo=%s" % foundInfo)
                # if foundInfo:
                infoValue = foundInfo.group("infoValue")
                print("infoValue=%s" % infoValue)
                InfoDict[eachInfoKey] = infoValue
            print("InfoDict=%s" % InfoDict)

            # if "seriesid" in InfoDict:
            carSeriesId = InfoDict["seriesid"] # 2123
            carModelDict["carSeriesId"] = carSeriesId
            # carModelDict["carSeriesName"] = InfoDict["seriesname"] # 哈弗H6
            # if "brandid" in InfoDict:
            carModelDict["carBrandId"] = InfoDict["brandid"] # 181
            # if "levelid" in InfoDict:
            carSeriesLevelId = InfoDict["levelid"] # 17
            carModelDict["carSeriesLevelId"] = carSeriesLevelId
            # if "levelname" in InfoDict:
            carModelDict["carSeriesLevelName"] = InfoDict["levelname"] # 紧凑型SUV
            # if "SeriesMinPrice" in InfoDict:
            carSeriesMinPrice = InfoDict["SeriesMinPrice"] # 9.80
            carModelDict["carSeriesMinPrice"] = self.to10KPrice(carSeriesMinPrice)
            # if "SeriesMaxPrice" in InfoDict:
            carSeriesMaxPrice = InfoDict["SeriesMaxPrice"] # 14.10
            carModelDict["carSeriesMaxPrice"] = self.to10KPrice(carSeriesMaxPrice)

            """
            <div class="series-list">
            。。。
                <li class="more-dropdown">
                    <a href="javascript:void(0);" target="_self" data-toggle="tab" class="tab-disabled" data-target="#specWrap-3">停售款 <i class="athm-iconfont athm-iconfont-arrowdown"></i></a>
                    <ul class="dropdown-con" id="haltList">
                        <li><a href="javascript:void(0);" target="_self" data-toggle="tab" data-yearid="11691">2019款</a></li>
                        ...
                        <li><a href="javascript:void(0);" target="_self" data-toggle="tab" data-yearid="3100">2011款</a></li>
                    </ul>
                </li>
            """
            haltADocGenerator = response.doc("li[class='more-dropdown'] ul[id='haltList'] li a").items()
            print("type(haltADocGenerator)=%s" % type(haltADocGenerator))
            print("haltADocGenerator=%s" % haltADocGenerator)
            haltADocList = list(haltADocGenerator)
            print("haltADocList=%s" % haltADocList)
            for curLiIdx, eachHatADoc in enumerate(haltADocList):
                print("%s [%d] %s" % ('%'*30, curLiIdx, '%'*30))
                self.processSingleHaltA(carModelDict, eachHatADoc)

            # """
            # <div class="information-summary">
            #     <dl class="information-price">
            #         ...
            #         <dd class="type">
            #             <span class="type__item">紧凑型车</span>
            # """
            # carLevelDoc = response.doc("div[class='information-summary'] dl[class='information-price'] dd[class='type'] span[class='type__item']").eq(0)
            # print("carLevelDoc=%s" % carLevelDoc)
            # carSeriesLevelName = carLevelDoc.text()
            # print("carSeriesLevelName=%s" % carSeriesLevelName)
            # carModelDict["carSeriesLevelName"] = carSeriesLevelName

            carSeriesContentDoc = response.doc("div[class='series-content']")
            # print("carSeriesContentDoc=%s" % carSeriesContentDoc)
            # carSpecWrapDoc = carSeriesContentDoc.find("div[class^='spec-wrap']")
            # carSpecWrapDoc = carSeriesContentDoc.find("div[class^='spec-wrap active']")
            carSpecWrapDocGenerator = carSeriesContentDoc.items("div[class^='spec-wrap']")
            print("carSpecWrapDocGenerator=%s" % carSpecWrapDocGenerator)
            carSpecWrapDocList = list(carSpecWrapDocGenerator)
            print("carSpecWrapDocList=%s" % carSpecWrapDocList)
            for curSpecWrapIdx, eachSpecWrapDoc in enumerate(carSpecWrapDocList):
                print("%s [%d] %s" % ('#'*30, curSpecWrapIdx, '#'*30))
                self.processSingleSpecWrapDiv(carModelDict, eachSpecWrapDoc)

    def processSingleCarDetailDiv(self, carModelDict, curCarDetailDoc):
        print("in processSingleCarDetailDiv")
        curCarModelGroupDict = copy.deepcopy(carModelDict)

        # <span class="years">2006款</span>
        modelYearDoc = curCarDetailDoc.find("span[class='years']")
        print("modelYearDoc=%s" % modelYearDoc)
        carModelYear = modelYearDoc.text()
        print("carModelYear=%s" % carModelYear)
        curCarModelGroupDict["carModelYear"] = carModelYear

        """
        <div class="modelswrap">
            <!-- 信息 start -->
            <div class="models_info">
                <dl class='models_prop'>
                    <dt>发动机：</dt>
                    <dd><span>1.3L</span><span>1.6L</span></dd>
                </dl>
                <dl class='models_prop'>
                    <dt>变速箱：</dt>
                    <dd><span>手动</span><span>自动</span></dd>
                    <dt>车身结构：</dt>
                    <dd><span>三厢</span></dd>
                </dl>
        """
        # modelsPropDdList = curCarDetailDoc.find("div[class='modelswrap'] div[class='models_info'] dl[class='models_prop'] dd")
        modelsPropDdGenerator = curCarDetailDoc.items("div[class='modelswrap'] div[class='models_info'] dl[class='models_prop'] dd")
        print("modelsPropDdGenerator=%s" % modelsPropDdGenerator)
        modelsPropDdList = list(modelsPropDdGenerator)
        print("modelsPropDdList=%s" % modelsPropDdList)
        engineValueDoc = modelsPropDdList[0]
        print("engineValueDoc=%s" % engineValueDoc)
        carModelEngine = engineValueDoc.text()
        print("carModelEngine=%s" % carModelEngine)

        gearBoxValueDoc = modelsPropDdList[1]
        print("gearBoxValueDoc=%s" % gearBoxValueDoc)
        carModelGearBox = gearBoxValueDoc.text()
        print("carModelGearBox=%s" % carModelGearBox)

        bodyStructureValueDoc = modelsPropDdList[2]
        print("bodyStructureValueDoc=%s" % bodyStructureValueDoc)
        carModelBodyStructure = bodyStructureValueDoc.text()
        print("carModelBodyStructure=%s" % carModelBodyStructure)

        curCarModelGroupDict["carModelGearBox"] = carModelGearBox # 手动自动
        curCarModelGroupDict["carModelDriveType"] = ""
        curCarModelGroupDict["carModelBodyStructure"] = carModelBodyStructure

        # curCarModelGroupDict["carModelEnvStandard"] = ""
        # carModelPower = carModelEngine
        # print("carModelPower=%s" % carModelPower)
        # curCarModelGroupDict["carModelPower"] = carModelPower
        curCarModelGroupDict["carModelEngine"] = carModelEngine # 1.3L1.6L

        carModelGroupName = "%s %s %s" % (carModelEngine, carModelGearBox, carModelBodyStructure)
        print("carModelGroupName=%s" % carModelGroupName)
        curCarModelGroupDict["carModelGroupName"] = carModelGroupName

        """
        <table class='models_tab tableline' cellspacing='0' cellpadding='0' border='0'>
            <tr>
                <td class='name_d'>
                    <div class='name'><a title='2006款 1.6L MT特别版GL' href='spec/2304/'>2006款 1.6L MT特别版GL</a></div>
                </td>
                <td class='price_d'>
                    <div class='price01'>8.18万</div>
                </td>
        """
        modelsTrDocGenerator = curCarDetailDoc.items("table[class^='models_tab'] tr")
        print("modelsTrDocGenerator=%s" % modelsTrDocGenerator)
        modelsTrDocList = list(modelsTrDocGenerator)
        print("modelsTrDocList=%s" % modelsTrDocList)
        for curTabIdx, eachModelTrDoc in enumerate(modelsTrDocList):
            print("%s [%d] %s" % ('='*30, curTabIdx, '='*30))
            self.processSingleModelsTr(curCarModelGroupDict, eachModelTrDoc)

    def processSingleModelsTr(self, curCarModelGroupDict, curModelTrDoc):
        curTrCarModeDict = copy.deepcopy(curCarModelGroupDict)
        # print("curModelTrDoc=%s" % curModelTrDoc)
        nameADoc = curModelTrDoc.find("td[class='name_d'] div[class='name'] a")
        print("nameADoc=%s" % nameADoc)
        carModelName = nameADoc.text()
        print("carModelName=%s" % carModelName)

        carModelSpecUrl = nameADoc.attr["href"]
        # bug -> wrong url:
        # https://www.autohome.com.cn/142/spec/2304/
        # need repace
        # https://www.autohome.com.cn/142/spec/2304/
        # to
        # https://www.autohome.com.cn/spec/2304/
        foundSpecId = re.search("spec/(?P<specId>\d+)", carModelSpecUrl)
        carModelSpecId = foundSpecId.group("specId")
        print("carModelSpecId=%s" % carModelSpecId) # 2304
        carModelSpecUrl = self.genSpecUrl(carModelSpecId)
        print("carModelSpecUrl=%s" % carModelSpecUrl)

        priceDivDoc = curModelTrDoc.find("td[class='price_d'] div[class='price01']")
        print("priceDivDoc=%s" % priceDivDoc)
        carModelMsrp = priceDivDoc.text()
        print("carModelMsrp=%s" % carModelMsrp)

        curTrCarModeDict["carModelName"] = carModelName
        curTrCarModeDict["carModelSpecUrl"] = carModelSpecUrl
        curTrCarModeDict["carModelMsrp"] = carModelMsrp

        self.processSingleResult(curTrCarModeDict)

    def processSingleHaltA(self, carModelDict, curHatADoc):
        curHaltCarDict = copy.deepcopy(carModelDict)
        print("curHatADoc=%s" % curHatADoc)
        yearName = curHatADoc.text()
        print("yearName=%s" % yearName)
        yearId = curHatADoc.attr["data-yearid"]
        print("yearId=%s" % yearId)

        # getHaltSpecUrl = "https://www.autohome.com.cn/ashx/car/Spec_ListByYearId.ashx?seriesid=%s&syearid=%s&levelid=%s" % (curHaltCarDict["carSeriesId"], yearId, curHaltCarDict["carSeriesLevelId"])
        carSeriesId = curHaltCarDict["carSeriesId"]
        carSeriesLevelId = curHaltCarDict["carSeriesLevelId"]
        if carSeriesId and carSeriesLevelId:
            getHaltSpecUrl = "https://www.autohome.com.cn/ashx/car/Spec_ListByYearId.ashx?seriesid=%s&syearid=%s&levelid=%s" % (carSeriesId, yearId, carSeriesLevelId)
            # https://www.autohome.com.cn/ashx/car/Spec_ListByYearId.ashx?seriesid=2123&syearid=10379&levelid=17
            print("getHaltSpecUrl=%s" % getHaltSpecUrl)
            self.crawl(getHaltSpecUrl,
                callback=self.haltCarSpecCallback,
                save=curHaltCarDict,
            )

    def processSingleSpecWrapDiv(self, curCarModelDict, curSpecWrapDoc):
        curSpecWrapCarDict = copy.deepcopy(curCarModelDict)
        # print("curSpecWrapDoc=%s" % curSpecWrapDoc)
        """
        <!--即将上市 start-->
        <div class="spec-wrap  active" id="specWrap-1">

            <dl class="halt-spec">
                <dt>
                    <div class="spec-name">
                        <span>参数配置未公布</span>
                    </div>

        <dl class="halt-spec">
            <dt>
                <div class="spec-name">
                    <span>1.5升 涡轮增压 169马力 国VI</span>
                </div>
        """
        # dlDoc = curSpecWrapDoc.find("dl[class='']")
        # dlDoc = curSpecWrapDoc.find("dl")
        dlListDocGenerator = curSpecWrapDoc.items("dl")
        print("dlListDocGenerator=%s" % dlListDocGenerator)
        dlDocList = list(dlListDocGenerator)
        print("dlDocList=%s" % dlDocList)
        for curDlIdx, eachDlDoc in enumerate(dlDocList):
            print("%s [%d] %s" % ('='*30, curDlIdx, '='*30))
            self.processSingleSpecDl(curSpecWrapCarDict, eachDlDoc)

    def processSingleSpecDl(self, curSpecWrapCarDict, curDlDoc):
        curDlCarDict = copy.deepcopy(curSpecWrapCarDict)
        # print("curDlDoc=%s" % curDlDoc)
        """
            <dt>
                <div class="spec-name">
                    <span>1.5升 涡轮增压 169马力 国VI</span>
        """
        dtDoc = curDlDoc.find("dt")
        # print("dtDoc=%s" % dtDoc)
        groupSpecNameSpanDoc = dtDoc.find("div[class='spec-name'] span")
        print("groupSpecNameSpanDoc=%s" % groupSpecNameSpanDoc)
        carModelGroupName = ""
        if groupSpecNameSpanDoc:
            carModelGroupName = groupSpecNameSpanDoc.text()
            print("carModelGroupName=%s" % carModelGroupName)

        curDlCarDict["carModelGroupName"] = carModelGroupName

        # <dd data-sift1="2020款" data-sift2="国VI" data-sift3="1.5T" data-sift4="7挡双离合" class="">
        ddListDoc = curDlDoc.items("dd")
        print("ddListDoc=%s" % ddListDoc)
        for curDdIdx, eachDdDoc in enumerate(ddListDoc):
            print("%s [%d] %s" % ('-'*30, curDdIdx, '-'*30))
            self.processSingleSiftDd(curDlCarDict, eachDdDoc)

    def processSingleSiftDd(self, curDlCarDict, curDdDoc):
        print("in processSingleSiftDd")
        curDdCarDict = copy.deepcopy(curDlCarDict)

        curDdAttr = curDdDoc.attr
        """
        正常：
            <dd data-sift1="2020款" data-sift2="国VI" data-sift3="1.5T" data-sift4="7挡双离合" class="">
                ...
        特殊：
            无sift：
                <dd data-electricspecid="47050">
            电动的 sift位置不同：
                https://www.autohome.com.cn/5240/
                    <dd data-electricspecid="42875" data-sift1="2019款" data-sift2="100KW" data-sift3="265公里" data-sift4="单速">
            混动 sift位置也不同：
                https://www.autohome.com.cn/4460/
                    <dd data-electricspecid="37077" data-sift1="2019款" data-sift2="国V" data-sift3="2.0T" data-sift4="8挡手自一体">
        """
        # print("curDdAttr=%s" % curDdAttr)
        # carModelDataSift1 = curDdAttr["data-sift1"]
        # print("carModelDataSift1=%s" % carModelDataSift1)
        carModelYear = curDdAttr["data-sift1"]
        print("carModelYear=%s" % carModelYear)
        carModelDataSift2 = curDdAttr["data-sift2"]
        print("carModelDataSift2=%s" % carModelDataSift2)
        carModelDataSift3 = curDdAttr["data-sift3"]
        print("carModelDataSift3=%s" % carModelDataSift3)
        carModelDataSift4 = curDdAttr["data-sift4"]
        print("carModelDataSift4=%s" % carModelDataSift4)

        curDdCarDict["carModelYear"] = carModelYear
        # curDdCarDict["carModelEnvStandard"] = carModelEnvStandard
        # curDdCarDict["carModelPower"] = carModelPower
        # curDdCarDict["carModelGearBox"] = carModelGearBox
        # curDdCarDict["carModelDataSift1"] = carModelDataSift1
        curDdCarDict["carModelDataSift2"] = carModelDataSift2
        curDdCarDict["carModelDataSift3"] = carModelDataSift3
        curDdCarDict["carModelDataSift4"] = carModelDataSift4

        """
        <div class="spec-name">
            <div class="name-param">
                <p data-gcjid="41511" id="spec_41511">
                    <a href="/spec/41511/#pvareaid=3454492" class="name">2020款 1.5GDIT 自动铂金舒适版</a>
                    <span class="athm-badge athm-badge--grey is-plain">停产在售</span>
                <span class="athm-badge athm-badge--orange">特惠</span></p>
                <p><span class="type-default">前置前驱</span><span class="type-default">7挡双离合</span></p>
            </div>
        </div>
        """
        specNameDoc = curDdDoc.find("div[class='spec-name']")
        # print("specNameDoc=%s" % specNameDoc)
        specADoc = specNameDoc.find("p a[class='name']")
        # print("specADoc=%s" % specADoc)
        carModelName = specADoc.text()
        print("carModelName=%s" % carModelName) # 2020款 1.5GDIT 自动铂金舒适版
        carModelSpecUrl = specADoc.attr["href"]
        print("carModelSpecUrl=%s" % carModelSpecUrl) # https://www.autohome.com.cn/spec/41511/#pvareaid=3454492
        typeDefaultListDoc = specNameDoc.items("p span[class='type-default']")
        print("typeDefaultListDoc=%s" % typeDefaultListDoc)
        typeDefaultList = list(typeDefaultListDoc)
        print("typeDefaultList=%s" % typeDefaultList)
        carModelDriveType = ""
        carModelGearBox = ""
        if typeDefaultList:
            """
            正常：
                <p>
                    <span class="type-default">前置前驱</span>
                    <span class="type-default">7挡双离合</span>
                </p>

            特殊：
                https://www.autohome.com.cn/4605/
                <p>
                    <span class="type-default">电动</span>
                    <span class="type-default">前置前驱</span>
                    <span class="type-default">AMT（组合10挡）</span>
                </p>

                https://www.autohome.com.cn/4460/
                <p>
                    <span class="type-default">电动</span>
                    <span class="type-default">前置四驱</span>
                    <span class="type-default">8挡手自一体</span>
                </p>

                https://www.autohome.com.cn/5240/
                <p>
                    <span class="type-default">电动</span>
                    <span class="type-default">前置前驱</span>
                    <span class="type-default">电动车单速变速箱</span>
                </p>
            """
            # spanTypeDefault0 = typeDefaultList[0]
            spanTypeDefault0 = typeDefaultList[-2]
            print("spanTypeDefault0=%s" % spanTypeDefault0)
            carModelDriveType = spanTypeDefault0.text()
            print("carModelDriveType=%s" % carModelDriveType)
            # 前置前驱
            # 前置四驱
            # spanTypeDefault1 = typeDefaultList[1]
            spanTypeDefault1 = typeDefaultList[-1]
            print("spanTypeDefault1=%s" % spanTypeDefault1)
            carModelGearBox = spanTypeDefault1.text()
            print("carModelGearBox=%s" % carModelGearBox)
            # 7挡双离合
            # AMT（组合10挡）
            # 8挡手自一体
            # 电动车单速变速箱

        curDdCarDict["carModelName"] = carModelName
        if not curDdCarDict["carModelYear"]:
            foundYearType = re.search("(?P<yearType>\d{4}款)", carModelName)
            if foundYearType:
                yearType = foundYearType.group("yearType")
                print("yearType=%s" % yearType)
                carModelYear = yearType
                print("extract year=%s from modelName=%s" % (carModelYear, carModelName))
                curDdCarDict["carModelYear"] = carModelYear

        curDdCarDict["carModelSpecUrl"] = carModelSpecUrl
        curDdCarDict["carModelDriveType"] = carModelDriveType # 前置前驱
        curDdCarDict["carModelGearBox"] = carModelGearBox # 7挡双离合

        """
        <div class="spec-guidance">
            <p class="guidance-price">
                <span>10.40万</span>
                <a href="//j.autohome.com.cn/pc/carcounter?type=1&amp;specId=41511&amp;pvareaid=3454617"><i class="athm-iconpng athm-iconpng-calculator"></i></a>
            </p>
        </div>

        <div class="spec-guidance">
            <p class="guidance-price">
                <span><span>暂无</span></span>
        """
        specGuidanceDoc = curDdDoc.find("div[class='spec-guidance']")
        # print("specGuidanceDoc=%s" % specGuidanceDoc)
        guidancePriceSpanDoc = specGuidanceDoc.find("p[class='guidance-price'] span")
        # print("guidancePriceSpanDoc=%s" % guidancePriceSpanDoc)
        carModelMsrp = guidancePriceSpanDoc.text()
        print("carModelMsrp=%s" % carModelMsrp)
        curDdCarDict["carModelMsrp"] = carModelMsrp

        self.processSingleResult(curDdCarDict)

    @catch_status_code_error
    def haltCarSpecCallback(self, response):
        prevCarModelDict = response.save
        carModelDict = copy.deepcopy(prevCarModelDict)
        print("carModelDict=%s" % carModelDict)

        respJson = response.json
        print("respJson=%s" % respJson)

        """
        [
            {
                "name": "1.5升 涡轮增压 169马力",
                "speclist": [
                    {
                        "specid": 36955,
                        "specname": "2019款 红标 1.5GDIT 自动舒适版",
                        "specstate": 40,
                        "minprice": 102000,
                        "maxprice": 102000,
                        "fueltype": 1,
                        "fueltypedetail": 1,
                        "driveform": "前置前驱",
                        "drivetype": "前驱",
                        "gearbox": "7挡双离合",
                        "evflag": "",
                        "newcarflag": "",
                        "subsidy": "",
                        "paramisshow": 1,
                        "videoid": 0,
                        "link2sc": "http://www.che168.com/china/hafu/hafuh6/7_8/",
                        "price2sc": "7.58万",
                        "price": "10.20万",
                        "syear": 2019
                    }, {
                        "specid": 36956,
                        "specname": "2019款 红标 1.5GDIT 自动都市版",
                        "specstate": 40,
                        "minprice": 109000,
                        "maxprice": 109000,
                        "fueltype": 1,
                        "fueltypedetail": 1,
                        "driveform": "前置前驱",
                        "drivetype": "前驱",
                        "gearbox": "7挡双离合",
                        "evflag": "",
                        "newcarflag": "",
                        "subsidy": "",
                        "paramisshow": 1,
                        "videoid": 0,
                        "link2sc": "",
                        "price2sc": "",
                        "price": "10.90万",
                        "syear": 2019
                    },
                    ...
        """
        if respJson:
            for eachModelGroupDict in respJson:
                modelGroupName = eachModelGroupDict["name"]
                modelSpecList = eachModelGroupDict["speclist"]
                for eachModelDict in modelSpecList:
                    curCarModelDict = copy.deepcopy(carModelDict)

                    carModelYear = "%s款" % eachModelDict["syear"]
                    # carModelSpecUrl = "%s/%s" % (CarSpecPrefix, eachModelDict["specid"])
                    carModelSpecUrl = self.genSpecUrl(eachModelDict["specid"])

                    curCarModelDict["carModelGroupName"] = modelGroupName
                    curCarModelDict["carModelYear"] = carModelYear
                    # curCarModelDict["carModelEnvStandard"] = ""
                    # curCarModelDict["carModelPower"] = ""
                    curCarModelDict["carModelDriveType"] = eachModelDict["drivetype"]
                    curCarModelDict["carModelGearBox"] = eachModelDict["gearbox"]
                    curCarModelDict["carModelName"] = eachModelDict["specname"]
                    curCarModelDict["carModelSpecUrl"] = carModelSpecUrl
                    curCarModelDict["carModelMsrp"] = eachModelDict["price"]

                    self.processSingleResult(curCarModelDict)

    @catch_status_code_error
    def processCarSpecConfig(self, curCarModelDict):
        print("in processCarSpecConfig")
        carModelDict = copy.deepcopy(curCarModelDict)
        print("carModelDict=%s" % carModelDict)
        carModelSpecUrl = carModelDict["carModelSpecUrl"]
        print("carModelSpecUrl=%s" % carModelSpecUrl)
        carModelSpecId = self.extractSpecId(carModelSpecUrl)
        print("carModelSpecId=%s" % carModelSpecId)
        carModelDict["carModelSpecId"] = carModelSpecId # 43593
        carConfigSpecUrl = self.genConfigSpecUrl(carModelSpecId)
        # https://car.autohome.com.cn/config/spec/43593.html
        print("carConfigSpecUrl=%s" % carConfigSpecUrl)

        self.crawl(carConfigSpecUrl,
            # fetch_type="js",
            callback=self.carConfigSpecCallback,
            save=carModelDict,
        )

    def getItemFirstValue(self, inputContent, itemIndex):
        print("in getItemFirstValue")
        # firstItemValue = self.extractTrFirstTdValue(inputContent, itemIndex)
        firstItemValue = self.extractDictListFirstValue(inputContent, itemIndex)
        return firstItemValue

    def extractDictListFirstValue(self, paramItemDictList, itemIndex):
        """
        [
            ...,
            {
                "id": 1149,
                "name": "能源类型",
                "pnid": "1_-1",
                "valueitems": [{
                    "specid": 39893,
                    "value": "纯电动"
                }, {
                    "specid": 42875,
                    "value": "纯电动"
                }]
            }
            ...,
            {
                "id": 1292,
                "name": "<span class='hs_kw39_configpl'></span><span class='hs_kw40_configpl'></span>(小时)",
                "pnid": "1_-1",
                "valueitems": [{
                    "specid": 39893,
                    "value": "0.6"
                }, {
                    "specid": 42875,
                    "value": "0.6"
                }]
            },
            ...
            ,
            {
                "id": 1255,
                "name": "整车<span class='hs_kw36_configpl'></span>",
                "pnid": "1_-1",
                "valueitems": [{
                    "specid": 39893,
                    "value": "三<span class='hs_kw7_configpl'></span>10<span class='hs_kw1_configpl'></span>公里"
                }, {
                    "specid": 42875,
                    "value": "三<span class='hs_kw7_configpl'></span>10<span class='hs_kw1_configpl'></span>公里"
                }]
            }
        ]
        """
        paramItemDict = paramItemDictList[itemIndex]
        print("paramItemDict=%s" % paramItemDict)
        firstItemValue = self.extractValueItemsValue(paramItemDict)
        print("firstItemValue=%s" % firstItemValue)
        return firstItemValue

    def extractValueItemsValue(self, curItemDict, valueIndex=0):
        """
        {
            "id": 0,
            "name": "上市<span class='hs_kw51_configrI'></span>",
            "pnid": "1_-1",
            "valueitems": [{
                "specid": 1006466,
                "value": "2018.05"
            }, {
                "specid": 1006465,
                "value": "2018.05"
            }, {
                "specid": 1006467,
                "value": "2018.05"
            }]
        }
        """
        valueItemList = curItemDict["valueitems"]
        print("valueItemList=%s" % valueItemList)
        # firstItemDict = valueItemList[0]
        specificItemDict = valueItemList[valueIndex]
        print("specificItemDict=%s" % specificItemDict)
        # specificItemDict={'specid': 43593, 'value': "<span class='hs_kw57_configxt'></span>-<span class='hs_kw21_configxt'></span><span class='hs_kw24_configxt'></span>"}
        specificItemValue = specificItemDict["value"]
        # specificItemValue=<span class='hs_kw57_configxt'></span>-<span class='hs_kw21_configxt'></span><span class='hs_kw24_configxt'></span>
        print("specificItemValue=%s" % specificItemValue)
        return specificItemValue

    # def extractTrFirstTdValue(self, rootDoc, trNumber, isRespDoc=False):
    def extractTrFirstTdValue(self, rootDoc, trNumber):
        """
        <tr data-pnid="1_-1" id="tr_2">
            <th>
                <div id="1149"><a href="https://car.autohome.com.cn/baike/detail_7_18_1149.html#pvareaid=2042252">能源类型</a>
                </div>
            </th>
            <td style="background:#F0F3F8;">
                <div>纯电动</div>
            </td>

        <tr data-pnid="1_-1" id="tr_3">
            <th>
                <div id="0">上市<span class="hs_kw40_configxv"></span></div>
            </th>
            <td style="background:#F0F3F8;">
                <div>2019.11</div>
            </td>
            <td>
                <div>2019.11</div>
            </td>
            <td>
                <div></div>
            </td>
            <td>
                <div></div>
            </td>
        </tr>

        <tr data-pnid="1_-1" id="tr_20" style="background: rgb(255, 255, 255);">
            <th>
                <div id="1255"><a href="https://car.autohome.com.cn/baike/detail_7_18_1255.html#pvareaid=2042252">整车<span
                    class="hs_kw36_configaJ"></span></a></div>
            </th>
            <td style="background:#F0F3F8;">
                <div>三<span class="hs_kw7_configaJ"></span>10<span class="hs_kw1_configaJ"></span>公里</div>
            </td>
            <td>
                <div>三<span class="hs_kw7_configaJ"></span>10<span class="hs_kw1_configaJ"></span>公里</div>
            </td>
            <td>
                <div></div>
            </td>
            <td>
                <div></div>
            </td>
        </tr>
        """
        trQuery = "tr[id='tr_%s']" % trNumber
        # print("trQuery=%s" % trQuery)
        trDoc = rootDoc.find(trQuery)
        # print("trDoc=%s" % trDoc)
        tdDocGenerator = trDoc.items("td")
        # print("tdDocGenerator=%s" % tdDocGenerator)
        tdDocList = list(tdDocGenerator)
        # print("tdDocList=%s" % tdDocList)
        firstTdDoc = tdDocList[0]
        # print("firstTdDoc=%s" % firstTdDoc)
        firstTdDivDoc = firstTdDoc.find("div")
        print("firstTdDivDoc=%s" % firstTdDivDoc)
        # if isRespDoc:
        #     respItem = firstTdDivDoc
        # else:
        #     firstItemValue = firstTdDivDoc.text()
        #     respItem = firstItemValue
        # print("respItem=%s" % respItem)
        # return respItem
        respItemHtml = firstTdDivDoc.html()
        print("respItemHtml=%s" % respItemHtml)
        return respItemHtml

    # def extractWholeWarranty(self, firstDivDoc):
    def extractWholeWarranty(self, firstDivHtml):
        print("in extractWholeWarranty")
        carModelWholeWarranty = ""
        # <div>三<span class="hs_kw7_configxv"></span>10<span class="hs_kw1_configxv"></span>公里</div>
        # print("firstDivDoc=%s" % firstDivDoc)
        # carModelWholeWarranty = firstDivDoc.text() # 三10公里
        # firstDivHtml = firstDivDoc.html()
        print("firstDivHtml=%s" % firstDivHtml)
        # 三<span class="hs_kw7_configCC"></span>10<span class="hs_kw1_configCC"></span>公里
        # carWholeQualityQuarantee = re.sub("[^<>]+(?P<firstSpan><span.+?></span>)[^<>]+(?P<secondSpan><span.+?></span>)[^<>]+", )
        foundYearDistance = re.search("(?P<warrantyYear>[^<>]+)<span.+?></span>(?P<distanceNumber>[^<>]+)<span.+?></span>(?P<distanceUnit>[^<>]+)", firstDivHtml)
        if foundYearDistance:
            warrantyYear = foundYearDistance.group("warrantyYear")
            distanceNumber = foundYearDistance.group("distanceNumber")
            distanceUnit = foundYearDistance.group("distanceUnit")
            carModelWholeWarranty = "%s年或%s万%s" % (warrantyYear, distanceNumber, distanceUnit)
        else:
            # special:
            # https://car.autohome.com.cn/config/spec/46700.html
            # <div>三<span class="hs_kw58_configWh"></span></div>
            # 三<span class="hs_kw58_configOf"></span>
            foundYearNotLimitDistance = re.search("(?P<warrantyYear>[^<>]+)<span.+?></span>", firstDivHtml)
            print("foundYearNotLimitDistance=%s" % foundYearNotLimitDistance)
            if foundYearNotLimitDistance:
                warrantyYear = foundYearNotLimitDistance.group("warrantyYear")
                print("warrantyYear=%s" % warrantyYear)
                carModelWholeWarranty = "%s年不限公里" % warrantyYear
        print("carModelWholeWarranty=%s" % carModelWholeWarranty)
        return carModelWholeWarranty

    def getWholeWarranty(self, inputContent, itemIndex):
        # firstDivDoc = self.getItemFirstValue(inputContent, itemIndex, isRespDoc=True)
        # print("firstDivDoc=%s" % firstDivDoc)
        # carModelWholeWarranty = self.extractWholeWarranty(firstDivDoc)
        firstDivDocHtml = self.getItemFirstValue(inputContent, itemIndex)
        print("firstDivDocHtml=%s" % firstDivDocHtml)
        carModelWholeWarranty = self.extractWholeWarranty(firstDivDocHtml)
        return carModelWholeWarranty

    @catch_status_code_error
    def carConfigSpecCallback(self, response):
        print("in carConfigSpecCallback")
        curCarModelDict = response.save
        print("curCarModelDict=%s" % curCarModelDict)
        carModelDict = copy.deepcopy(curCarModelDict)

        configSpecHtml = response.text
        print("configSpecHtml=%s" % configSpecHtml)
        print("")

        # for debug
        return

        # # config json item index - spec table html item index = 2
        # ItemIndexDiff = 2

        # isUseSpecTableHtml = True
        # isUseConfigJson = False
        # valueContent = None
        # energyTypeIdx = 2

        # # Method 1: after run js, extract item value from spec table html
        # """
        # <table class="tbcs" id="tab_0" style="width: 932px;">
        #     <tbody>
        #         <tr>
        #             <th class="cstitle" show="1" pid="tab_0" id="nav_meto_0" colspan="5">
        #             <h3><span>基本参数</span></h3>
        #             </th>
        #         </tr>
        #         <tr data-pnid="1_-1" id="tr_0">
        # """
        # tbodyDoc = response.doc("table[id='tab_0'] tbody")
        # print("tbodyDoc=%s" % tbodyDoc)
        # valueContent = tbodyDoc
        # isUseSpecTableHtml = True
        # isUseConfigJson = False
        # energyTypeIdx = 2

        # Method 2: not run js, extract item value from config json
        # get value from config json
        # var config = {"message" ...... "returncode":"0","taskid":"8be676a3-e023-4fa9-826d-09cd42a1810c","time":"2020-08-27 20:56:17"};
        foundConfigJson = re.search("var\s*config\s*=\s*(?P<configJson>\{[^;]+\});", configSpecHtml)
        print("foundConfigJson=%s" % foundConfigJson)
        if foundConfigJson:
            configJson = foundConfigJson.group("configJson")
            print("configJson=%s" % configJson)
            # configDict = json.loads(configJson, encoding="utf-8")
            configDict = json.loads(configJson)
            print("configDict=%s" % configDict)

            # if "result" in configDict:
            configResultDict = configDict["result"]
            print("configResultDict=%s" % configResultDict)
            # if "paramtypeitems" in configResultDict:
            paramTypeItemDictList = configResultDict["paramtypeitems"]
            print("paramTypeItemDictList=%s" % paramTypeItemDictList)
            # paramTypeItemNum = len(paramTypeItemDictList)
            # print("paramTypeItemNum=%s" % paramTypeItemNum)
            basicParamDict = paramTypeItemDictList[0]
            print("basicParamDict=%s" % basicParamDict)
            basicItemDictList = basicParamDict["paramitems"]
            print("basicItemDictList=%s" % basicItemDictList)
            # print("type(basicItemDictList)=%s" % type(basicItemDictList))
            # basicItemNum = len(basicItemDictList)
            # print("basicItemNum=%s" % basicItemNum)

            # valueContent = basicItemDictList
            # isUseSpecTableHtml = False
            # isUseConfigJson = True

            # process each basic parameter
            basicItemDictLen = len(basicItemDictList)
            print("basicItemDictLen=%s" % basicItemDictLen)
            for curIdx, eachItemDict in enumerate(basicItemDictList):
                print("[%d] eachItemDict=%s" % (curIdx, eachItemDict))
                curItemId = eachItemDict["id"]
                print("curItemId=%s" % curItemId)
                curItemName = eachItemDict["name"]
                print("curItemName=%s" % curItemName)
                curItemFirstValue = self.extractValueItemsValue(eachItemDict)
                print("curItemFirstValue=%s" % curItemFirstValue)

                curIdNameKeyMapDict = None
                if curItemId != 0:
                    curIdNameKeyMapDict = self.findMappingDict(curItemId)
                else:
                    # id = 0
                    foundSpan = re.search("<span", curItemName)
                    print("foundSpan=%s" % foundSpan)
                    isSpecialName = bool(foundSpan)
                    print("isSpecialName=%s" % isSpecialName)
                    if isSpecialName:
                        # id=0 and contain '<span' special name
                        foundSuffixHour = re.search("</span>\(小时\)$", curItemName)
                        print("foundSuffixHour=%s" % foundSuffixHour)
                        isSpecialSuffixHour = bool(foundSuffixHour)
                        print("isSpecialSuffixHour=%s" % isSpecialSuffixHour)
                        if isSpecialSuffixHour:
                            prevIsQuickCharge = self.isPrevItemIsQuickCharge(curIdx, basicItemDictList)
                            print("prevIsQuickCharge=%s" % prevIsQuickCharge)
                            if prevIsQuickCharge:
                                # current is MUST 慢充时间(小时)
                                curIdNameKeyMapDict = {
                                    "id": 0,
                                    # "name": "<span class='hs_kw10_configpl'></span><span class='hs_kw40_configpl'></span>(小时)",
                                    "name": "慢充时间(小时)",
                                    "namePattern": "</span>\(小时\)$",
                                    "key": "carModelSlowCharge",
                                }

                            if not curIdNameKeyMapDict:
                                prevIsActualTestEnduranceMileage = self.isPrevItemIsActualTestEnduranceMileage(curIdx, basicItemDictList)
                                print("prevIsActualTestEnduranceMileage=%s" % prevIsActualTestEnduranceMileage)
                                if prevIsActualTestEnduranceMileage:
                                    # current is MUST 实测快充时间(小时)
                                    curIdNameKeyMapDict = {
                                        "id": 0,
                                        # "name": "<span class='hs_kw22_configpl'></span><span class='hs_kw39_configpl'></span><span class='hs_kw40_configpl'></span>(小时)",
                                        "name": "实测快充时间(小时)",
                                        "namePattern": "</span>\(小时\)$",
                                        "key": "carModelActualTestQuickCharge",
                                    }

                            if not curIdNameKeyMapDict:
                                prevPrevIsActualTestEnduranceMileage = self.isPrevPrevItemIsActualTestEnduranceMileage(curIdx, basicItemDictList)
                                print("prevPrevIsActualTestEnduranceMileage=%s" % prevPrevIsActualTestEnduranceMileage)
                                if prevPrevIsActualTestEnduranceMileage:
                                    # current is MUST 实测慢充时间(小时)
                                    curIdNameKeyMapDict = {
                                        "id": 0,
                                        # "name": "<span class='hs_kw22_configpl'></span><span class='hs_kw10_configpl'></span><span class='hs_kw40_configpl'></span>(小时)",
                                        "name": "实测慢充时间(小时)",
                                        "namePattern": "</span>\(小时\)$",
                                        "key": "carModelActualTestSlowCharge",
                                    }
                        else:
                            curIdNameKeyMapDict = self.findMappingDict(0, curItemName)
                    else:
                        curIdNameKeyMapDict = self.findMappingDict(0, curItemName)

                print("curIdNameKeyMapDict=%s" % curIdNameKeyMapDict)
                if curIdNameKeyMapDict:
                    curItemKey = curIdNameKeyMapDict["key"]
                    print("curItemKey=%s" % curItemKey)
                    # processedItemValue = self.processSpecialKeyValue(curItemKey, curItemFirstValue, configSpecHtml)
                    processedItemValue = self.processSpecialKeyValue(curItemKey, curItemFirstValue, response)
                    print("processedItemValue=%s" % processedItemValue)
                    carModelDict[curItemKey] = processedItemValue
                    print("+++ added %s=%s" % (curItemKey, processedItemValue))

            print("after extract all item value: carModelDict=%s" % carModelDict)

        self.saveSingleResult(carModelDict)

        # if isUseConfigJson:
        #     energyTypeIdx += ItemIndexDiff

        # if valueContent:
        #     self.processDiffEneryTypeCar(carModelDict, valueContent, energyTypeIdx, isUseConfigJson, ItemIndexDiff)
        # else:
        #     self.saveSingleResult(carModelDict)

    # def processSpecialKeyValue(self, itemKey, itemValue, curHtml):
    def processSpecialKeyValue(self, itemKey, itemValue, response):
        print("in processSpecialKeyValue")
        print("itemKey=%s, itemValue=%s" % (itemKey, itemValue))
        if itemKey == "carModelWholeWarranty":
            print("process special carModelWholeWarranty")
            # 整车质保
            # 三<span class='hs_kw5_configJS'></span>10<span class='hs_kw0_configJS'></span>公里
            itemValue = self.extractWholeWarranty(itemValue)
            print("itemValue=%s" % itemValue)
        elif itemKey == "carModelBodyStructure":
            print("process special carModelBodyStructure value")
            # (1) https://www.autohome.com.cn/spec/46292/#pvareaid=3454492
            # 5门7座<span class='hs_kw3_configFS'></span>
            # -> 5门7座MPV
            #
            # (2) https://www.autohome.com.cn/spec/1002900/
            # <span class='hs_kw21_configqk'></span>
            # -> 皮卡
            foundSpan = re.search("(?P<bodySpan><span.+?</span>)", itemValue)
            print("foundSpan=%s" % foundSpan)
            if foundSpan:
                bodySpan = foundSpan.group("bodySpan")
                print("bodySpan=%s" % bodySpan)
                # extract body structure
                """
                <div class="filtrate-list filtrate-list-col2">
                    <span class="title">车身结构：</span>
                    <label class="lbTxt" for="PL2$!{1 - 1}">
                        <input type="checkbox" class="selectTr_input" id="PL2$!{1 - 1}" value="MPV" name="carStruct">
                        MPV
                    </label>
                </div>
                """
                # soup = BeautifulSoup(curHtml, "html.parser")
                # # print("soup=%s" % soup)
                # # bodyStructureSpanSoup = soup.find(text="车身结构：", attrs={"class":"title"})
                # # # print("bodyStructureSpanSoup=%s" % bodyStructureSpanSoup)
                # # # emptySoup = bodyStructureSpanSoup.next_sibling
                # # # print("emptySoup=%s" % emptySoup)
                # # # siblingLabelSoup = emptySoup.next_sibling
                # # # print("siblingLabelSoup=%s" % siblingLabelSoup)
                # # parentDivSoup = bodyStructureSpanSoup.parent
                # # print("parentDivSoup=%s" % parentDivSoup)
                # # inputSoup = parentDivSoup.find("input", attrs={"type":"checkbox", "class":"selectTr_input", "name":"carStruct"})
                # carStructSoup = soup.find("input", attrs={"type":"checkbox", "class":"selectTr_input", "name":"carStruct"})
                # print("carStructSoup=%s" % carStructSoup)
                carStructDoc = response.doc("input[name=carStruct]")
                print("carStructDoc=%s" % carStructDoc)
                bodyStructureValue = carStructDoc.attr["value"]
                print("bodyStructureValue=%s" % bodyStructureValue)
                itemValue = itemValue.replace(bodySpan, bodyStructureValue)
                print("itemValue=%s" % itemValue)

        return itemValue

    def isPrevItemIsQuickCharge(self, curIdx, itemDictList):
        print("in isPrevItemIsQuickCharge")
        print("curIdx=%s" % curIdx)

        prevIsQuickCharge = False

        if curIdx > 0:
            prevIdx = curIdx - 1
            print("prevIdx=%s" % prevIdx)
            prevItemDict = itemDictList[prevIdx]
            print("prevItemDict=%s" % prevItemDict)
            prevItemId = prevItemDict["id"]
            print("prevItemId=%s" % prevItemId)
            prevItemName = prevItemDict["name"]
            print("prevItemName=%s" % prevItemName)
            """
                "id": 1292,
                # "name": "<span class='hs_kw39_configpl'></span><span class='hs_kw40_configpl'></span>(小时)",
                "name": "快充时间(小时)",
            """
            QuickChargeItemId = 1292
            if prevItemId == QuickChargeItemId:
                prevIsQuickCharge = True

        print("prevIsQuickCharge=%s" % prevIsQuickCharge)
        return prevIsQuickCharge

    def checkIsActualTestEnduranceMileage(self, prevSomeNum, curIdx, itemDictList):
        print("in checkIsActualTestEnduranceMileage")
        print("prevSomeNum=%s, curIdx=%s" % (prevSomeNum, curIdx))

        isActualTestEnduranceMileage = False

        minAllowIdx = prevSomeNum - 1

        if curIdx > minAllowIdx:
            prevSomeIdx = curIdx - prevSomeNum
            print("prevSomeIdx=%s" % prevSomeIdx)
            prevSomeItemDict = itemDictList[prevSomeIdx]
            print("prevSomeItemDict=%s" % prevSomeItemDict)
            prevSomeItemId = prevSomeItemDict["id"]
            print("prevSomeItemId=%s" % prevSomeItemId)
            prevSomeItemName = prevSomeItemDict["name"]
            print("prevSomeItemName=%s" % prevSomeItemName)

            if prevSomeItemId == 0:
                """
                    "id": 0,
                    # "name": "<span class='hs_kw22_configpl'></span>续航里程(km)",
                    "name": "实测续航里程(km)",
                    "namePattern": "</span>续航里程\(km\)$",
                    "key": "carModelActualTestEnduranceMileage",
                """
                foundActualTestEnduranceMileage = re.search("</span>续航里程\(km\)$", prevSomeItemName)
                print("foundActualTestEnduranceMileage=%s" % foundActualTestEnduranceMileage)
                if foundActualTestEnduranceMileage:
                    isActualTestEnduranceMileage = True

        print("isActualTestEnduranceMileage=%s" % isActualTestEnduranceMileage)
        return isActualTestEnduranceMileage

    def isPrevItemIsActualTestEnduranceMileage(self, curIdx, itemDictList):
        print("in isPrevItemIsActualTestEnduranceMileage")
        print("curIdx=%s" % curIdx)
        return self.checkIsActualTestEnduranceMileage(1, curIdx, itemDictList)

    def isPrevPrevItemIsActualTestEnduranceMileage(self, curIdx, itemDictList):
        print("in isPrevPrevItemIsActualTestEnduranceMileage")
        print("curIdx=%s" % curIdx)
        return self.checkIsActualTestEnduranceMileage(2, curIdx, itemDictList)

    def findMappingDict(self, itemId=0, itemName=""):
        foundMapDict = None

        paramIdNameKeyMapDict = [
            # 汽油车 参数
            # https://car.autohome.com.cn/config/spec/41572.html
            # https://car.autohome.com.cn/config/spec/1006465.html
            {
                "id": 1149,
                "name": "能源类型",
                "key": "carEnergyType",
            }, {
                "id": 1311,
                "name": "环保标准",
                "key": "carModelEnvStandard",
            }, {
                "id": 0,
                # "name": "上市<span class='hs_kw51_configvR'></span>", # 上市时间
                "name": "上市时间",
                "namePattern": "^上市",
                "key": "carModelReleaseTime",
            }, {
                "id": 1185,
                # "name": "<span class='hs_kw40_configvR'></span><span class='hs_kw15_configvR'></span>(kW)",
                "name": "最大功率(kW)",
                "key": "carModelMaxPower",
            }, {
                "id": 1186,
                # "name": "<span class='hs_kw40_configvR'></span><span class='hs_kw61_configvR'></span>(N·m)",
                "name": "最大扭矩(N·m)",
                "key": "carModelMaxTorque",
            }, {
                "id": 1150,
                "name": "发动机",
                "key": "carModelEngine",
            }, {
                "id": 1245,
                "name": "变速箱",
                "key": "carModelGearBox",
            }, {
                "id": 1148,
                "name": "长*宽*高(mm)",
                "key": "carModelSize",
            }, {
                "id": 1147,
                "name": "车身结构",
                "key": "carModelBodyStructure",
            }, {
                "id": 1246,
                "name": "最高车速(km/h)",
                "key": "carModelMaxSpeed",
            }, {
                "id": 1250,
                "name": "官方0-100km/h加速(s)",
                "key": "carModelOfficialSpeedupTime",
            }, {
                "id": 1252,
                # "name": "<span class='hs_kw26_configvR'></span>0-100km/h加速(s)",
                "name": "实测0-100km/h加速(s)",
                "key": "carModelActualTestSpeedupTime",
            }, {
                "id": 1253,
                # "name": "<span class='hs_kw26_configvR'></span>100-0km/h制动(m)",
                "name": "实测100-0km/h制动(m)",
                "key": "carModelActualTestBrakeDistance",
            }, {
                "id": 1251,
                # "name": "工信部<span class='hs_kw10_configvR'></span><span class='hs_kw43_configvR'></span>(L/100km)",
                "name": "工信部综合油耗(L/100km)",
                "key": "carModelMiitCompositeFuelConsumption",
            }, {
                "id": 1254,
                # "name": "<span class='hs_kw26_configvR'></span><span class='hs_kw43_configvR'></span>(L/100km)",
                "name": "实测油耗(L/100km)",
                "key": "carModelActualFuelConsumption",
            }, {
                "id": 1255,
                # "name": "整车<span class='hs_kw73_configvR'></span>",
                "name": "整车质保",
                "key": "carModelWholeWarranty",
            },

            # 电动车 参数
            # https://car.autohome.com.cn/config/spec/39893.html
            # https://car.autohome.com.cn/config/spec/42875.html
            {
                "id": 1291,
                "name": "工信部纯电续航里程(km)",
                "key": "carModelMiitEnduranceMileagePureElectric",
            }, {
                "id": 1292,
                # "name": "<span class='hs_kw39_configpl'></span><span class='hs_kw40_configpl'></span>(小时)",
                "name": "快充时间(小时)",
                "key": "carModelQuickCharge",
            # }, {
            #     "id": 0,
            #     # "name": "<span class='hs_kw10_configpl'></span><span class='hs_kw40_configpl'></span>(小时)",
            #     "name": "慢充时间(小时)",
            #     "namePattern": "</span>\(小时\)$",
            #     "key": "carModelSlowCharge",
            }, {
                "id": 0,
                # https://car.autohome.com.cn/config/spec/39893.html
                # {'id': 0, 'name': "<span class='hs_kw39_configMh'></span><span class='hs_kw11_configMh'></span>百分比", 'pnid': '1_-1', 'valueitems': [{'specid': 39893, 'value': '80'}, {'specid': 42875, 'value': '80'}]}
                "name": "快充电量百分比",
                "namePattern": "</span>百分比$",
                "key": "carModelQuickChargePercent",
            }, {
                "id": 0,
                "name": "电动机(Ps)",
                "key": "carModelHorsePowerElectric",
            }, {
                "id": 0,
                # "name": "<span class='hs_kw22_configpl'></span>续航里程(km)",
                "name": "实测续航里程(km)",
                "namePattern": "</span>续航里程\(km\)$",
                "key": "carModelActualTestEnduranceMileage",
            # }, {
            #     "id": 0,
            #     # "name": "<span class='hs_kw22_configpl'></span><span class='hs_kw39_configpl'></span><span class='hs_kw40_configpl'></span>(小时)",
            #     "name": "实测快充时间(小时)",
            #     "namePattern": "</span>\(小时\)$",
            #     "key": "carModelActualTestQuickCharge",
            # }, {
            #     "id": 0,
            #     # "name": "<span class='hs_kw22_configpl'></span><span class='hs_kw10_configpl'></span><span class='hs_kw40_configpl'></span>(小时)",
            #     "name": "实测慢充时间(小时)",
            #     "namePattern": "</span>\(小时\)$",
            #     "key": "carModelActualTestSlowCharge",
            }
        ]

        isItemZero = itemId == 0
        print("isItemZero=%s" % isItemZero)

        foundSpan = re.search("<span", itemName)
        print("foundSpan=%s" % foundSpan)
        isSpecialName = bool(foundSpan)
        print("isSpecialName=%s" % isSpecialName)
        isNotSpecialName = not isSpecialName
        print("isNotSpecialName=%s" % isNotSpecialName)

        if not isItemZero:
            for eachMapDict in paramIdNameKeyMapDict:
                eachItemId = eachMapDict["id"]
                if eachItemId == itemId:
                    foundMapDict = eachMapDict
                    break

        if not foundMapDict:
            if itemName and isNotSpecialName:
                for eachMapDict in paramIdNameKeyMapDict:
                    eachItemName = eachMapDict["name"]
                    if eachItemName == itemName:
                        foundMapDict = eachMapDict
                        break

        if not foundMapDict:
            if (isItemZero and isSpecialName):
                for eachMapDict in paramIdNameKeyMapDict:
                    if "namePattern" in eachMapDict:
                        eachItemNamePattern = eachMapDict["namePattern"]
                        print("eachItemNamePattern=%s" % eachItemNamePattern)
                        foundMatchName = re.search(eachItemNamePattern, itemName)
                        print("foundMatchName=%s" % foundMatchName)
                        if foundMatchName:
                            foundMapDict = eachMapDict
                            break
        print("foundMapDict=%s from id=%s, name=%s" % (foundMapDict, itemId, itemName))
        return foundMapDict

    def processDiffEneryTypeCar(self, carModelDict, valueContent, energyTypeIdx, isUseConfigJson, ItemIndexDiff):
        carEnergyType = self.getItemFirstValue(valueContent, energyTypeIdx)
        # 纯电动 / 汽油 / 插电式混合动力 / 油电混合
        carModelDict["carEnergyType"] = carEnergyType

        if carEnergyType == "汽油":
            # https://car.autohome.com.cn/config/spec/43593.html
            # https://car.autohome.com.cn/config/spec/41572.html

            # self.processGasolineCar(valueContent, carModelDict)

            # https://car.autohome.com.cn/config/spec/1006466.html

            gasolineCarKeyIdxMapDict = {
                "carModelEnvStandard" : 3,
                "carModelReleaseTime" : 4,
                "carModelMaxPower" : 5,
                "carModelMaxTorque" : 6,
                "carModelEngine" : 7,
                "carModelGearBox" : 8,
                "carModelSize" : 9,
                "carModelBodyStructure" : 10,
                "carModelMaxSpeed" : 11,
                "carModelOfficialSpeedupTime" : 12,
                "carModelActualTestSpeedupTime" : 13,
                "carModelActualTestBrakeDistance" : 14,
                "carModelMiitCompositeFuelConsumption" : 15,
                "carModelActualFuelConsumption" : 16,
            }
            wholeWarrantyIdx = 17

            if isUseConfigJson:
                for eachKey in gasolineCarKeyIdxMapDict.keys():
                    gasolineCarKeyIdxMapDict[eachKey] += ItemIndexDiff
                wholeWarrantyIdx += ItemIndexDiff

            self.processSingleEneryTypeCar(gasolineCarKeyIdxMapDict, valueContent, wholeWarrantyIdx, carModelDict)

        elif carEnergyType == "纯电动":
            # https://car.autohome.com.cn/config/spec/42875.html

            # self.processPureElectricCar(valueContent, carModelDict)

            pureElectricCarKeyIdxMapDict = {
                "carModelReleaseTime": 3,
                "carModelMiitEnduranceMileagePureElectric": 4,
                "carModelQuickCharge": 5,
                "carModelSlowCharge": 6,
                "carModelQuickChargePercent": 7,
                "carModelMaxPower": 8,
                "carModelMaxTorque": 9,
                "carModelHorsePowerElectric": 10,
                "carModelSize": 11,
                "carModelBodyStructure": 12,
                "carModelMaxSpeed": 13,
                "carModelOfficialSpeedupTime": 14,
                "carModelActualTestSpeedupTime": 15,
                "carModelActualTestBrakeDistance": 16,
                "carModelActualTestEnduranceMileage": 17,
                "carModelActualTestQuickCharge": 18,
                "carModelActualTestSlowCharge": 19,
            }
            wholeWarrantyIdx = 20

            if isUseConfigJson:
                for eachKey in pureElectricCarKeyIdxMapDict.keys():
                    pureElectricCarKeyIdxMapDict[eachKey] += ItemIndexDiff
                wholeWarrantyIdx += ItemIndexDiff

            self.processSingleEneryTypeCar(pureElectricCarKeyIdxMapDict, valueContent, wholeWarrantyIdx, carModelDict)

        elif carEnergyType == "插电式混合动力":
            # https://car.autohome.com.cn/config/series/4460.html

            # self.processPhevCar(valueContent, carModelDict)

            phevCarKeyIdxMapDict = {
                "carModelEnvStandard": 3,
                "carModelReleaseTime": 4,
                "carModelMiitEnduranceMileagePureElectric": 5,
                "carModelQuickCharge": 6,
                "carModelSlowCharge": 7,
                "carModelQuickChargePercent": 8,
                "carModelMaxPower": 9,
                "carModelMaxTorque": 10,
                "carModelEngine": 11,
                "carModelHorsePowerElectric": 12,
                "carModelGearBox": 13,
                "carModelSize": 14,
                "carModelBodyStructure": 15,
                "carModelMaxSpeed": 16,
                "carModelOfficialSpeedupTime": 17,
                "carModelActualTestSpeedupTime": 18,
                "carModelActualTestBrakeDistance": 19,
                "carModelActualTestEnduranceMileage": 20,
                "carModelActualTestQuickCharge": 21,
                "carModelActualTestSlowCharge": 22,
                "carModelMiitCompositeFuelConsumption": 23,
                "carModelActualFuelConsumption": 24,
            }
            wholeWarrantyIdx = 25

            if isUseConfigJson:
                for eachKey in phevCarKeyIdxMapDict.keys():
                    phevCarKeyIdxMapDict[eachKey] += ItemIndexDiff
                wholeWarrantyIdx += ItemIndexDiff

            self.processSingleEneryTypeCar(phevCarKeyIdxMapDict, valueContent, wholeWarrantyIdx, carModelDict)

        elif carEnergyType == "油电混合":
            # https://car.autohome.com.cn/config/spec/35507.html

            # self.processHevCar(valueContent, carModelDict)

            hevCarKeyIdxMapDict = {
                "carModelEnvStandard": 3,
                "carModelReleaseTime": 4,
                "carModelMaxPower": 5,
                "carModelMaxTorque": 6,
                "carModelEngine": 7,
                "carModelHorsePowerElectric": 8,
                "carModelGearBox": 9,
                "carModelSize": 10,
                "carModelBodyStructure": 11,
                "carModelMaxSpeed": 12,
                "carModelOfficialSpeedupTime": 13,
                "carModelActualTestSpeedupTime": 14,
                "carModelActualTestBrakeDistance": 15,
                "carModelMiitCompositeFuelConsumption": 16,
                "carModelActualFuelConsumption": 17,
            }
            wholeWarrantyIdx = 18

            if isUseConfigJson:
                for eachKey in hevCarKeyIdxMapDict.keys():
                    hevCarKeyIdxMapDict[eachKey] += ItemIndexDiff
                wholeWarrantyIdx += ItemIndexDiff

            self.processSingleEneryTypeCar(hevCarKeyIdxMapDict, valueContent, wholeWarrantyIdx, carModelDict)
        else:
            errMsg = "TODO: add support %s!" % carEnergyType
            raise Exception(errMsg)

    def processSingleEneryTypeCar(self, keyIdxMapDict, valueContent, wholeWarrantyIdx, carModelDict):
        keyList = keyIdxMapDict.keys()
        keyListLen = len(keyList)
        print("keyListLen=%s" % keyListLen)
        for eachItemKey in keyList:
            print("eachItemKey=%s" % eachItemKey)
            eachItemIndex = keyIdxMapDict[eachItemKey]
            print("eachItemIndex=%s" % eachItemIndex)
            eachItemValue = self.getItemFirstValue(valueContent, eachItemIndex)
            print("eachItemValue=%s" % eachItemValue)
            carModelDict[eachItemKey] = eachItemValue

        # 整车质保
        carModelWholeWarranty = self.getWholeWarranty(valueContent, wholeWarrantyIdx) # 三年或10万公里
        print("carModelWholeWarranty=%s" % carModelWholeWarranty)
        carModelDict["carModelWholeWarranty"] = carModelWholeWarranty

        self.saveSingleResult(carModelDict)

    # def processGasolineCar(self, valueContent, carModelDict):
    #     # 汽油

    #     # https://car.autohome.com.cn/config/spec/43593.html
    #     # https://car.autohome.com.cn/config/spec/41572.html

    #     # 环保标准
    #     carModelEnvStandard = self.getItemFirstValue(valueContent, 3) # 国VI
    #     carModelDict["carModelEnvStandard"] = carModelEnvStandard

    #     # 上市时间
    #     carModelReleaseTime = self.getItemFirstValue(valueContent, 4) # 2020.04
    #     carModelDict["carModelReleaseTime"] = carModelReleaseTime

    #     # 最大功率(kW)
    #     carModelMaxPower = self.getItemFirstValue(valueContent, 5) # 110
    #     carModelDict["carModelMaxPower"] = carModelMaxPower

    #     # 最大扭矩(N·m)
    #     carModelMaxTorque = self.getItemFirstValue(valueContent, 6) # 250
    #     carModelDict["carModelMaxTorque"] = carModelMaxTorque

    #     # 发动机
    #     carModelEngine = self.getItemFirstValue(valueContent, 7) # 1.4T 150马力 L4
    #     carModelDict["carModelEngine"] = carModelEngine

    #     # 变速箱
    #     carModelGearBox = self.getItemFirstValue(valueContent, 8) # 7挡双离合
    #     carModelDict["carModelGearBox"] = carModelGearBox

    #     # 长*宽*高(mm)
    #     carModelSize = self.getItemFirstValue(valueContent, 9) # 4312*1785*1426
    #     carModelDict["carModelSize"] = carModelSize

    #     # 车身结构
    #     carModelBodyStructure = self.getItemFirstValue(valueContent, 10) # 5门5座两厢车
    #     carModelDict["carModelBodyStructure"] = carModelBodyStructure

    #     # 最高车速(km/h)
    #     carModelMaxSpeed = self.getItemFirstValue(valueContent, 11) # 200
    #     carModelDict["carModelMaxSpeed"] = carModelMaxSpeed

    #     # 官方0-100km/h加速(s)
    #     carModelOfficialSpeedupTime = self.getItemFirstValue(valueContent, 12) # 8.4
    #     carModelDict["carModelOfficialSpeedupTime"] = carModelOfficialSpeedupTime

    #     # 实测0-100km/h加速(s)
    #     carModelActualTestSpeedupTime = self.getItemFirstValue(valueContent, 13) # -
    #     carModelDict["carModelActualTestSpeedupTime"] = carModelActualTestSpeedupTime

    #     # 实测100-0km/h制动(m)
    #     carModelActualTestBrakeDistance = self.getItemFirstValue(valueContent, 14) # -
    #     carModelDict["carModelActualTestBrakeDistance"] = carModelActualTestBrakeDistance

    #     # 工信部综合油耗(L/100km)
    #     carModelMiitCompositeFuelConsumption = self.getItemFirstValue(valueContent, 15) # 5.8
    #     carModelDict["carModelMiitCompositeFuelConsumption"] = carModelMiitCompositeFuelConsumption

    #     # 实测油耗(L/100km)
    #     carModelActualFuelConsumption = self.getItemFirstValue(valueContent, 16) # -
    #     carModelDict["carModelActualFuelConsumption"] = carModelActualFuelConsumption

    #     self.saveSingleResult(carModelDict)

    # def processPureElectricCar(self, valueContent, carModelDict):
    #     # 纯电动

    #     # https://car.autohome.com.cn/config/spec/42875.html

    #     # 上市时间
    #     carModelReleaseTime = self.getItemFirstValue(valueContent, 3) # 2019.11
    #     carModelDict["carModelReleaseTime"] = carModelReleaseTime

    #     # 工信部纯电续航里程(km)
    #     carModelMiitEnduranceMileagePureElectric = self.getItemFirstValue(valueContent, 4) # 265
    #     carModelDict["carModelMiitEnduranceMileagePureElectric"] = carModelMiitEnduranceMileagePureElectric

    #     # 快充时间(小时)
    #     carModelQuickCharge = self.getItemFirstValue(valueContent, 5) # 0.6
    #     carModelDict["carModelQuickCharge"] = carModelQuickCharge

    #     # 慢充时间(小时)
    #     carModelSlowCharge = self.getItemFirstValue(valueContent, 6) # 17
    #     carModelDict["carModelSlowCharge"] = carModelSlowCharge

    #     # 快充电量百分比
    #     carModelQuickChargePercent = self.getItemFirstValue(valueContent, 7) # 80
    #     carModelDict["carModelQuickChargePercent"] = carModelQuickChargePercent

    #     # 最大功率(kW)
    #     carModelMaxPower = self.getItemFirstValue(valueContent, 8) # 100
    #     carModelDict["carModelMaxPower"] = carModelMaxPower

    #     # 最大扭矩(N·m)
    #     carModelMaxTorque = self.getItemFirstValue(valueContent, 9) # 290
    #     carModelDict["carModelMaxTorque"] = carModelMaxTorque

    #     # 电动机(Ps)
    #     carModelHorsePowerElectric = self.getItemFirstValue(valueContent, 10) # 136
    #     carModelDict["carModelHorsePowerElectric"] = carModelHorsePowerElectric

    #     # 长*宽*高(mm)
    #     carModelSize = self.getItemFirstValue(valueContent, 11) # 4237*1785*1548
    #     carModelDict["carModelSize"] = carModelSize

    #     # 车身结构
    #     carModelBodyStructure = self.getItemFirstValue(valueContent, 12) # 5门5座SUV
    #     carModelDict["carModelBodyStructure"] = carModelBodyStructure

    #     # 最高车速(km/h)
    #     carModelMaxSpeed = self.getItemFirstValue(valueContent, 13) # 150
    #     carModelDict["carModelMaxSpeed"] = carModelMaxSpeed

    #     # 官方0-100km/h加速(s)
    #     carModelOfficialSpeedupTime = self.getItemFirstValue(valueContent, 14) # -
    #     carModelDict["carModelOfficialSpeedupTime"] = carModelOfficialSpeedupTime

    #     # 实测0-100km/h加速(s)
    #     carModelActualTestSpeedupTime = self.getItemFirstValue(valueContent, 15) # -
    #     carModelDict["carModelActualTestSpeedupTime"] = carModelActualTestSpeedupTime

    #     # 实测100-0km/h制动(m)
    #     carModelActualTestBrakeDistance = self.getItemFirstValue(valueContent, 16) # -
    #     carModelDict["carModelActualTestBrakeDistance"] = carModelActualTestBrakeDistance

    #     # 实测续航里程(km)
    #     carModelActualTestEnduranceMileage = self.getItemFirstValue(valueContent, 17) # -
    #     carModelDict["carModelActualTestEnduranceMileage"] = carModelActualTestEnduranceMileage

    #     # 实测快充时间(小时)
    #     carModelActualTestQuickCharge = self.getItemFirstValue(valueContent, 18) # -
    #     carModelDict["carModelActualTestQuickCharge"] = carModelActualTestQuickCharge

    #     # 实测慢充时间(小时)
    #     carModelActualTestSlowCharge = self.getItemFirstValue(valueContent, 19) # -
    #     carModelDict["carModelActualTestSlowCharge"] = carModelActualTestSlowCharge

    #     # 整车质保
    #     carModelWholeWarranty = self.getWholeWarranty(valueContent, 20) # 三年或10万公里
    #     carModelDict["carModelWholeWarranty"] = carModelWholeWarranty

    #     self.saveSingleResult(carModelDict)

    # def processPhevCar(self, valueContent, carModelDict):
    #     # 插电式混合动力 = PHEV = Plug-in Hybrid Electric vehicle

    #     # https://car.autohome.com.cn/config/series/4460.html

    #     # 环保标准
    #     carModelEnvStandard = self.getItemFirstValue(valueContent, 3) # 国V
    #     carModelDict["carModelEnvStandard"] = carModelEnvStandard

    #     # 上市时间
    #     carModelReleaseTime = self.getItemFirstValue(valueContent, 4) # 2018.11
    #     carModelDict["carModelReleaseTime"] = carModelReleaseTime

    #     # 工信部纯电续航里程(km)
    #     carModelMiitEnduranceMileagePureElectric = self.getItemFirstValue(valueContent, 5) # 56
    #     carModelDict["carModelMiitEnduranceMileagePureElectric"] = carModelMiitEnduranceMileagePureElectric

    #     # 快充时间(小时)
    #     carModelQuickCharge = self.getItemFirstValue(valueContent, 6) # 2.5
    #     carModelDict["carModelQuickCharge"] = carModelQuickCharge

    #     # 慢充时间(小时)
    #     carModelSlowCharge = self.getItemFirstValue(valueContent, 7) # 10.8
    #     carModelDict["carModelSlowCharge"] = carModelSlowCharge

    #     # 快充电量百分比
    #     carModelQuickChargePercent = self.getItemFirstValue(valueContent, 8) # -
    #     carModelDict["carModelQuickChargePercent"] = carModelQuickChargePercent

    #     # 最大功率(kW)
    #     carModelMaxPower = self.getItemFirstValue(valueContent, 9) # 270
    #     carModelDict["carModelMaxPower"] = carModelMaxPower

    #     # 最大扭矩(N·m)
    #     carModelMaxTorque = self.getItemFirstValue(valueContent, 10) # 700
    #     carModelDict["carModelMaxTorque"] = carModelMaxTorque

    #     # 发动机
    #     carModelEngine = self.getItemFirstValue(valueContent, 11) # 2.0T 252马力 L4
    #     carModelDict["carModelEngine"] = carModelEngine

    #     # 电动机(Ps)
    #     carModelHorsePowerElectric = self.getItemFirstValue(valueContent, 12) # 128
    #     carModelDict["carModelHorsePowerElectric"] = carModelHorsePowerElectric

    #     # 变速箱
    #     carModelGearBox = self.getItemFirstValue(valueContent, 13) # 8挡手自一体
    #     carModelDict["carModelGearBox"] = carModelGearBox

    #     # 长*宽*高(mm)
    #     carModelSize = self.getItemFirstValue(valueContent, 14) # 5071*1968*1716
    #     carModelDict["carModelSize"] = carModelSize

    #     # 车身结构
    #     carModelBodyStructure = self.getItemFirstValue(valueContent, 15) # 5门5座SUV
    #     carModelDict["carModelBodyStructure"] = carModelBodyStructure

    #     # 最高车速(km/h)
    #     carModelMaxSpeed = self.getItemFirstValue(valueContent, 16) # 228
    #     carModelDict["carModelMaxSpeed"] = carModelMaxSpeed

    #     # 官方0-100km/h加速(s)
    #     carModelOfficialSpeedupTime = self.getItemFirstValue(valueContent, 17) # 5.9
    #     carModelDict["carModelOfficialSpeedupTime"] = carModelOfficialSpeedupTime

    #     # 实测0-100km/h加速(s)
    #     carModelActualTestSpeedupTime = self.getItemFirstValue(valueContent, 18) # -
    #     carModelDict["carModelActualTestSpeedupTime"] = carModelActualTestSpeedupTime

    #     # 实测100-0km/h制动(m)
    #     carModelActualTestBrakeDistance = self.getItemFirstValue(valueContent, 19) # -
    #     carModelDict["carModelActualTestBrakeDistance"] = carModelActualTestBrakeDistance

    #     # 实测续航里程(km)
    #     carModelActualTestEnduranceMileage = self.getItemFirstValue(valueContent, 20) # -
    #     carModelDict["carModelActualTestEnduranceMileage"] = carModelActualTestEnduranceMileage

    #     # 实测快充时间(小时)
    #     carModelActualTestQuickCharge = self.getItemFirstValue(valueContent, 21) # -
    #     carModelDict["carModelActualTestQuickCharge"] = carModelActualTestQuickCharge

    #     # 实测慢充时间(小时)
    #     carModelActualTestSlowCharge = self.getItemFirstValue(valueContent, 22) # -
    #     carModelDict["carModelActualTestSlowCharge"] = carModelActualTestSlowCharge

    #     # 工信部综合油耗(L/100km)
    #     carModelMiitCompositeFuelConsumption = self.getItemFirstValue(valueContent, 23) # 2.4
    #     carModelDict["carModelMiitCompositeFuelConsumption"] = carModelMiitCompositeFuelConsumption

    #     # 实测油耗(L/100km)
    #     carModelActualFuelConsumption = self.getItemFirstValue(valueContent, 24) # -
    #     carModelDict["carModelActualFuelConsumption"] = carModelActualFuelConsumption

    #     # 整车质保
    #     carModelWholeWarranty = self.getWholeWarranty(valueContent, 25) # 三年或10万公里
    #     carModelDict["carModelWholeWarranty"] = carModelWholeWarranty

    #     self.saveSingleResult(carModelDict)

    # def processHevCar(self, valueContent, carModelDict):
    #     # 混合电动汽车=HEV=Hybrid Electric Vehicle

    #     # https://car.autohome.com.cn/config/spec/35507.html

    #     # 环保标准
    #     carModelEnvStandard = self.getItemFirstValue(valueContent, 3) # 国IV(国V)
    #     carModelDict["carModelEnvStandard"] = carModelEnvStandard

    #     # 上市时间
    #     carModelReleaseTime = self.getItemFirstValue(valueContent, 4) # 2018.08
    #     carModelDict["carModelReleaseTime"] = carModelReleaseTime

    #     # 最大功率(kW)
    #     carModelMaxPower = self.getItemFirstValue(valueContent, 5) # 100
    #     carModelDict["carModelMaxPower"] = carModelMaxPower

    #     # 最大扭矩(N·m)
    #     carModelMaxTorque = self.getItemFirstValue(valueContent, 6) # -
    #     carModelDict["carModelMaxTorque"] = carModelMaxTorque

    #     # 发动机
    #     carModelEngine = self.getItemFirstValue(valueContent, 7) # 1.8L 99马力 L4
    #     carModelDict["carModelEngine"] = carModelEngine

    #     # 电动机(Ps)
    #     carModelHorsePowerElectric = self.getItemFirstValue(valueContent, 8) # 82
    #     carModelDict["carModelHorsePowerElectric"] = carModelHorsePowerElectric

    #     # 变速箱
    #     carModelGearBox = self.getItemFirstValue(valueContent, 9) # E-CVT无级变速
    #     carModelDict["carModelGearBox"] = carModelGearBox

    #     # 长*宽*高(mm)
    #     carModelSize = self.getItemFirstValue(valueContent, 10) # 4360*1765*1455
    #     carModelDict["carModelSize"] = carModelSize

    #     # 车身结构
    #     carModelBodyStructure = self.getItemFirstValue(valueContent, 11) # 5门5座SUV
    #     carModelDict["carModelBodyStructure"] = carModelBodyStructure

    #     # 最高车速(km/h)
    #     carModelMaxSpeed = self.getItemFirstValue(valueContent, 12) # -
    #     carModelDict["carModelMaxSpeed"] = carModelMaxSpeed

    #     # 官方0-100km/h加速(s)
    #     carModelOfficialSpeedupTime = self.getItemFirstValue(valueContent, 13) # -
    #     carModelDict["carModelOfficialSpeedupTime"] = carModelOfficialSpeedupTime

    #     # 实测0-100km/h加速(s)
    #     carModelActualTestSpeedupTime = self.getItemFirstValue(valueContent, 14) # -
    #     carModelDict["carModelActualTestSpeedupTime"] = carModelActualTestSpeedupTime

    #     # 实测100-0km/h制动(m)
    #     carModelActualTestBrakeDistance = self.getItemFirstValue(valueContent, 15) # -
    #     carModelDict["carModelActualTestBrakeDistance"] = carModelActualTestBrakeDistance

    #     # 工信部综合油耗(L/100km)
    #     carModelMiitCompositeFuelConsumption = self.getItemFirstValue(valueContent, 16) # 4.6
    #     carModelDict["carModelMiitCompositeFuelConsumption"] = carModelMiitCompositeFuelConsumption

    #     # 实测油耗(L/100km)
    #     carModelActualFuelConsumption = self.getItemFirstValue(valueContent, 17) # -
    #     carModelDict["carModelActualFuelConsumption"] = carModelActualFuelConsumption

    #     # 整车质保
    #     carModelWholeWarranty = self.getWholeWarranty(valueContent, 18) # 六年或15万公里
    #     carModelDict["carModelWholeWarranty"] = carModelWholeWarranty

    #     self.saveSingleResult(carModelDict)

    def processSingleResult(self, curCarModelDict):
        print("in processSingleResult")
        self.processCarSpecConfig(curCarModelDict)

        # self.saveSingleResult(curCarModelDict)

    def saveSingleResult(self, curCarModelDict):
        carModelDict = copy.deepcopy(curCarModelDict)
        # print("before filter: carModelDict=%s" % carModelDict)
        # process
        for curKey, curValue in carModelDict.items():
            print("curKey=%s, curValue=%s" % (curKey, curValue))
            if curValue is None:
                carModelDict[curKey] = ""
            elif curValue == "-":
                # '-' -> ''
                carModelDict[curKey] = ""
            elif "暂无" in curValue:
                # '暂无 暂无 暂无', '暂无' -> ''
                carModelDict[curKey] = ""
        # print("after filter: carModelDict=%s" % carModelDict)
        """
        插电式混合动力
            https://www.autohome.com.cn/spec/37077/
                {
                    "carBrandId": "33",
                    "carBrandLogoUrl": "https://car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png",
                    "carBrandName": "奥迪",
                    "carEnergyType": "插电式混合动力",
                    "carMerchantName": "奥迪(进口)",
                    "carMerchantUrl": "https://car.autohome.com.cn/price/brand-33-79.html#pvareaid=2042363",
                    "carModelActualFuelConsumption": "",
                    "carModelActualTestBrakeDistance": "",
                    "carModelActualTestEnduranceMileage": "",
                    "carModelActualTestQuickCharge": "",
                    "carModelActualTestSlowCharge": "",
                    "carModelActualTestSpeedupTime": "",
                    "carModelBodyStructure": "5门5座SUV",
                    "carModelDataSift2": "国V",
                    "carModelDataSift3": "2.0T",
                    "carModelDataSift4": "8挡手自一体",
                    "carModelDriveType": "前置四驱",
                    "carModelEngine": "2.0T 252马力 L4",
                    "carModelEnvStandard": "国V",
                    "carModelGearBox": "8挡手自一体",
                    "carModelGroupName": "2.0升 涡轮增压 252马力 国V",
                    "carModelHorsePowerElectric": "128",
                    "carModelMaxPower": "270",
                    "carModelMaxSpeed": "228",
                    "carModelMaxTorque": "700",
                    "carModelMiitCompositeFuelConsumption": "2.4",
                    "carModelMiitEnduranceMileagePureElectric": "56",
                    "carModelMsrp": "79.08万",
                    "carModelName": "2019款 55 e-tron",
                    "carModelOfficialSpeedupTime": "5.9",
                    "carModelQuickCharge": "2.5",
                    "carModelQuickChargePercent": "",
                    "carModelReleaseTime": "2018.11",
                    "carModelSize": "5071*1968*1716",
                    "carModelSlowCharge": "10.8",
                    "carModelSpecId": "37077",
                    "carModelSpecUrl": "https://www.autohome.com.cn/spec/37077/#pvareaid=3454492",
                    "carModelWholeWarranty": "三年或10万公里",
                    "carModelYear": "2019款",
                    "carSeriesId": "4460",
                    "carSeriesLevelId": "19",
                    "carSeriesLevelName": "中大型SUV",
                    "carSeriesMainImgUrl": "https://car3.autoimg.cn/cardfs/product/g3/M01/45/A0/380x285_0_q87_autohomecar__ChcCRVwuElKAXUwcAAfU0sgxuiw392.jpg",
                    "carSeriesMaxPrice": "79.08万",
                    "carSeriesMinPrice": "79.08万",
                    "carSeriesMsrp": "79.08-79.08万",
                    "carSeriesMsrpUrl": "https://www.autohome.com.cn/4460/price.html#pvareaid=101446",
                    "carSeriesName": "奥迪Q7新能源",
                    "carSeriesUrl": "https://www.autohome.com.cn/4460/#levelsource=000000000_0&pvareaid=101594"
                }

        汽油
            https://www.autohome.com.cn/spec/43593/
                {
                    "carBrandId": "33",
                    "carBrandLogoUrl": "https://car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png",
                    "carBrandName": "奥迪",
                    "carEnergyType": "汽油",
                    "carMerchantName": "一汽-大众奥迪",
                    "carMerchantUrl": "https://car.autohome.com.cn/price/brand-33-9.html#pvareaid=2042363",
                    "carModelActualFuelConsumption": "",
                    "carModelActualTestBrakeDistance": "",
                    "carModelActualTestSpeedupTime": "",
                    "carModelBodyStructure": "5门5座两厢车",
                    "carModelDataSift2": "国VI",
                    "carModelDataSift3": "1.4T",
                    "carModelDataSift4": "7挡双离合",
                    "carModelDriveType": "前置前驱",
                    "carModelEngine": "1.4T 150马力 L4",
                    "carModelEnvStandard": "国VI",
                    "carModelGearBox": "7挡双离合",
                    "carModelGroupName": "1.4升 涡轮增压 150马力 国VI",
                    "carModelMaxPower": "110",
                    "carModelMaxSpeed": "200",
                    "carModelMaxTorque": "250",
                    "carModelMiitCompositeFuelConsumption": "5.8",
                    "carModelMsrp": "19.32万",
                    "carModelName": "2020款 改款 Sportback 35 TFSI 进取型 国VI",
                    "carModelOfficialSpeedupTime": "8.4",
                    "carModelReleaseTime": "2020.04",
                    "carModelSize": "4312*1785*1426",
                    "carModelSpecId": "43593",
                    "carModelSpecUrl": "https://www.autohome.com.cn/spec/43593/#pvareaid=3454492",
                    "carModelWholeWarranty": "三年或10万公里",
                    "carModelYear": "2020款",
                    "carSeriesId": "3170",
                    "carSeriesLevelId": "3",
                    "carSeriesLevelName": "紧凑型车",
                    "carSeriesMainImgUrl": "https://car3.autoimg.cn/cardfs/product/g27/M0A/C9/C8/380x285_0_q87_autohomecar__ChsEnV2uZRCAIMoIAAcQmM2ogcg307.jpg",
                    "carSeriesMaxPrice": "23.46万",
                    "carSeriesMinPrice": "19.32万",
                    "carSeriesMsrp": "19.32-23.46万",
                    "carSeriesMsrpUrl": "https://www.autohome.com.cn/3170/price.html#pvareaid=101446",
                    "carSeriesName": "奥迪A3",
                    "carSeriesUrl": "https://www.autohome.com.cn/3170/#levelsource=000000000_0&pvareaid=101594"
                }

            https://car.autohome.com.cn/config/spec/16147.html
                {
                    "carBrandId": "91",
                    "carBrandLogoUrl": "https://car3.autoimg.cn/cardfs/series/g26/M05/AE/94/100x100_f40_autohomecar__wKgHEVs9tm6ASWlTAAAUz_2mWTY720.png",
                    "carBrandName": "红旗",
                    "carEnergyType": "汽油",
                    "carMerchantName": "一汽红旗",
                    "carMerchantUrl": "https://car.autohome.com.cn/price/brand-91-190.html#pvareaid=2042363",
                    "carModelActualFuelConsumption": "",
                    "carModelActualTestBrakeDistance": "",
                    "carModelActualTestSpeedupTime": "",
                    "carModelBodyStructure": "4门5座三厢车",
                    "carModelDriveType": "后驱",
                    "carModelEngine": "2.0T 204马力 L4",
                    "carModelEnvStandard": "国IV(国V)",
                    "carModelGearBox": "6挡手自一体",
                    "carModelGroupName": "2.0升 涡轮增压 204马力",
                    "carModelMaxPower": "150",
                    "carModelMaxSpeed": "",
                    "carModelMaxTorque": "260",
                    "carModelMiitCompositeFuelConsumption": "9.8",
                    "carModelMsrp": "37.98万",
                    "carModelName": "2013款 2.0T 尊贵型",
                    "carModelOfficialSpeedupTime": "",
                    "carModelReleaseTime": "2013.05",
                    "carModelSize": "5095*1875*1485",
                    "carModelSpecId": "16147",
                    "carModelSpecUrl": "https://www.autohome.com.cn/spec/16147/",
                    "carModelWholeWarranty": "四年或10万公里",
                    "carModelYear": "2013款",
                    "carSeriesId": "2771",
                    "carSeriesLevelId": "5",
                    "carSeriesLevelName": "中大型车",
                    "carSeriesMainImgUrl": "https://car3.autoimg.cn/cardfs/product/g27/M07/94/4F/380x285_0_q87_autohomecar__ChsEnV6MPieAbbAgAAiIIauI0dE436.jpg",
                    "carSeriesMaxPrice": "31.78万",
                    "carSeriesMinPrice": "25.28万",
                    "carSeriesMsrp": "25.28-31.78万",
                    "carSeriesMsrpUrl": "https://www.autohome.com.cn/2771/price.html#pvareaid=101446",
                    "carSeriesName": "红旗H7",
                    "carSeriesUrl": "https://www.autohome.com.cn/2771/#levelsource=000000000_0&pvareaid=101594"
                }

            https://www.autohome.com.cn/spec/46144/
            {
                "carBrandId": "36",
                "carBrandLogoUrl": "https://car3.autoimg.cn/cardfs/series/g26/M00/AF/E7/100x100_f40_autohomecar__wKgHHVs9u6mAaY6mAAA2M840O5c440.png",
                "carBrandName": "奔驰",
                "carEnergyType": "汽油",
                "carMerchantName": "北京奔驰",
                "carMerchantUrl": "https://car.autohome.com.cn/price/brand-36-152.html#pvareaid=2042363",
                "carModelActualFuelConsumption": "",
                "carModelActualTestBrakeDistance": "39.01",
                "carModelActualTestEnduranceMileage": "",
                "carModelActualTestQuickCharge": "",
                "carModelActualTestSlowCharge": "",
                "carModelActualTestSpeedupTime": "9.01",
                "carModelBodyStructure": "5门5座SUV",
                "carModelDataSift2": "国VI",
                "carModelDataSift3": "1.3T",
                "carModelDataSift4": "7挡双离合",
                "carModelDriveType": "前置前驱",
                "carModelEngine": "1.3T 163马力 L4",
                "carModelEnvStandard": "国VI",
                "carModelGearBox": "7挡双离合",
                "carModelGroupName": "1.3升 涡轮增压 163马力 国VI",
                "carModelHorsePowerElectric": "",
                "carModelMaxPower": "120",
                "carModelMaxSpeed": "207",
                "carModelMaxTorque": "250",
                "carModelMiitCompositeFuelConsumption": "6.1",
                "carModelMiitEnduranceMileagePureElectric": "",
                "carModelMsrp": "30.38万",
                "carModelName": "2020款 GLA 200",
                "carModelOfficialSpeedupTime": "9",
                "carModelQuickCharge": "",
                "carModelQuickChargePercent": "",
                "carModelReleaseTime": "2020.07",
                "carModelSize": "4417*1834*1610",
                "carModelSlowCharge": "",
                "carModelSpecId": "46144",
                "carModelSpecUrl": "https://www.autohome.com.cn/spec/46144/#pvareaid=3454492",
                "carModelWholeWarranty": "三年不限公里",
                "carModelYear": "2020款",
                "carSeriesId": "3248",
                "carSeriesLevelId": "17",
                "carSeriesLevelName": "紧凑型SUV",
                "carSeriesMainImgUrl": "https://car3.autoimg.cn/cardfs/product/g27/M04/4F/39/380x285_0_q87_autohomecar__ChwFkV8eQZSAPD0IAAz4_sk-g8M069.jpg",
                "carSeriesMaxPrice": "30.38万",
                "carSeriesMinPrice": "30.38万",
                "carSeriesMsrp": "30.38-30.38万",
                "carSeriesMsrpUrl": "https://www.autohome.com.cn/3248/price.html#pvareaid=101446",
                "carSeriesName": "奔驰GLA",
                "carSeriesUrl": "https://www.autohome.com.cn/3248/#levelsource=000000000_0&pvareaid=101594"
            }

        纯电动
            https://www.autohome.com.cn/spec/42875/
                {
                    "carBrandId": "33",
                    "carBrandLogoUrl": "https://car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png",
                    "carBrandName": "奥迪",
                    "carEnergyType": "纯电动",
                    "carMerchantName": "一汽-大众奥迪",
                    "carMerchantUrl": "https://car.autohome.com.cn/price/brand-33-9.html#pvareaid=2042363",
                    "carModelActualTestBrakeDistance": "",
                    "carModelActualTestEnduranceMileage": "",
                    "carModelActualTestQuickCharge": "",
                    "carModelActualTestSlowCharge": "",
                    "carModelActualTestSpeedupTime": "",
                    "carModelBodyStructure": "5门5座SUV",
                    "carModelDataSift2": "100KW",
                    "carModelDataSift3": "265公里",
                    "carModelDataSift4": "单速",
                    "carModelDriveType": "前置前驱",
                    "carModelGearBox": "电动车单速变速箱",
                    "carModelGroupName": "电动 136马力",
                    "carModelHorsePowerElectric": "136",
                    "carModelMaxPower": "100",
                    "carModelMaxSpeed": "150",
                    "carModelMaxTorque": "290",
                    "carModelMiitEnduranceMileagePureElectric": "265",
                    "carModelMsrp": "22.68万",
                    "carModelName": "2019款 Q2L e-tron 纯电智酷型",
                    "carModelOfficialSpeedupTime": "",
                    "carModelQuickCharge": "0.6",
                    "carModelQuickChargePercent": "80",
                    "carModelReleaseTime": "2019.11",
                    "carModelSize": "4237*1785*1548",
                    "carModelSlowCharge": "17",
                    "carModelSpecId": "42875",
                    "carModelSpecUrl": "https://www.autohome.com.cn/spec/42875/#pvareaid=3454492",
                    "carModelWholeWarranty": "三年或10万公里",
                    "carModelYear": "2019款",
                    "carSeriesId": "5240",
                    "carSeriesLevelId": "16",
                    "carSeriesLevelName": "小型SUV",
                    "carSeriesMainImgUrl": "https://car2.autoimg.cn/cardfs/product/g26/M08/5E/E9/380x285_0_q87_autohomecar__ChsEnFz-PK6AKWR5AAb_0TkX5cE445.jpg",
                    "carSeriesMaxPrice": "23.73万",
                    "carSeriesMinPrice": "22.68万",
                    "carSeriesMsrp": "22.68-23.73万",
                    "carSeriesMsrpUrl": "https://www.autohome.com.cn/5240/price.html#pvareaid=101446",
                    "carSeriesName": "奥迪Q2L e-tron",
                    "carSeriesUrl": "https://www.autohome.com.cn/5240/#levelsource=000000000_0&pvareaid=101594"
                }

        油电混合
            https://car.autohome.com.cn/config/spec/35507.html
                {
                    "carBrandId": "52",
                    "carBrandLogoUrl": "https://car2.autoimg.cn/cardfs/series/g29/M02/B0/BE/100x100_f40_autohomecar__ChcCSFs91WqAGpOHAABVaN6-df4803.png",
                    "carBrandName": "雷克萨斯",
                    "carEnergyType": "油电混合",
                    "carMerchantName": "雷克萨斯",
                    "carMerchantUrl": "https://car.autohome.com.cn/price/brand-52-65.html#pvareaid=2042363",
                    "carModelActualFuelConsumption": "",
                    "carModelActualTestBrakeDistance": "",
                    "carModelActualTestEnduranceMileage": "",
                    "carModelActualTestQuickCharge": "",
                    "carModelActualTestSlowCharge": "",
                    "carModelActualTestSpeedupTime": "",
                    "carModelBodyStructure": "5门5座两厢车",
                    "carModelDataSift2": "",
                    "carModelDataSift3": "",
                    "carModelDataSift4": "",
                    "carModelDriveType": "前驱",
                    "carModelEngine": "1.8L 99马力 L4",
                    "carModelEnvStandard": "国IV(国V)",
                    "carModelGearBox": "E-CVT无级变速",
                    "carModelGroupName": "1.8升 99马力",
                    "carModelHorsePowerElectric": "82",
                    "carModelMaxPower": "100",
                    "carModelMaxSpeed": "",
                    "carModelMaxTorque": "",
                    "carModelMiitCompositeFuelConsumption": "4.6",
                    "carModelMiitEnduranceMileagePureElectric": "",
                    "carModelMsrp": "26.70万",
                    "carModelName": "2018款 CT200h 多彩生活特别限量版",
                    "carModelOfficialSpeedupTime": "",
                    "carModelQuickCharge": "",
                    "carModelQuickChargePercent": "",
                    "carModelReleaseTime": "2018.08",
                    "carModelSize": "4360*1765*1455",
                    "carModelSlowCharge": "",
                    "carModelSpecId": "35507",
                    "carModelSpecUrl": "https://www.autohome.com.cn/spec/35507/",
                    "carModelWholeWarranty": "六年或15万公里",
                    "carModelYear": "2018款",
                    "carSeriesId": "2063",
                    "carSeriesLevelId": "3",
                    "carSeriesLevelName": "紧凑型车",
                    "carSeriesMainImgUrl": "https://car2.autoimg.cn/cardfs/product/g29/M09/FB/2F/380x285_0_q87_autohomecar__ChsEfl9GUImAT_62AAnvWT7-yrA464.jpg",
                    "carSeriesMaxPrice": "28.20万",
                    "carSeriesMinPrice": "21.50万",
                    "carSeriesMsrp": "21.50-28.20万",
                    "carSeriesMsrpUrl": "https://www.autohome.com.cn/2063/price.html#pvareaid=101446",
                    "carSeriesName": "雷克萨斯CT",
                    "carSeriesUrl": "https://www.autohome.com.cn/2063/#levelsource=000000000_0&pvareaid=101594"
                }
        """

        carAllKeyList = [
            "carBrandName",
            "carBrandId",
            "carBrandLogoUrl",
            "carMerchantName",
            "carMerchantUrl",
            "carSeriesId",
            "carSeriesName",
            "carSeriesUrl",
            "carSeriesMsrp",
            "carSeriesMsrpUrl",
            "carSeriesMainImgUrl",
            "carSeriesMinPrice",
            "carSeriesMaxPrice",
            "carSeriesLevelName",
            "carSeriesLevelId",
            "carModelName",
            "carModelSpecUrl",
            "carModelSpecId",
            "carModelMsrp",
            "carModelYear",
            "carModelGearBox",
            "carModelDriveType",
            "carModelBodyStructure",
            "carModelEngine",
            "carModelGroupName",
            "carModelDataSift2",
            "carModelDataSift3",
            "carModelDataSift4",
            "carEnergyType",
            "carModelEnvStandard",
            "carModelReleaseTime",
            "carModelMaxPower",
            "carModelMaxTorque",
            "carModelSize",
            "carModelMaxSpeed",
            "carModelOfficialSpeedupTime",
            "carModelActualTestSpeedupTime",
            "carModelActualTestBrakeDistance",
            "carModelMiitCompositeFuelConsumption",
            "carModelActualFuelConsumption",
            "carModelMiitEnduranceMileagePureElectric",
            "carModelQuickCharge",
            "carModelSlowCharge",
            "carModelQuickChargePercent",
            "carModelHorsePowerElectric",
            "carModelActualTestEnduranceMileage",
            "carModelActualTestQuickCharge",
            "carModelActualTestSlowCharge",
            "carModelWholeWarranty",
        ]

        for eachCarKey in carAllKeyList:
            if eachCarKey not in carModelDict:
                print("found miss key: %s" % eachCarKey)
                carModelDict[eachCarKey] = ""

        carModelSpecUrl = carModelDict["carModelSpecUrl"]
        self.send_message(self.project_name, carModelDict, url=carModelSpecUrl)

    def on_message(self, project, msg):
        print("on_message: msg=%s" % msg)
        return msg

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-09-08 13:56:21

结果

输出结果

全部结果：autohome_20200901_car44451.xlsx

在线下载：autohome_20200901_car44451.xlsx

截图效果

单个结果举例

单个车型车系的详细数据

汽油

https://www.autohome.com.cn/spec/43593/

  {
      "carBrandId": "33",
      "carBrandLogoUrl": "https://car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png",
      "carBrandName": "奥迪",
      "carEnergyType": "汽油",
      "carMerchantName": "一汽-大众奥迪",
      "carMerchantUrl": "https://car.autohome.com.cn/price/brand-33-9.html#pvareaid=2042363",
      "carModelActualFuelConsumption": "",
      "carModelActualTestBrakeDistance": "",
      "carModelActualTestSpeedupTime": "",
      "carModelBodyStructure": "5门5座两厢车",
      "carModelDataSift2": "国VI",
      "carModelDataSift3": "1.4T",
      "carModelDataSift4": "7挡双离合",
      "carModelDriveType": "前置前驱",
      "carModelEngine": "1.4T 150马力 L4",
      "carModelEnvStandard": "国VI",
      "carModelGearBox": "7挡双离合",
      "carModelGroupName": "1.4升 涡轮增压 150马力 国VI",
      "carModelMaxPower": "110",
      "carModelMaxSpeed": "200",
      "carModelMaxTorque": "250",
      "carModelMiitCompositeFuelConsumption": "5.8",
      "carModelMsrp": "19.32万",
      "carModelName": "2020款 改款 Sportback 35 TFSI 进取型 国VI",
      "carModelOfficialSpeedupTime": "8.4",
      "carModelReleaseTime": "2020.04",
      "carModelSize": "4312*1785*1426",
      "carModelSpecId": "43593",
      "carModelSpecUrl": "https://www.autohome.com.cn/spec/43593/#pvareaid=3454492",
      "carModelWholeWarranty": "三年或10万公里",
      "carModelYear": "2020款",
      "carSeriesId": "3170",
      "carSeriesLevelId": "3",
      "carSeriesLevelName": "紧凑型车",
      "carSeriesMainImgUrl": "https://car3.autoimg.cn/cardfs/product/g27/M0A/C9/C8/380x285_0_q87_autohomecar__ChsEnV2uZRCAIMoIAAcQmM2ogcg307.jpg",
      "carSeriesMaxPrice": "23.46万",
      "carSeriesMinPrice": "19.32万",
      "carSeriesMsrp": "19.32-23.46万",
      "carSeriesMsrpUrl": "https://www.autohome.com.cn/3170/price.html#pvareaid=101446",
      "carSeriesName": "奥迪A3",
      "carSeriesUrl": "https://www.autohome.com.cn/3170/#levelsource=000000000_0&pvareaid=101594"
  }

https://car.autohome.com.cn/config/spec/16147.html

  {
      "carBrandId": "91",
      "carBrandLogoUrl": "https://car3.autoimg.cn/cardfs/series/g26/M05/AE/94/100x100_f40_autohomecar__wKgHEVs9tm6ASWlTAAAUz_2mWTY720.png",
      "carBrandName": "红旗",
      "carEnergyType": "汽油",
      "carMerchantName": "一汽红旗",
      "carMerchantUrl": "https://car.autohome.com.cn/price/brand-91-190.html#pvareaid=2042363",
      "carModelActualFuelConsumption": "",
      "carModelActualTestBrakeDistance": "",
      "carModelActualTestSpeedupTime": "",
      "carModelBodyStructure": "4门5座三厢车",
      "carModelDriveType": "后驱",
      "carModelEngine": "2.0T 204马力 L4",
      "carModelEnvStandard": "国IV(国V)",
      "carModelGearBox": "6挡手自一体",
      "carModelGroupName": "2.0升 涡轮增压 204马力",
      "carModelMaxPower": "150",
      "carModelMaxSpeed": "",
      "carModelMaxTorque": "260",
      "carModelMiitCompositeFuelConsumption": "9.8",
      "carModelMsrp": "37.98万",
      "carModelName": "2013款 2.0T 尊贵型",
      "carModelOfficialSpeedupTime": "",
      "carModelReleaseTime": "2013.05",
      "carModelSize": "5095*1875*1485",
      "carModelSpecId": "16147",
      "carModelSpecUrl": "https://www.autohome.com.cn/spec/16147/",
      "carModelWholeWarranty": "四年或10万公里",
      "carModelYear": "2013款",
      "carSeriesId": "2771",
      "carSeriesLevelId": "5",
      "carSeriesLevelName": "中大型车",
      "carSeriesMainImgUrl": "https://car3.autoimg.cn/cardfs/product/g27/M07/94/4F/380x285_0_q87_autohomecar__ChsEnV6MPieAbbAgAAiIIauI0dE436.jpg",
      "carSeriesMaxPrice": "31.78万",
      "carSeriesMinPrice": "25.28万",
      "carSeriesMsrp": "25.28-31.78万",
      "carSeriesMsrpUrl": "https://www.autohome.com.cn/2771/price.html#pvareaid=101446",
      "carSeriesName": "红旗H7",
      "carSeriesUrl": "https://www.autohome.com.cn/2771/#levelsource=000000000_0&pvareaid=101594"
  }

https://www.autohome.com.cn/spec/46144/

  {
      "carBrandId": "36",
      "carBrandLogoUrl": "https://car3.autoimg.cn/cardfs/series/g26/M00/AF/E7/100x100_f40_autohomecar__wKgHHVs9u6mAaY6mAAA2M840O5c440.png",
      "carBrandName": "奔驰",
      "carEnergyType": "汽油",
      "carMerchantName": "北京奔驰",
      "carMerchantUrl": "https://car.autohome.com.cn/price/brand-36-152.html#pvareaid=2042363",
      "carModelActualFuelConsumption": "",
      "carModelActualTestBrakeDistance": "39.01",
      "carModelActualTestEnduranceMileage": "",
      "carModelActualTestQuickCharge": "",
      "carModelActualTestSlowCharge": "",
      "carModelActualTestSpeedupTime": "9.01",
      "carModelBodyStructure": "5门5座SUV",
      "carModelDataSift2": "国VI",
      "carModelDataSift3": "1.3T",
      "carModelDataSift4": "7挡双离合",
      "carModelDriveType": "前置前驱",
      "carModelEngine": "1.3T 163马力 L4",
      "carModelEnvStandard": "国VI",
      "carModelGearBox": "7挡双离合",
      "carModelGroupName": "1.3升 涡轮增压 163马力 国VI",
      "carModelHorsePowerElectric": "",
      "carModelMaxPower": "120",
      "carModelMaxSpeed": "207",
      "carModelMaxTorque": "250",
      "carModelMiitCompositeFuelConsumption": "6.1",
      "carModelMiitEnduranceMileagePureElectric": "",
      "carModelMsrp": "30.38万",
      "carModelName": "2020款 GLA 200",
      "carModelOfficialSpeedupTime": "9",
      "carModelQuickCharge": "",
      "carModelQuickChargePercent": "",
      "carModelReleaseTime": "2020.07",
      "carModelSize": "4417*1834*1610",
      "carModelSlowCharge": "",
      "carModelSpecId": "46144",
      "carModelSpecUrl": "https://www.autohome.com.cn/spec/46144/#pvareaid=3454492",
      "carModelWholeWarranty": "三年不限公里",
      "carModelYear": "2020款",
      "carSeriesId": "3248",
      "carSeriesLevelId": "17",
      "carSeriesLevelName": "紧凑型SUV",
      "carSeriesMainImgUrl": "https://car3.autoimg.cn/cardfs/product/g27/M04/4F/39/380x285_0_q87_autohomecar__ChwFkV8eQZSAPD0IAAz4_sk-g8M069.jpg",
      "carSeriesMaxPrice": "30.38万",
      "carSeriesMinPrice": "30.38万",
      "carSeriesMsrp": "30.38-30.38万",
      "carSeriesMsrpUrl": "https://www.autohome.com.cn/3248/price.html#pvareaid=101446",
      "carSeriesName": "奔驰GLA",
      "carSeriesUrl": "https://www.autohome.com.cn/3248/#levelsource=000000000_0&pvareaid=101594"
  }

插电式混合动力

https://www.autohome.com.cn/spec/37077/

{
    "carBrandId": "33",
    "carBrandLogoUrl": "https://car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png",
    "carBrandName": "奥迪",
    "carEnergyType": "插电式混合动力",
    "carMerchantName": "奥迪(进口)",
    "carMerchantUrl": "https://car.autohome.com.cn/price/brand-33-79.html#pvareaid=2042363",
    "carModelActualFuelConsumption": "",
    "carModelActualTestBrakeDistance": "",
    "carModelActualTestEnduranceMileage": "",
    "carModelActualTestQuickCharge": "",
    "carModelActualTestSlowCharge": "",
    "carModelActualTestSpeedupTime": "",
    "carModelBodyStructure": "5门5座SUV",
    "carModelDataSift2": "国V",
    "carModelDataSift3": "2.0T",
    "carModelDataSift4": "8挡手自一体",
    "carModelDriveType": "前置四驱",
    "carModelEngine": "2.0T 252马力 L4",
    "carModelEnvStandard": "国V",
    "carModelGearBox": "8挡手自一体",
    "carModelGroupName": "2.0升 涡轮增压 252马力 国V",
    "carModelHorsePowerElectric": "128",
    "carModelMaxPower": "270",
    "carModelMaxSpeed": "228",
    "carModelMaxTorque": "700",
    "carModelMiitCompositeFuelConsumption": "2.4",
    "carModelMiitEnduranceMileagePureElectric": "56",
    "carModelMsrp": "79.08万",
    "carModelName": "2019款 55 e-tron",
    "carModelOfficialSpeedupTime": "5.9",
    "carModelQuickCharge": "2.5",
    "carModelQuickChargePercent": "",
    "carModelReleaseTime": "2018.11",
    "carModelSize": "5071*1968*1716",
    "carModelSlowCharge": "10.8",
    "carModelSpecId": "37077",
    "carModelSpecUrl": "https://www.autohome.com.cn/spec/37077/#pvareaid=3454492",
    "carModelWholeWarranty": "三年或10万公里",
    "carModelYear": "2019款",
    "carSeriesId": "4460",
    "carSeriesLevelId": "19",
    "carSeriesLevelName": "中大型SUV",
    "carSeriesMainImgUrl": "https://car3.autoimg.cn/cardfs/product/g3/M01/45/A0/380x285_0_q87_autohomecar__ChcCRVwuElKAXUwcAAfU0sgxuiw392.jpg",
    "carSeriesMaxPrice": "79.08万",
    "carSeriesMinPrice": "79.08万",
    "carSeriesMsrp": "79.08-79.08万",
    "carSeriesMsrpUrl": "https://www.autohome.com.cn/4460/price.html#pvareaid=101446",
    "carSeriesName": "奥迪Q7新能源",
    "carSeriesUrl": "https://www.autohome.com.cn/4460/#levelsource=000000000_0&pvareaid=101594"
}

纯电动

https://www.autohome.com.cn/spec/42875/

{
    "carBrandId": "33",
    "carBrandLogoUrl": "https://car2.autoimg.cn/cardfs/series/g26/M0B/AE/B3/100x100_f40_autohomecar__wKgHEVs9u5WAV441AAAKdxZGE4U148.png",
    "carBrandName": "奥迪",
    "carEnergyType": "纯电动",
    "carMerchantName": "一汽-大众奥迪",
    "carMerchantUrl": "https://car.autohome.com.cn/price/brand-33-9.html#pvareaid=2042363",
    "carModelActualTestBrakeDistance": "",
    "carModelActualTestEnduranceMileage": "",
    "carModelActualTestQuickCharge": "",
    "carModelActualTestSlowCharge": "",
    "carModelActualTestSpeedupTime": "",
    "carModelBodyStructure": "5门5座SUV",
    "carModelDataSift2": "100KW",
    "carModelDataSift3": "265公里",
    "carModelDataSift4": "单速",
    "carModelDriveType": "前置前驱",
    "carModelGearBox": "电动车单速变速箱",
    "carModelGroupName": "电动 136马力",
    "carModelHorsePowerElectric": "136",
    "carModelMaxPower": "100",
    "carModelMaxSpeed": "150",
    "carModelMaxTorque": "290",
    "carModelMiitEnduranceMileagePureElectric": "265",
    "carModelMsrp": "22.68万",
    "carModelName": "2019款 Q2L e-tron 纯电智酷型",
    "carModelOfficialSpeedupTime": "",
    "carModelQuickCharge": "0.6",
    "carModelQuickChargePercent": "80",
    "carModelReleaseTime": "2019.11",
    "carModelSize": "4237*1785*1548",
    "carModelSlowCharge": "17",
    "carModelSpecId": "42875",
    "carModelSpecUrl": "https://www.autohome.com.cn/spec/42875/#pvareaid=3454492",
    "carModelWholeWarranty": "三年或10万公里",
    "carModelYear": "2019款",
    "carSeriesId": "5240",
    "carSeriesLevelId": "16",
    "carSeriesLevelName": "小型SUV",
    "carSeriesMainImgUrl": "https://car2.autoimg.cn/cardfs/product/g26/M08/5E/E9/380x285_0_q87_autohomecar__ChsEnFz-PK6AKWR5AAb_0TkX5cE445.jpg",
    "carSeriesMaxPrice": "23.73万",
    "carSeriesMinPrice": "22.68万",
    "carSeriesMsrp": "22.68-23.73万",
    "carSeriesMsrpUrl": "https://www.autohome.com.cn/5240/price.html#pvareaid=101446",
    "carSeriesName": "奥迪Q2L e-tron",
    "carSeriesUrl": "https://www.autohome.com.cn/5240/#levelsource=000000000_0&pvareaid=101594"
}

油电混合

https://car.autohome.com.cn/config/spec/35507.html

{
    "carBrandId": "52",
    "carBrandLogoUrl": "https://car2.autoimg.cn/cardfs/series/g29/M02/B0/BE/100x100_f40_autohomecar__ChcCSFs91WqAGpOHAABVaN6-df4803.png",
    "carBrandName": "雷克萨斯",
    "carEnergyType": "油电混合",
    "carMerchantName": "雷克萨斯",
    "carMerchantUrl": "https://car.autohome.com.cn/price/brand-52-65.html#pvareaid=2042363",
    "carModelActualFuelConsumption": "",
    "carModelActualTestBrakeDistance": "",
    "carModelActualTestEnduranceMileage": "",
    "carModelActualTestQuickCharge": "",
    "carModelActualTestSlowCharge": "",
    "carModelActualTestSpeedupTime": "",
    "carModelBodyStructure": "5门5座两厢车",
    "carModelDataSift2": "",
    "carModelDataSift3": "",
    "carModelDataSift4": "",
    "carModelDriveType": "前驱",
    "carModelEngine": "1.8L 99马力 L4",
    "carModelEnvStandard": "国IV(国V)",
    "carModelGearBox": "E-CVT无级变速",
    "carModelGroupName": "1.8升 99马力",
    "carModelHorsePowerElectric": "82",
    "carModelMaxPower": "100",
    "carModelMaxSpeed": "",
    "carModelMaxTorque": "",
    "carModelMiitCompositeFuelConsumption": "4.6",
    "carModelMiitEnduranceMileagePureElectric": "",
    "carModelMsrp": "26.70万",
    "carModelName": "2018款 CT200h 多彩生活特别限量版",
    "carModelOfficialSpeedupTime": "",
    "carModelQuickCharge": "",
    "carModelQuickChargePercent": "",
    "carModelReleaseTime": "2018.08",
    "carModelSize": "4360*1765*1455",
    "carModelSlowCharge": "",
    "carModelSpecId": "35507",
    "carModelSpecUrl": "https://www.autohome.com.cn/spec/35507/",
    "carModelWholeWarranty": "六年或15万公里",
    "carModelYear": "2018款",
    "carSeriesId": "2063",
    "carSeriesLevelId": "3",
    "carSeriesLevelName": "紧凑型车",
    "carSeriesMainImgUrl": "https://car2.autoimg.cn/cardfs/product/g29/M09/FB/2F/380x285_0_q87_autohomecar__ChsEfl9GUImAT_62AAnvWT7-yrA464.jpg",
    "carSeriesMaxPrice": "28.20万",
    "carSeriesMinPrice": "21.50万",
    "carSeriesMsrp": "21.50-28.20万",
    "carSeriesMsrpUrl": "https://www.autohome.com.cn/2063/price.html#pvareaid=101446",
    "carSeriesName": "雷克萨斯CT",
    "carSeriesUrl": "https://www.autohome.com.cn/2063/#levelsource=000000000_0&pvareaid=101594"
}

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-09-08 13:57:43

百度热榜

百度热榜

此处以 爬取百度热榜的内容列表 为例，去演示如何使用PySpider

代码crawlBaiduHotList_PySpider_1501

#!/usr/bin/env python
# -*- encoding: utf-8 -*-
# Created on 2020-07-31 15:01:00
# Project: crawlBaiduHotList_PySpider_1501

from pyspider.libs.base_handler import *
from pyspider.database import connect_database

class Handler(BaseHandler):
    crawl_config = {
    }

    # @every(minutes=24 * 60)
    def on_start(self):
        UserAgent_Chrome_Mac = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36"
        curHeaderDict = {
            "User-Agent": UserAgent_Chrome_Mac,
        }
        self.crawl('https://www.baidu.com/', callback=self.baiduHome, headers=curHeaderDict)

    # @config(age=10 * 24 * 60 * 60)
    def baiduHome(self, response):
        # for eachItem in response.doc('span[class="title-content-title"]').items():

        titleItemGenerator = response.doc('span[class="title-content-title"]').items()
        titleItemList = list(titleItemGenerator)
        print("titleItemList=%s" % titleItemList)
        # for eachItem in titleItemList:
        for curIdx, eachItem in enumerate(titleItemList):
            print("[%d] eachItem=%s" % (curIdx, eachItem))
            itemTitleStr = eachItem.text()
            print("itemTitleStr=%s" % itemTitleStr)
            curUrl = "%s#%d" % (response.url, curIdx)
            print("curUrl=%s" % curUrl)
            curResult = {
                # "url": response.url,
                # "url": curUrl,
                "百度热榜标题": itemTitleStr,
            }
            # return curResult
            # self.send_message(self.project_name, curResult, url=response.url)
            self.send_message(self.project_name, curResult, url=curUrl)

    def on_message(self, project, msg):
        print("on_message: msg=", msg)
        return msg

效果和结果

调试

运行

点击Results -> Results结果

点击CSV-> 导出CSV

CSV预览

更多细节

详见：

【已解决】用Python爬虫框架PySpider实现爬虫爬取百度热榜内容列表

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-04-14 20:37:47

附录

附录

下面列出相关参考资料。

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2020-07-30 14:00:04

参考资料

参考资料

【未解决】用Python爬取汽车之家的车型车系详细数据

【已解决】Mac中恢复或重新搭建PySpider开发环境

【已解决】PySpider访问汽车之家报错：requests.exceptions.HTTPError HTTP 403 Forbidden

【已解决】PySpider中获取PyQuery获取到节点的子元素

【已解决】PySpider中PyQuery如何匹配某个元素中多个属性值

【已解决】汽车之家车型车系数据：支持新版车系页面

【已解决】汽车之家车型车系数据：支持旧版车系页面

【已解决】汽车之家车型车系数据：抓取车型的详细参数配置

【未解决】汽车之家车型车系数据：想办法破解js和css加密的文字显示

【已解决】汽车之家车型车系数据：解决部分车型基本参数字段个数不一致问题

【已解决】汽车之家车型车系数据：优化去掉js加速抓取车型参数配置

【规避解决】汽车之家配置页面无法直接获取到文字数据

【已解决】汽车之家车型车系数据：carBrandId是空

【未解决】汽车之家车型车系数据：能源类型是空白的车型

【已解决】汽车之家车型车系数据：车身结构的值包含span标签

【未解决】汽车之家车型车系数据：有些参数字段的值包含span特殊标签

【已解决】汽车之家车型车系数据：抓取车型的详细参数配置

【已解决】车型车系数据缺失如红旗H5等部分车型数据

【已解决】用Python爬虫框架PySpider实现爬虫爬取百度热榜内容列表

【已解决】PySpider中如何在单个页面返回多个结果保存到自带的Results页面中的列表中

【已解决】PySpider抓包百度热榜标题列表结果

【已解决】Mac中安装phantomjs

【已解决】Mac中启动PySpider

【已解决】Mac中pip安装pycurl报错：fatal error openssl/ssl.h file not found

【已解决】Mac中给Python3安装PySpider

【已解决】CentOS7中安装全局的PySpider并配置和运行

【已解决】CentOS中全局的PySpider中如何指定data目录位置

【未解决】PySpider运行批量下载时报错：HTTP 599 Operation timed out after milliseconds with out of bytes received

【已解决】PySpider无法继续爬取剩余绘本数据

【已解决】PySpider中如何更改默认5000端口

【已解决】PySpider中使用IP代理proxy出错：requests.exceptions.HTTPError HTTP 404 Not Found

【已解决】写Python爬虫爬取汽车之家品牌车系车型数据 – 在路上

【无法解决】PySpider的部署运行而非调试界面上RUN运行

HTML解析库Python版jQuery：PyQuery

pyspider是开源强大的python爬虫系统 - pyspider中文网

【已解决】PySpider中保存数据到mysql

【已解决】PySpider中如何清空之前运行的数据和正在运行的任务 – 在路上

【未解决】pyspider中如何给phantomjs传递额外参数 – 在路上

【已解决】PySpider中页面部分内容不显示 – 在路上

【未解决】pyspider运行出错：FETCH_ERROR HTTP 599 Connection timed out after milliseconds

【记录】Mac中安装和运行pyspider

【整理】pyspider vs scrapy

【已解决】pyspider中phantomjs中的proxy是什么意思 – 在路上

【已解决】pyspider中运行result_worker出错：ModuleNotFoundError No module named mysql

【已解决】PySpider中传递参数给下一级且当下一级失败时也可以执行

【已解决】pyspider中出错：TypeError __init__() got an unexpected keyword argument resultdb – 在路上

【已解决】pyspider运行出错：ImportError pycurl libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)

【已解决】pyspider中pymysql中insert失败且except部分代码没有执行

【已解决】pyspider运行出错：Error Could not create web server listening on port 25555 – 在路上

【已解决】pyspider中的css选择器不工作 – 在路上

【已解决】pyspider中如何写规则去提取网页内容 – 在路上

【已解决】PySpider如何把json结果数据保存到csv或excel文件中 – 在路上

【已解决】PySpider中如何单个页面返回多个json数据结果 – 在路上

【已解决】Mac或Win中用Excel打开UTF8编码的csv文件显示乱码

【已解决】pyspider中如何加载汽车之家页面中的更多内容 – 在路上

【已解决】PySpider中如何发送POST请求且传递格式为application/x-www-form-urlencoded的form data参数 – 在路上

【已解决】PySpider中如何强制让重复的url地址继续爬取 – 在路上

安装pyspider遇到的坑（python3.6）_盛夏88688的博客-CSDN博客_python 3.6 with报错use async with instead

How to delete a spider and the data? · Issue #380 · binux/pyspider

Frequently Asked Questions - pyspider

python爬虫 - crawl 连接网页超时，HTTP 599 - SegmentFault 思否

crifan.com，使用署名4.0国际(CC BY 4.0)协议发布 all right reserved，powered by Gitbook最后更新： 2021-09-08 13:44:31