网上免费代理失效率很高，其原因不外乎以下几种：
1.大量代理代理服务器，主要是由肉鸡主机构成，有些是用户机被木马了，也有一些是VPS，其中也有一些高质量的肉鸡服务器，但这种高质量的代理一般不会免费提供给用户。
2.网上太多人在玩爬虫类或群发类的开发，对这种免费代理的需求较高，这么多人在使用这些质量较差的代理，极有可能导致代理处理不过来而宕机
3.网管发现了，关闭了代理。
4.修复了漏洞。

所以说要想抢得更好的代理资源，需要如下：
1.新代理刚发现，就马上测试并使用,不要考虑存放太久，否则被别人使用过多了，它就更容易失效。
2.在进行代理可行性测试时，避免不必要的浪费测试。
3.测试的代理的连通性，必须选择高质量地址，首先CDN资源如query.min.js，再到一些大站如qq首页等。

目前稳定使用scrapy的依赖环境如下：
pip install Twisted==14.0.2 w3lib==1.11.0 queuelib==1.2.2 lxml==3.3.0 pyOpenSSL==0.12 cssselect==0.9.1 six==1.9.0 scrapy==0.24.4 pycurl==7.19.5.1 django==1.4.18 mysql-python==1.2.5 service_identity==14.0.0 selenium==2.44.0 simplejson==3.6.5
pip install Twisted==14.0.2 https://pypi.python.org/packages/source/T/Twisted/Twisted-14.0.2.tar.bz2
pip install w3lib==1.11.0
pip install queuelib==1.2.2
pip install lxml==3.3.0
pip install pyOpenSSL==0.12
pip install cssselect==0.9.1
pip install six==1.9.0
pip install scrapy==0.24.4
pip install pycurl==7.19.5.1
pip install django==1.4.18
pip install mysql-python==1.2.5
pip install service_identity==14.0.0
pip install selenium==2.44.0
pip install simplejson==3.6.5
【目前列出的版本为在Centos5.8上运行的版本，现移植到6.6后，仍保持一致,按如下版本安装，可解决SSL或HTTPS的崩溃BUG，2015年8月14日备注】
characteristic (14.3.0)
cssselect (0.9.1)
Django (1.4.18)
lxml (3.3.0)
MySQL-python (1.2.5)
pip (1.5.6)
pyasn1 (0.1.7)
pyasn1-modules (0.0.5)
pycurl (7.19.5.1)
pyOpenSSL (0.12)
pypm (1.4.3)
queuelib (1.2.2)
Scrapy (0.24.4)
selenium (2.44.0)
service-identity (14.0.0)
setuptools (5.2)
simplejson (3.6.5)
six (1.9.0)
Twisted (14.0.2)
w3lib (1.11.0)
wsgiref (0.1.2)
zope.interface (4.1.2)

Twisted 15.0 appears to have changed the signature of the _getEndpoint method on twisted.web.client.Agent. This causes the http11 handler to throw exceptions like so:
Traceback (most recent call last):
File “/usr/share/python/spotify-prelude2-directed-crawlers/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py”, line 38, in process_request
return download_func(request=request, spider=spider)
File “/usr/share/python/spotify-prelude2-directed-crawlers/local/lib/python2.7/site-packages/scrapy/core/downloader/__init__.py”, line 123, in _enqueue_request
self._process_queue(spider, slot)
File “/usr/share/python/spotify-prelude2-directed-crawlers/local/lib/python2.7/site-packages/scrapy/core/downloader/__init__.py”, line 143, in _process_queue
dfd = self._download(slot, request, spider)
File “/usr/share/python/spotify-prelude2-directed-crawlers/local/lib/python2.7/site-packages/scrapy/core/downloader/__init__.py”, line 154, in _download
dfd = mustbe_deferred(self.handlers.download_request, request, spider)
— —
File “/usr/share/python/spotify-prelude2-directed-crawlers/local/lib/python2.7/site-packages/scrapy/utils/defer.py”, line 39, in mustbe_deferred
result = f(*args, **kw)
File “/usr/share/python/spotify-prelude2-directed-crawlers/local/lib/python2.7/site-packages/scrapy/core/downloader/handlers/__init__.py”, line 40, in download_request
return handler(request, spider)
File “/usr/share/python/spotify-prelude2-directed-crawlers/local/lib/python2.7/site-packages/scrapy/core/downloader/handlers/http11.py”, line 36, in download_request
return agent.download_request(request)
File “/usr/share/python/spotify-prelude2-directed-crawlers/local/lib/python2.7/site-packages/scrapy/core/downloader/handlers/http11.py”, line 174, in download_request
d = agent.request(method, url, headers, bodyproducer)
File “/usr/share/python/spotify-prelude2-directed-crawlers/local/lib/python2.7/site-packages/twisted/web/client.py”, line 1560, in request
endpoint = self._getEndpoint(parsedURI)
exceptions.TypeError: _getEndpoint() takes exactly 4 arguments (2 given)

That method’s signature in Twisted 15.0.0 is def _getEndpoint(self, uri): while in version 14.0.2 it isdef _getEndpoint(self, scheme, host, port):