架构师

您现在的位置是:首页 > 程序人生 > 网络爬虫

网络爬虫

webmagic无法抓取只支持TLS1.2的https站点(protocol_version)

架构师小跟班 2020-07-18 网络爬虫
WebMagic默认的HttpClient只会用TLSv1去请求,对于某些只支持TLS1.2的站点(例如 https://juejin.im/) ,就会报错:javax.net.ssl.SSLException: Received fatal alert: protocol_ve

WebMagic默认的HttpClient只会用TLSv1去请求,对于某些只支持TLS1.2的站点(例如 https://juejin.im/) ,就会报错:

javax.net.ssl.SSLException: Received fatal alert: protocol_version

at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)

at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)

at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)

at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)

at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)

at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)

at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)

at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)

at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)

at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)

at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)

at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)

at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)

at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)

at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

at us.codecraft.webmagic.downloader.HttpClientDownloader.download(HttpClientDownloader.java:85)

现在的修改方式是在HttpClientGenerator中构建SSLConnectionSocketFactory时加上支持。

作者回复:

更新会在0.7.4版本发布。

临时适配方式,修改HttpClientGenerator中的buildSSLConnectionSocketFactory方法,

return new SSLConnectionSocketFactory(createIgnoreVerifySSL(), new String[]{"SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2"},

                    null,

                    new DefaultHostnameVerifier())

重写自己实现的HttpClientDownloader,并设置到Spider中。

-----------------------------------------

三年过去了,作者的0.7.4版本还没有发布。

大家参考另一篇博客试试吧:https://www.jiagou1216.com/life/crawler/1004.html


文章评论