脚本宝典收集整理的这篇文章主要介绍了爬虫 谷歌浏览器版本配置汇总(自用),脚本宝典觉得挺不错的,现在分享给大家,也给大家做个参考。
创建对象池,通过对象池获取(销毁)webclient对象
/** * @description: webclient对象 * @author: miracle-god * @time: 2021/11/15 10:18 */ @Slf4j public class ChromeDriverFactory extends BasePooledObjectFactory<WebClient> { /** * @description 创建webclient对象 * @author miracle-god * @date 2021/7/22 13:38 **/ @Override public WebClient create() { log.info("[HtmlUnitFactory]create webclient start"); WebClient webClient = new WebClient(BrowserVersion.FIREFOX); // 解决证书不信任问题 webClient.getOptions().setUseInsecureSSL(true); // 禁用css ,一般来说 css没啥用 webClient.getOptions().setCssEnabled(false); webClient.getOptions().setThrowExceptionOnFailingStatusCode(false); //节约带宽,不下载图片 webClient.getOptions().setDownloadImages(false); //支持重定向 webClient.getOptions().setRedirectEnabled(true); //关闭cookie管理 webClient.getCookieManager().setCookiesEnabled(false); //伪装正常浏览器 // webClient.addRequestHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"); // webClient.addRequestHeader("User-Agent", "Mozilla/5.0 (Linux; Android 10; ELE-AL00 Build/HUAWEIELE-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/76.0.3809.89 Mobile Safari/537.36 T7/12.20.0.0 SP-engine/2.33.0 baiduboxapp"); // 设置支持 js webClient.getOptions().setJavaScriptEnabled(true); // 不抛出异常 webClient.getOptions().setThrowExceptionOnScriptError(false); webClient.getOptions().setDoNotTrackEnabled(true); webClient.getOptions().setHistorySizeLimit(50); // 最好设置一下超时 webClient.getOptions().setTimeout(5 * 1000); //设置文件缓存加速 webClient.getCache().setMaxSize(100); webClient.setJavaScriptErrorListener(new JavaScriptErrorListener() { @Override public void scriptException(HtmlPage page, ScriptException scriptException) { log.info("[JavaScriptErrorListener] scriptException do noting"); } @Override public void timeoutError(HtmlPage page, long allowedTime, long executionTime) { log.info("[JavaScriptErrorListener] timeoutError do noting"); } @Override public void malformedScriptURL(HtmlPage page, String url, MalformedURLException malformedURLException) { log.info("[JavaScriptErrorListener] malformedScriptURL do noting"); } @Override public void loadScriptError(HtmlPage page, URL scriptUrl, Exception exception) { log.info("[JavaScriptErrorListener] loadScriptError do noting"); } @Override public void warn(String message, String sourceName, int line, String lineSource, int lineOffset) { log.info("[JavaScriptErrorListener] warn do noting"); } }); // 支持ajax webClient.setAjaxController(new NicelyResynchronizingAjaxController()); return webClient; } @Override public PooledObject<WebClient> wrap(WebClient webClient) { return new DefaultPooledObject<>(webClient); } /** * 销毁FtpClient对象 * * @param webClientPooledObject */ @Override public void destroyObject(PooledObject<WebClient> webClientPooledObject) { if (webClientPooledObject == null) { return; } WebClient object = webClientPooledObject.getObject(); object.close(); } }
以上是脚本宝典为你收集整理的爬虫 谷歌浏览器版本配置汇总(自用)全部内容,希望文章能够帮你解决爬虫 谷歌浏览器版本配置汇总(自用)所遇到的问题。
本图文内容来源于网友网络收集整理提供,作为学习参考使用,版权属于原作者。
如您有任何意见或建议可联系处理。小编QQ:384754419,请注明来意。