當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

HttpClient v4.5 简单抓取主页数据

發布時間：2023/12/13 编程问答 17 豆豆

生活随笔收集整理的這篇文章主要介紹了 HttpClient v4.5 简单抓取主页数据小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

? ? ? ? 由于工作原因，需要每隔半小時刷新一些網頁，并查看上面的數據是否有更新。這件事能否自動化進行呢？查找了下Java相關的資料，蹦出一個關鍵詞：HttpClient。

? ? ? ? HttpClient是常用Http客戶端庫，相關的資料也不少，只是網上找到的資料好多都是不能用于4.5版的HttpClient，還是需要自己摸索。

? ? ? ? 在eclipse里新建一個maven工程(maven 3)，在pom.xml中做如下設置：

1 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 2 <modelVersion>4.0.0</modelVersion> 3 <groupId>test</groupId> 4 <artifactId>admin.test.httpclient</artifactId> 5 <version>0.0.1-SNAPSHOT</version> 6 <dependencies> 7 <dependency> 8 <groupId>org.apache.httpcomponents</groupId> 9 <artifactId>httpclient</artifactId> 10 <version>4.5</version> 11 </dependency> 17 </dependencies> 18 <build> 19 <finalName>MvnTest</finalName> 20 <plugins> 21 <plugin> 22 <artifactId>maven-compiler-plugin</artifactId> 23 <version>2.0.2</version> 24 <configuration> 25 <source>1.5</source> 26 <target>1.5</target> 27 </configuration> 28 </plugin> 29 </plugins> 30 </build> 31 </project>

? ? ? ? 在pom.xml上運行"maven install"命令完成之后，在“Maven Dependencies”下有了四個jar包：

? ? ? ? 拿某個知名網站發送GET請求做測試，看看效果：

1 public class HttpClientTest { 2 private static String HOST = "www.sina.com"; 3 private static String BASE_URL = "http://"+HOST+"/"; 4 public static void main(String[] args) throws ClientProtocolException, IOException 5 CloseableHttpClient httpClient = HttpClients.createDefault(); 6 /// 設置GET請求參數，URL一定要以"http://"開頭 7 HttpGet getReq = new HttpGet(BASE_URL); 8 /// 設置請求報頭，模擬Chrome瀏覽器 9 getReq.addHeader("Accept", "application/json, text/javascript, */*; q=0.01"); 10 getReq.addHeader("Accept-Encoding", "gzip,deflate,sdch"); 11 getReq.addHeader("Accept-Language", "zh-CN,zh;q=0.8"); 12 getReq.addHeader("Content-Type", "text/html; charset=UTF-8"); 13 getReq.addHeader("Host", HOST); 14 getReq.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36"); 15 /// 發送GET請求 16 CloseableHttpResponse rep = httpClient.execute(getReq); 17 /// 從HTTP響應中取出頁面內容 18 HttpEntity repEntity = rep.getEntity(); 19 String content = EntityUtils.toString(repEntity); 20 /// 打印出頁面的內容： 21 System.out.println(content); 22 /// 關閉連接 23 rep.close(); 24 httpClient.close(); 25 } 26 }

? ? ? ? 得到的頁面內容：

1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 2  3 <html xmlns="http://www.w3.org/1999/xhtml"> 4 <head> 5 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> 6 <title>WWW.SINA.COM</title> 7 <meta name="keywords" content="sina, ??°???" /> 8 <meta name="description" content="??°???é??é??" /> 9 10 <style type="text/css"> 11  48 </style> 49 50  51 <script type="text/javascript" src="http://ui.sina.com/assets/js/swfobject.js"></script> 52 53  54 <script type="text/javascript"> 55 var flashvars = {}; 56 var params = {}; 57 params.base = ""; 58 params.menu = "true"; 59 params.scale = "noscale"; 60 params.bgcolor = "#fff"; 61 params.quality = "best"; 62 // params.allowfullscreen = "true"; 63 params.salign = "c"; 64 params.wmode = "window"; 65 var attributes = {}; 66 swfobject.embedSWF("http://ui.sina.com/rm/toyota/091110/toyota_120x60_4_091110.swf", "btn5", "120", "60", "9.0.0", "expressInstall.swf", flashvars, params, attributes); 67 68 </script> 69  70 71 </head> 72 <body> 73 74  75 <div style='position:absolute;top:0;left:0;width:0;height:0;z-index:1'><div style='position:absolute;top:0;left:0;width:1;height:1;'><iframe id='SUDA_FC' src='' width=1 height=1 SCROLLING=NO FRAMEBORDER=0></iframe></div><div style='position:absolute;top:0;left:0;width:0;height:0;visibility:hidden' id='SUDA_CS_DIV'></div></div> 76 <script type="text/javascript"> 77 // 80 </script> 81 <script type="text/javascript"> 82 // 85 </script> 86 <noScript> 87 <div style='position:absolute;top:0;left:0;width:0;height:0;visibility:hidden'><img width=0 height=0 src='http://beacon.sina.com.cn/a.gif?noScript' border='0' alt='' /></div> 88 </noScript> 89  90 91 <div id="wrap"> 92  93 <div id="header"> 94 <h1><em>??°???????????±???é???§?</em></h1> 95 <ul> 96 <li><a href="http://english.sina.com/index.html" onclick="_S_uaTrack('global_guide', 'english');">Sina English</a></li> 97 </ul> 98 <div class="clearDiv"></div> 99 </div> 100 101  102 <div id="map"> 103 <img src="http://ui.sina.com/assets/img/www/worldmap.jpg" alt="" name="map1" width="775" height="248" border="0" usemap="#Map1" id="Map1" /> 104 105 <map name="Map1" id=""> 106 <area shape="rect" coords="173,81,299,137" href="http://home.sina.com" target="_self" alt="????????°???" title="????????°???" onclick="_S_uaTrack('global_guide', 'us');" /> 107 <area shape="rect" coords="468,81,572,129" href="http://www.sina.com.cn" target="_self" alt="????????°???" title="????????°???" onclick="_S_uaTrack('global_guide', 'beijing');" /> 108 <area shape="rect" coords="482,145,578,184" href="http://www.sina.com.hk" target="_self" alt="é???????°???" title="é???????°???" onclick="_S_uaTrack('global_guide', 'hongkong');" /> 109 <area shape="rect" coords="658,123,755,162" href="http://www.sina.com.tw" target="_self" alt="??°?????°???" title="??°?????°???" onclick="_S_uaTrack('global_guide', 'taipei');" /> 110 </map> 111 </div> 112 113  114 <div id="channel"> 115 <img src="http://ui.sina.com/assets/img/www/categories-120918.gif" alt="" width="775" height="44" border="0" usemap="#Map4" id="Map4" /> 116 117 <map name="Map4" id=""> 118 <area shape="rect" target="_self" alt="??????" coords="4,3,76,35" href="http://us.weibo.com" onclick="_S_uaTrack('global_guide', 'weibo');" /> 119 <area shape="rect" target="_self" alt="??????" coords="95,3,166,37" href="http://google.sina.com/" onclick="_S_uaTrack('global_guide', 'search');" /> 120 <area shape="rect" target="_self" alt="è??é??" coords="171,2,241,38" href="http://video.sina.com" onclick="_S_uaTrack('global_guide', 'video');" /> 121 <area shape="rect" target="_self" alt="??¤???" coords="257,3,328,39" href="http://match.sina.com/" onclick="_S_uaTrack('global_guide', 'match');" /> 122 <area shape="rect" target="_self" alt="???é??" coords="432,3,496,36" href="http://travel.sina.com/" onclick="_S_uaTrack('global_guide', 'travel');" /> 123 <area shape="rect" target="_self" alt="é??é??" coords="509,2,582,35" href="http://yp.sina.com/" onclick="_S_uaTrack('global_guide', 'yellow');" /> 124 <area shape="rect" target="_self" alt="?????????" coords="590,2,679,33" href="http://sina.echineselearning.com/" onclick="_S_uaTrack('global_guide', 'chinese');" /> 125 <area shape="rect" target="_self" alt="è?????" coords="335,3,417,38" href="http://bbs.sina.com/" onclick="_S_uaTrack('global_guide', 'bbs');" /> 126 <area shape="rect" target="_self" alt="??????" coords="688,1,772,35" href="http://deals.sina.com" onclick="_S_uaTrack('global_guide', 'deals');" /> 127 </map> 128 </div> 129 130  131 <div id="ads"> 132 <ul> 133 <li class="bnr728"> 134 <script type="text/javascript"> 135 //<![CDATA[ 136 ord = window.ord || Math.floor(Math.random()*1E16); 137 document.write('<script type="text/javascript" src="http://ad.doubleclick.net/adj/us.homepage/;pos=top;sz=728x90;ord=' + ord + '?"><\/script>'); 138 //]]> 139 </script> 140 <noscript><a href="http://ad.doubleclick.net/jump/us.homepage/;pos=top;sz=728x90;ord=123456789?" target="_blank" ><img src="http://ad.doubleclick.net/ad/us.homepage/;pos=top;sz=728x90;ord=123456789?" border="0" alt="" /></a></noscript> 141  142 143 </li> 144 145 <li class="bnr120"><script type="text/javascript" src="http://dailynews.sina.com/gb/ads/www/120_60/2.js"></script></li> 146 <li class="bnr120"><script type="text/javascript" src="http://dailynews.sina.com/gb/ads/www/120_60/3.js"></script></li> 147 <li class="bnr120"><script type="text/javascript" src="http://dailynews.sina.com/gb/ads/www/120_60/4.js"></script></li> 148 <li class="bnr120"><script type="text/javascript" src="http://dailynews.sina.com/gb/ads/www/120_60/5.js"></script></li> 149 <li class="bnr120"><script type="text/javascript" src="http://dailynews.sina.com/gb/ads/www/120_60/6.js"></script></li> 150 151 <li class="bnr120"><script type="text/javascript" src="http://dailynews.sina.com/gb/ads/www/120_60/7.js"></script></li> 152 <li class="bnr120"><script type="text/javascript" src="http://dailynews.sina.com/gb/ads/www/120_60/8.js"></script></li> 153 <li class="bnr120"><script type="text/javascript" src="http://dailynews.sina.com/gb/ads/www/120_60/9.js"></script></li> 154 <li class="bnr120"><script type="text/javascript" src="http://dailynews.sina.com/gb/ads/www/120_60/10.js"></script></li> 155 <li class="bnr120"><script type="text/javascript" src="http://dailynews.sina.com/gb/ads/www/120_60/11.js"></script></li> 156 157 </ul> 158 159 <div class="clearDiv"></div> 160 </div> 161  162 163  164 <div id="footer"> 165 <ul> 166 <li><a href="http://corp.sina.com.cn/eng/">About SINA</a></li> 167 <li>|</li> 168 <li><a href="http://corp.sina.com.cn/eng/sina_rela_eng.htm">Investor</a></li> 169 <li>|</li> 170 <li><a href="http://mediakit.sina.com/">Media Kit</a></li> 171 <li>|</li> 172 <li><a href="http://mediakit.sina.com/contact.html">Comments or Question?</a></li> 173 <br /><br /> 174 <li class="copyright">Copyright © 1996-2015 SINA Corporation, All Rights Reserved</li> 175 </ul> 176 </div> 177 178 </div> 179 180  181 <div id="flvideo"> 182 <script type="text/javascript" src="http://dailynews.sina.com/gb/ads/common/floatingvideo.js"></script> 183 </div> 184 185  186 <script type="text/javascript" src="//secure-us.imrworldwide.com/v60.js"></script> 187 <script type="text/javascript"> 188 var pvar = { cid: "us-sina", content: "0", server: "secure-us" }; 189 var feat = { surveys_enabled: 1, sample_rate: 0.1 }; 190 var trac = nol_t(pvar, feat); 191 trac.record().post().do_sample(); 192 </script> 193 <noscript> 194 <div> 195 <img src="//secure-us.imrworldwide.com/cgi-bin/m?ci=us-sina&cg=0&cc=1&ts=noscript" width="1" height="1" alt="" /> 196 </div> 197 </noscript> 198  199 200 </body> 201 </html> HTML Code

? ? ? ? OK，這樣就可以抓到網站主頁的數據了。現在的HttpClient對于gzip格式的響應解析做得很好，在內部就解壓縮了，不需要使用者做特殊處理。

??????? 后續還需要做一個桌面的應用，能夠隔幾分鐘輪詢頁面，并將所需部分內容是否更新的狀態通知給用戶的功能。

轉載于:https://www.cnblogs.com/dsdk2008/p/4745243.html

總結

以上是生活随笔為你收集整理的HttpClient v4.5 简单抓取主页数据的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇：打勾显示输入的密码 --EditText
下一篇：国乒队长马龙保研北京体育大学：世界排名第