當(dāng)前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

毕设项目 - 大数据+爬虫疫情分析可视化

發(fā)布時(shí)間：2023/12/10 编程问答 30 豆豆

生活随笔收集整理的這篇文章主要介紹了毕设项目 - 大数据+爬虫疫情分析可视化小編覺得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.

文章目錄

0 前言
1 課題介紹
2 運(yùn)行效果
3 關(guān)鍵代碼
- 3.1 數(shù)據(jù)爬蟲
- 3.2 可視化部分
4 最后

0 前言

🔥 這兩年開始畢業(yè)設(shè)計(jì)和畢業(yè)答辯的要求和難度不斷提升，傳統(tǒng)的畢設(shè)題目缺少創(chuàng)新和亮點(diǎn)，往往達(dá)不到畢業(yè)答辯的要求，這兩年不斷有學(xué)弟學(xué)妹告訴學(xué)長(zhǎng)自己做的項(xiàng)目系統(tǒng)達(dá)不到老師的要求。

為了大家能夠順利以及最少的精力通過畢設(shè)，學(xué)長(zhǎng)分享優(yōu)質(zhì)畢業(yè)設(shè)計(jì)項(xiàng)目，今天要分享的是

🚩 基于python/大數(shù)據(jù)的疫情分析與可視化系統(tǒng)

🥇學(xué)長(zhǎng)這里給一個(gè)題目綜合評(píng)分(每項(xiàng)滿分5分)

難度系數(shù)：4分
工作量：4分
創(chuàng)新點(diǎn)：3分

🧿 選題指導(dǎo), 項(xiàng)目分享：

https://gitee.com/yaa-dc/warehouse-1/blob/master/python/README.md

1 課題介紹

全球Covid-19大危機(jī)影響我們的生活，我們的出行、交流、教育、經(jīng)濟(jì)等都發(fā)生了巨大的變化，全球疫情大數(shù)據(jù)可視化分析與展示，可用于社會(huì)各界接入疫情數(shù)據(jù)，感知疫情相關(guān)情況的實(shí)時(shí)交互式態(tài)勢(shì)，是重要的疫情分析、防控決策依據(jù)。

我國(guó)爆發(fā)的疫情，對(duì)我們的日常生活帶來了極大的影響，疫情嚴(yán)重期間，大家都談“疫”色變，大家對(duì)于了解疫情的情況具有巨大的需求；并且，目前來看我國(guó)仍然存在疫情二次爆發(fā)的可能，大家對(duì)于疫情的情況跟蹤也急于了解。

基于這個(gè)情況，學(xué)長(zhǎng)對(duì)疫情的數(shù)據(jù)進(jìn)行了爬取和可視化的展示和疫情的追蹤，也就是學(xué)長(zhǎng)設(shè)計(jì)的作品。

2 運(yùn)行效果

3 關(guān)鍵代碼

PS：篇幅有限，學(xué)長(zhǎng)僅展示部分關(guān)鍵代碼

3.1 數(shù)據(jù)爬蟲

疫情數(shù)據(jù)爬蟲，就是給網(wǎng)站發(fā)起請(qǐng)求，并從響應(yīng)中提取需要的數(shù)據(jù)

1、發(fā)起請(qǐng)求，獲取響應(yīng)

通過http庫，對(duì)目標(biāo)站點(diǎn)進(jìn)行請(qǐng)求。等同于自己打開瀏覽器輸入網(wǎng)址
常用庫：urllib、requests
服務(wù)器會(huì)返回請(qǐng)求的內(nèi)容一般為：HTML、文檔、JSON字符串等

2、解析內(nèi)容

尋找自己需要的信息，也就是利用正則表達(dá)式或者其他庫提取目標(biāo)信息
常用庫：re、beautifulsoup4

3、保存數(shù)據(jù)

將解析到的數(shù)據(jù)持久化到數(shù)據(jù)庫中

import pymysql import time import json import traceback #追蹤異常 import requests def get_tencent_data(): """:return: 返回歷史數(shù)據(jù)和當(dāng)日詳細(xì)數(shù)據(jù)"""url = ''url_his=''#最基本的反爬蟲headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36',}r = requests.get(url, headers) #使用requests請(qǐng)求res = json.loads(r.text) # json字符串轉(zhuǎn)字典data_all = json.loads(res['data'])#再加上history的配套東西r_his=requests.get(url_his,headers)res_his=json.loads(r_his.text)data_his=json.loads(res_his['data'])history = {} # 歷史數(shù)據(jù)for i in data_his["chinaDayList"]:ds = "2020." + i["date"]tup = time.strptime(ds, "%Y.%m.%d")ds = time.strftime("%Y-%m-%d", tup) # 改變時(shí)間格式,不然插入數(shù)據(jù)庫會(huì)報(bào)錯(cuò)，數(shù)據(jù)庫是datetime類型confirm = i["confirm"]suspect = i["suspect"]heal = i["heal"]dead = i["dead"]history[ds] = {"confirm": confirm, "suspect": suspect, "heal": heal, "dead": dead}for i in data_his["chinaDayAddList"]:ds = "2020." + i["date"]tup = time.strptime(ds, "%Y.%m.%d")ds = time.strftime("%Y-%m-%d", tup)confirm = i["confirm"]suspect = i["suspect"]heal = i["heal"]dead = i["dead"]history[ds].update({"confirm_add": confirm, "suspect_add": suspect, "heal_add": heal, "dead_add": dead})details = [] # 當(dāng)日詳細(xì)數(shù)據(jù)update_time = data_all["lastUpdateTime"]data_country = data_all["areaTree"] # list 25個(gè)國(guó)家data_province = data_country[0]["children"] # 中國(guó)各省for pro_infos in data_province:province = pro_infos["name"] # 省名for city_infos in pro_infos["children"]:city = city_infos["name"]confirm = city_infos["total"]["confirm"]confirm_add = city_infos["today"]["confirm"]heal = city_infos["total"]["heal"]dead = city_infos["total"]["dead"]details.append([update_time, province, city, confirm, confirm_add, heal, dead])return history, details

數(shù)據(jù)表結(jié)構(gòu)

history表存儲(chǔ)每日的總數(shù)據(jù)

CREATE TABLE history ( ds datetime NOT NULL COMMENT ‘日期’, confirm int(11) DEFAULT NULL COMMENT ‘累計(jì)確診’, confirm_add int(11) DEFAULT NULL COMMENT ‘當(dāng)日新增確診’, suspect int(11) DEFAULT NULL COMMENT ‘剩余疑似’, suspect_add int(11) DEFAULT NULL COMMENT ‘當(dāng)日新增疑似’, heal int(11) DEFAULT NULL COMMENT ‘累計(jì)治愈’, heal_add int(11) DEFAULT NULL COMMENT ‘當(dāng)日新增治愈’, dead int(11) DEFAULT NULL COMMENT ‘累計(jì)死亡’, dead_add int(11) DEFAULT NULL COMMENT ‘當(dāng)日新增死亡’, PRIMARY KEY (ds) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

details表存儲(chǔ)每日的詳細(xì)數(shù)據(jù)

CREATE TABLE details ( id int(11) NOT NULL AUTO_INCREMENT, update_time datetime DEFAULT NULL COMMENT ‘?dāng)?shù)據(jù)最后更新時(shí)間’, province varchar(50) DEFAULT NULL COMMENT ‘省’, city varchar(50) DEFAULT NULL COMMENT ‘市’, confirm int(11) DEFAULT NULL COMMENT ‘累計(jì)確診’, confirm_add int(11) DEFAULT NULL COMMENT ‘新增確診’, heal int(11) DEFAULT NULL COMMENT ‘累計(jì)治愈’, dead int(11) DEFAULT NULL COMMENT ‘累計(jì)死亡’, PRIMARY KEY (id) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

整體的數(shù)據(jù)庫圖表：

3.2 可視化部分

echarts繪制圖表

def get_c1_data():""":return: 返回大屏div id=c1 的數(shù)據(jù)"""# 因?yàn)闀?huì)更新多次數(shù)據(jù)，取時(shí)間戳最新的那組數(shù)據(jù)sql = "select sum(confirm)," \"(select suspect from history order by ds desc limit 1)," \"sum(heal)," \"sum(dead) " \"from details " \"where update_time=(select update_time from details order by update_time desc limit 1) "res = query(sql)res_list = [str(i) for i in res[0]]res_tuple=tuple(res_list)return res_tuple

中國(guó)疫情地圖實(shí)現(xiàn)

def get_c2_data():""":return: 返回各省數(shù)據(jù)"""# 因?yàn)闀?huì)更新多次數(shù)據(jù)，取時(shí)間戳最新的那組數(shù)據(jù)sql = "select province,sum(confirm) from details " \"where update_time=(select update_time from details " \"order by update_time desc limit 1) " \"group by province"res = query(sql)return res

全國(guó)累計(jì)趨勢(shì)

def get_l1_data():""":return:返回每天歷史累計(jì)數(shù)據(jù)"""sql = "select ds,confirm,suspect,heal,dead from history"res = query(sql)return resdef get_l2_data():""":return:返回每天新增確診和疑似數(shù)據(jù)"""sql = "select ds,confirm_add,suspect_add from history"res = query(sql)return resdef get_r1_data():""":return: 返回非湖北地區(qū)城市確診人數(shù)前5名"""sql = 'SELECT city,confirm FROM ' \'(select city,confirm from details ' \'where update_time=(select update_time from details order by update_time desc limit 1) ' \'and province not in ("湖北","北京","上海","天津","重慶") ' \'union all ' \'select province as city,sum(confirm) as confirm from details ' \'where update_time=(select update_time from details order by update_time desc limit 1) ' \'and province in ("北京","上海","天津","重慶") group by province) as a ' \'ORDER BY confirm DESC LIMIT 5'res = query(sql)return res

疫情熱搜

def get_r2_data():""":return: 返回最近的20條熱搜"""sql = 'select content from hotsearch order by id desc limit 20'res = query(sql) # 格式 (('民警抗疫一線奮戰(zhàn)16天犧牲1037364',), ('四川再派兩批醫(yī)療隊(duì)1537382',)return res

4 最后