ML之NB、LoR：基于NB和LoR算法對Kaggle IMDB影評數(shù)據(jù)集(國外類似豆瓣電影)情感分析進行分類

注：LoR code waiting

?

?

目錄

輸出結(jié)果

核心代碼

---

?

?

輸出結(jié)果

數(shù)據(jù)集：Dataset之IMDB影評數(shù)據(jù)集：IMDB影評數(shù)據(jù)集的簡介、下載、使用方法之詳細(xì)攻略

?

核心代碼

#1、讀取數(shù)據(jù)并做一些基本的預(yù)處理(比如說把評論部分的html標(biāo)簽去掉等等) def review_to_wordlist(review):'''把IMDB的評論轉(zhuǎn)成詞序列'''review_text = BeautifulSoup(review,"lxml").get_text() # 去掉HTML標(biāo)簽，拿到內(nèi)容review_text = re.sub("[^a-zA-Z]"," ", review_text) # 用正則表達式取出符合規(guī)范的部分words = review_text.lower().split() # 小寫化所有的詞，并轉(zhuǎn)成詞listreturn words # 返回words# 使用pandas讀入訓(xùn)練和測試csv文件 train = pd.read_csv('F:/File_Python/Resources/Kaggle Film critic emotion/labeledTrainData.tsv', header=0, delimiter="\t", quoting=3) test = pd.read_csv('F:/File_Python/Resources/Kaggle Film critic emotion/testData.tsv', header=0, delimiter="\t", quoting=3 ) y_train = train['sentiment'] # 取出情感標(biāo)簽，positive/褒 或者 negative/貶 train_data = [] # 將訓(xùn)練和測試數(shù)據(jù)都轉(zhuǎn)成詞list for i in range(0,len(train['review'])):train_data.append(" ".join(review_to_wordlist(train['review'][i]))) test_data = [] for i in range(0,len(test['review'])):test_data.append(" ".join(review_to_wordlist(test['review'][i]))) print(train_data) print(y_train)#2、特征處理：從數(shù)據(jù)里面拿到有區(qū)分度的特征，采用TF-IDF向量方法 from sklearn.feature_extraction.text import TfidfVectorizer as TFIV # 初始化TFIV對象，去停用詞，加2元語言模型 tfv = TFIV(min_df=3, max_features=None, strip_accents='unicode', analyzer='word',token_pattern=r'\w{1,}', ngram_range=(1, 2), use_idf=1,smooth_idf=1,sublinear_tf=1, stop_words = 'english')X_all = train_data + test_data # 合并訓(xùn)練和測試集以便進行TFIDF向量化操作 len_train = len(train_data)tfv.fit(X_all) X_all = tfv.transform(X_all)X = X_all[:len_train] # 恢復(fù)成訓(xùn)練集和測試集部分 X_test = X_all[len_train:]# 3、利用NB算法，多項式樸素貝葉斯 from sklearn.naive_bayes import MultinomialNB as MNBmodel_NB = MNB() model_NB.fit(X, y_train) #特征數(shù)據(jù)直接灌進來 MNB(alpha=1.0, class_prior=None, fit_prior=True)from sklearn.cross_validation import cross_val_score import numpy as np print ("多項式貝葉斯分類器20折交叉驗證得分: ", np.mean(cross_val_score(model_NB, X, y_train, cv=20, scoring='roc_auc')))#4、利用LoR算法 from sklearn.linear_model import LogisticRegression as LR from sklearn.grid_search import GridSearchCV# 設(shè)定grid search的參數(shù) grid_values = {'C':[30]} # 設(shè)定打分為roc_auc model_LR = GridSearchCV(LR(penalty = 'L2', dual = True, random_state = 0), grid_values, scoring = 'roc_auc', cv = 20) # 數(shù)據(jù)灌進來 model_LR.fit(X,y_train) # 20折交叉驗證，開始漫長的等待... GridSearchCV(cv=20, estimator=LogisticRegression(C=1.0, class_weight=None, dual=True, fit_intercept=True, intercept_scaling=1, penalty='L2', random_state=0, tol=0.0001),fit_params={}, iid=True, loss_func=None, n_jobs=1,param_grid={'C': [30]}, pre_dispatch='2*n_jobs', refit=True,score_func=None, scoring='roc_auc', verbose=0) #輸出結(jié)果 print (model_LR.grid_scores_)

?

?

?

相關(guān)文章推薦

NLP系列(4)_樸素貝葉斯實戰(zhàn)與進階

?

總結(jié)

以上是生活随笔為你收集整理的ML之NB、LoR：基于NB和LoR算法对Kaggle IMDB影评数据集(国外类似豆瓣电影)情感分析进行分类的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。