TF-IDF

前言

前段時間，又具體看了自己以前整理的TF-IDF，這里把它發布在博客上，知識就是需要不斷的重復的，否則就感覺生疏了。

TF-IDF理解

TF-IDF（term frequency–inverse document frequency）是一種用于資訊檢索與資訊探勘的常用加權技術, TFIDF的主要思想是：如果某個詞或短語在一篇文章中出現的頻率TF高，并且在其他文章中很少出現，則認為此詞或者短語具有很好的類別區分能力，適合用來分類。TFIDF實際上是：TF * IDF，TF詞頻(Term Frequency)，IDF反文檔頻率(Inverse Document Frequency)。TF表示詞條在文檔d中出現的頻率。IDF的主要思想是：如果包含詞條t的文檔越少，也就是n越小，IDF越大，則說明詞條t具有很好的類別區分能力。如果某一類文檔C中包含詞條t的文檔數為m，而其它類包含t的文檔總數為k，顯然所有包含t的文檔數n=m + k，當m大的時候，n也大，按照IDF公式得到的IDF的值會小，就說明該詞條t類別區分能力不強。但是實際上，如果一個詞條在一個類的文檔中頻繁出現，則說明該詞條能夠很好代表這個類的文本的特征，這樣的詞條應該給它們賦予較高的權重，并選來作為該類文本的特征詞以區別與其它類文檔。這就是IDF的不足之處.

TF公式：

???????

以上式子中??是該詞在文件中的出現次數，而分母則是在文件中所有字詞的出現次數之和。

IDF公式：

??

• |D|：語料庫中的文件總數
• ：包含詞語的文件數目（即的文件數目）如果該詞語不在語料庫中，就會導致被除數為零，因此一般情況下使用

然后

TF-IDF案例

案例：假如一篇文件的總詞語數是100個，而詞語“母牛”出現了3次，那么“母牛”一詞在該文件中的詞頻就是3/100=0.03。一個計算文件頻率 (DF) 的方法是測定有多少份文件出現過“母牛”一詞，然后除以文件集里包含的文件總數。所以，如果“母牛”一詞在1,000份文件出現過，而文件總數是10,000,000份的話，其逆向文件頻率就是 lg(10,000,000 / 1,000)=4。最后的TF-IDF的分數為0.03 * 4=0.12。

TF-IDF實現（Java）

這里采用了外部插件IKAnalyzer-2012.jar，用其進行分詞，插件和測試文件可以從這里下載：點擊

具體代碼如下：

package tfidf;import java.io.*; import java.util.*;import org.wltea.analyzer.lucene.IKAnalyzer;public class ReadFiles {/*** @param args*/ private static ArrayList<String> FileList = new ArrayList<String>(); // the list of file//get list of file for the directory, including sub-directory of itpublic static List<String> readDirs(String filepath) throws FileNotFoundException, IOException{try{File file = new File(filepath);if(!file.isDirectory()){System.out.println("輸入的[]");System.out.println("filepath:" + file.getAbsolutePath());}else{String[] flist = file.list();for(int i = 0; i < flist.length; i++){File newfile = new File(filepath + "\\" + flist[i]);if(!newfile.isDirectory()){FileList.add(newfile.getAbsolutePath());}else if(newfile.isDirectory()) //if file is a directory, call ReadDirs {readDirs(filepath + "\\" + flist[i]);} }}}catch(FileNotFoundException e){System.out.println(e.getMessage());}return FileList;}//read filepublic static String readFile(String file) throws FileNotFoundException, IOException{StringBuffer strSb = new StringBuffer(); //String is constant， StringBuffer can be changed.InputStreamReader inStrR = new InputStreamReader(new FileInputStream(file), "gbk"); //byte streams to character streamsBufferedReader br = new BufferedReader(inStrR); String line = br.readLine();while(line != null){strSb.append(line).append("\r\n");line = br.readLine(); }return strSb.toString();}//word segmentationpublic static ArrayList<String> cutWords(String file) throws IOException{ArrayList<String> words = new ArrayList<String>();String text = ReadFiles.readFile(file);IKAnalyzer analyzer = new IKAnalyzer();words = analyzer.split(text);return words;}//term frequency in a file, times for each wordpublic static HashMap<String, Integer> normalTF(ArrayList<String> cutwords){HashMap<String, Integer> resTF = new HashMap<String, Integer>();for(String word : cutwords){if(resTF.get(word) == null){resTF.put(word, 1);System.out.println(word);}else{resTF.put(word, resTF.get(word) + 1);System.out.println(word.toString());}}return resTF;}//term frequency in a file, frequency of each wordpublic static HashMap<String, Float> tf(ArrayList<String> cutwords){HashMap<String, Float> resTF = new HashMap<String, Float>();int wordLen = cutwords.size();HashMap<String, Integer> intTF = ReadFiles.normalTF(cutwords); Iterator iter = intTF.entrySet().iterator(); //iterator for that get from TFwhile(iter.hasNext()){Map.Entry entry = (Map.Entry)iter.next();resTF.put(entry.getKey().toString(), Float.parseFloat(entry.getValue().toString()) / wordLen);System.out.println(entry.getKey().toString() + " = "+ Float.parseFloat(entry.getValue().toString()) / wordLen);}return resTF;} //tf times for filepublic static HashMap<String, HashMap<String, Integer>> normalTFAllFiles(String dirc) throws IOException{HashMap<String, HashMap<String, Integer>> allNormalTF = new HashMap<String, HashMap<String,Integer>>();List<String> filelist = ReadFiles.readDirs(dirc);for(String file : filelist){HashMap<String, Integer> dict = new HashMap<String, Integer>();ArrayList<String> cutwords = ReadFiles.cutWords(file); //get cut word for one file dict = ReadFiles.normalTF(cutwords);allNormalTF.put(file, dict);} return allNormalTF;}//tf for all filepublic static HashMap<String,HashMap<String, Float>> tfAllFiles(String dirc) throws IOException{HashMap<String, HashMap<String, Float>> allTF = new HashMap<String, HashMap<String, Float>>();List<String> filelist = ReadFiles.readDirs(dirc);for(String file : filelist){HashMap<String, Float> dict = new HashMap<String, Float>();ArrayList<String> cutwords = ReadFiles.cutWords(file); //get cut words for one file dict = ReadFiles.tf(cutwords);allTF.put(file, dict);}return allTF;}public static HashMap<String, Float> idf(HashMap<String,HashMap<String, Float>> all_tf){HashMap<String, Float> resIdf = new HashMap<String, Float>();HashMap<String, Integer> dict = new HashMap<String, Integer>();int docNum = FileList.size();for(int i = 0; i < docNum; i++){HashMap<String, Float> temp = all_tf.get(FileList.get(i));Iterator iter = temp.entrySet().iterator();while(iter.hasNext()){Map.Entry entry = (Map.Entry)iter.next();String word = entry.getKey().toString();if(dict.get(word) == null){dict.put(word, 1);}else {dict.put(word, dict.get(word) + 1);}}}System.out.println("IDF for every word is:");Iterator iter_dict = dict.entrySet().iterator();while(iter_dict.hasNext()){Map.Entry entry = (Map.Entry)iter_dict.next();float value = (float)Math.log(docNum / Float.parseFloat(entry.getValue().toString()));resIdf.put(entry.getKey().toString(), value);System.out.println(entry.getKey().toString() + " = " + value);}return resIdf;}public static void tf_idf(HashMap<String,HashMap<String, Float>> all_tf,HashMap<String, Float> idfs){HashMap<String, HashMap<String, Float>> resTfIdf = new HashMap<String, HashMap<String, Float>>();int docNum = FileList.size();for(int i = 0; i < docNum; i++){String filepath = FileList.get(i);HashMap<String, Float> tfidf = new HashMap<String, Float>();HashMap<String, Float> temp = all_tf.get(filepath);Iterator iter = temp.entrySet().iterator();while(iter.hasNext()){Map.Entry entry = (Map.Entry)iter.next();String word = entry.getKey().toString();Float value = (float)Float.parseFloat(entry.getValue().toString()) * idfs.get(word); tfidf.put(word, value);}resTfIdf.put(filepath, tfidf);}System.out.println("TF-IDF for Every file is :");DisTfIdf(resTfIdf);}public static void DisTfIdf(HashMap<String, HashMap<String, Float>> tfidf){Iterator iter1 = tfidf.entrySet().iterator();while(iter1.hasNext()){Map.Entry entrys = (Map.Entry)iter1.next();System.out.println("FileName: " + entrys.getKey().toString());System.out.print("{");HashMap<String, Float> temp = (HashMap<String, Float>) entrys.getValue();Iterator iter2 = temp.entrySet().iterator();while(iter2.hasNext()){Map.Entry entry = (Map.Entry)iter2.next(); System.out.print(entry.getKey().toString() + " = " + entry.getValue().toString() + ", ");}System.out.println("}");}}public static void main(String[] args) throws IOException {// TODO Auto-generated method stubString file = "D:/testfiles";HashMap<String,HashMap<String, Float>> all_tf = tfAllFiles(file);System.out.println();HashMap<String, Float> idfs = idf(all_tf);System.out.println();tf_idf(all_tf, idfs);}}

結果如下圖：

常見問題

沒有加入lucene jar包

?

?

lucene包和je包版本不適合

轉載于:https://www.cnblogs.com/ywl925/p/3275878.html

總結

以上是生活随笔為你收集整理的TF-IDF理解及其Java实现的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。