Scikit-learn文本聚类实例分析

Scikit-learn文本聚类实例分析，很多新手对此不是很清楚，为了帮助大家解决这个难题，下面小编将为大家详细讲解，有这方面需求的人可以来学习下，希望你能有所收获。

# -*- coding=utf-8 -*-"""text category"""from sklearn.datasets import fetch_20newsgroupsfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.feature_extraction.text import TfidfTransformerfrom sklearn.naive_bayes import MultinomialNBcategories = ['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.med']twenty_train = fetch_20newsgroups(subset='train', categories=categories, shuffle=True, random_state=42)print len(twenty_train.data)len(twenty_train.filenames)count_vect = CountVectorizer()X_train_counts = count_vect.fit_transform(twenty_train.data)print X_train_counts.shapeprint count_vect.vocabulary_.get('algorithm')tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)X_train_tf = tf_transformer.transform(X_train_counts)print X_train_tf.shapetfidf_transformer = TfidfTransformer()X_train_tfidf = tf_transformer.fit_transform(X_train_counts)print X_train_tfidf.shapeclf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)docs_new = ['God is love', 'OpenGl on the Gpu is fast']X_new_counts = count_vect.transform(docs_new)X_new_tfidf = tfidf_transformer.fit_transform(X_new_counts)predicted = clf.predict(X_new_tfidf)for doc, category in zip(docs_new, predicted):    print '%r=>%s' % (doc, twenty_train.target_names[category]

对fetch_20newsgroups中的2257条文档进行分类

1. 统计每个词出现的次数

2. 用tf-idf统计词频，tf是在一个文档里每个单词出现的次数除以文档的单词总数，idf是总的文档数除以包含该单词的文档数，再取对数；tf * idf就是这里用到的值，值越大表明单词越重要，或越相关。

例子具体做法：

1. 先计算了每个单词出现的次数

2. 然后计算了tf-idf值

3. 然后带入模型进行训练

4. 最后预测了两个新文档的类型

结果：

'God is love'=> soc.religion.christian'OpenGL on the GPU is fast'=> comp.graphics

看完上述内容是否对您有帮助呢？如果还想对相关知识有进一步的了解或阅读更多相关文章，请关注行业资讯频道，感谢您对的支持。