首先要声明,如果条件允许,不要在windows下做类似的事情,我这里是在折腾。
如果只需要下载代码,相应的代码,我已经上传了github,可以在这里下载到:
word2vec_win32
编译工具为:VS2013
具体的做法为:
1、到google code下载代码https://code.google.com/p/word2vec/
2、根据makefile,创建VS2013工程
3、进行调整,保证编译成功
3.1、所有c文件,添加下面的宏定义
#define _CRT_SECURE_NO_WARNINGS
3.2、将部分const修改为define,比如
#define MAX_STRING 100
3.3、用_aligned_malloc函数,替换posix_memalign函数
#define posix_memalign(p, a, s) (((*(p)) = _aligned_malloc((s), (a))), *(p) ?0 :errno)
3.4、下载windows下的pthread库,pthreads-win32,并修改include及link配置
3.5、编译成功
4、可执行文件说明
word2vec:词转向量,或者进行聚类
word2phrase:词转词组,用于预处理,可重复使用(运行一遍则生成2个词的短语,运行两遍则形成4个词的短语)
compute-accuracy:校验模型精度
distance:输入一个词A,返回最相近的词(A=》?)
word-analogy:输入三个词A,B,C,返回(如果A=》B,C=》?)
5、进行测试
5.1下载测试资料
http://mattmahoney.net/dc/text8.zip
5.2建立模型
>word2vec -train text8 -output vectors.bin -cbow 1 -size 200 -window 8 -negative 25 -hs 0 -sample 1e-4 -threads 20 -binary 1 -iter 15 Starting training using file text8 Vocab size: 71291 Words in train file: 16718843 Alpha: 0.000005 Progress: 100.10% Words/thread/sec: 13.74k
5.3校验模型精度
>compute-accuracy vectors.bin 30000 < questions-word s.txt capital-common-countries: ACCURACY TOP1: 80.83 % (409 / 506) Total accuracy: 80.83 % Semantic accuracy: 80.83 % Syntactic accuracy: -1.#J % capital-world: ACCURACY TOP1: 62.65 % (884 / 1411) Total accuracy: 67.45 % Semantic accuracy: 67.45 % Syntactic accuracy: -1.#J % currency: ACCURACY TOP1: 23.13 % (62 / 268) Total accuracy: 62.01 % Semantic accuracy: 62.01 % Syntactic accuracy: -1.#J % city-in-state: ACCURACY TOP1: 46.85 % (736 / 1571) Total accuracy: 55.67 % Semantic accuracy: 55.67 % Syntactic accuracy: -1.#J % family: ACCURACY TOP1: 77.45 % (237 / 306) Total accuracy: 57.31 % Semantic accuracy: 57.31 % Syntactic accuracy: -1.#J % gram1-adjective-to-adverb: ACCURACY TOP1: 19.44 % (147 / 756) Total accuracy: 51.37 % Semantic accuracy: 57.31 % Syntactic accuracy: 19.44 % gram2-opposite: ACCURACY TOP1: 24.18 % (74 / 306) Total accuracy: 49.75 % Semantic accuracy: 57.31 % Syntactic accuracy: 20.81 % gram3-comparative: ACCURACY TOP1: 64.92 % (818 / 1260) Total accuracy: 52.74 % Semantic accuracy: 57.31 % Syntactic accuracy: 44.75 % gram4-superlative: ACCURACY TOP1: 39.53 % (200 / 506) Total accuracy: 51.77 % Semantic accuracy: 57.31 % Syntactic accuracy: 43.81 % gram5-present-participle: ACCURACY TOP1: 40.32 % (400 / 992) Total accuracy: 50.33 % Semantic accuracy: 57.31 % Syntactic accuracy: 42.91 % gram6-nationality-adjective: ACCURACY TOP1: 84.46 % (1158 / 1371) Total accuracy: 55.39 % Semantic accuracy: 57.31 % Syntactic accuracy: 53.88 % gram7-past-tense: ACCURACY TOP1: 39.79 % (530 / 1332) Total accuracy: 53.42 % Semantic accuracy: 57.31 % Syntactic accuracy: 51.00 % gram8-plural: ACCURACY TOP1: 61.39 % (609 / 992) Total accuracy: 54.11 % Semantic accuracy: 57.31 % Syntactic accuracy: 52.38 % gram9-plural-verbs: ACCURACY TOP1: 33.38 % (217 / 650) Total accuracy: 53.01 % Semantic accuracy: 57.31 % Syntactic accuracy: 50.86 % Questions seen / total: 12227 19544 62.56 %
5.4查找关系最近的单词
>distance vectors.bin Enter word or sentence (EXIT to break): china Word: china Position in vocabulary: 486 Word Cosine distance ------------------------------------------------------------------------ taiwan 0.649276 japan 0.624836 hainan 0.567946 kalmykia 0.562871 tibet 0.562600 prc 0.553833 tuva 0.553255 korea 0.552685 chinese 0.545661 xiamen 0.542703 liao 0.542607 jiang 0.540888 manchuria 0.540783 wuhan 0.537735 yunnan 0.535809 hunan 0.535770 hangzhou 0.524340 yong 0.523802 sichuan 0.517254 guangdong 0.514874 liang 0.511881 jin 0.511389 india 0.508853 xinjiang 0.505971 taiwanese 0.503072 qing 0.502909 shanghai 0.502771 shandong 0.499169 jiangxi 0.495940 nanjing 0.492893 guangzhou 0.492788 zhao 0.490396 shenzhen 0.489658 singapore 0.489428 hubei 0.488228 harbin 0.488112 liaoning 0.484283 zhejiang 0.484192 joseon 0.483718 mongolia 0.481411 Enter word or sentence (EXIT to break):
5.5根据A=>B,得到C=>?
>word-analogy vectors.bin Enter three words (EXIT to break): china beijing canada Word: china Position in vocabulary: 486 Word: beijing Position in vocabulary: 3880 Word: canada Position in vocabulary: 474 Word Distance ------------------------------------------------------------------------ toronto 0.624131 montreal 0.559667 mcgill 0.519338 calgary 0.518366 ryerson 0.515524 ottawa 0.515316 alberta 0.509334 edmonton 0.498436 moncton 0.488861 quebec 0.487712 canadian 0.475655 saskatchewan 0.460744 fredericton 0.460354 ontario 0.458213 montrealers 0.435611 vancouver 0.429893 saskatoon 0.416954 dieppe 0.404408 iqaluit 0.401143 canadians 0.398137 winnipeg 0.397547 labatt 0.393893 city 0.386245 bilingualism 0.386245 columbia 0.384754 provincial 0.383439 banff 0.382603 metro 0.382367 molson 0.379343 nunavut 0.375992 montr 0.373883 francophones 0.373512 brunswick 0.364261 manitoba 0.360447 bec 0.359977 francophone 0.358556 leafs 0.353035 ellensburg 0.352787 curling 0.351973 cdn 0.347580 Enter three words (EXIT to break):
5.6进行聚类,输出结果(classes为0时,就是向量输出了)
>word2vec -train text8 -output classes.txt -cbow 1 -size 200 -window 8 -negative 25 -hs 0 -sample 1e-4 -threads 20 -iter 15 -classes 500 Starting training using file text8 Vocab size: 71291 Words in train file: 16718843 Alpha: 0.000005 Progress: 100.10% Words/thread/sec: 14.72k
5.7原来程序中,还有三个测试脚本,是处理词组的,由于要用到linux命令sed,awk等,大家还是到Cygwin或MinGW下运行吧
VS2015使用,并要生成x64 release的话,需要修改一些部分:
1.在pthread.h中修改:
#if _MSC_VER 链接器->附加库目录 -> ..\pthreads-win32\lib\x64;
主要是之前用的这个人的项目:
https://github.com/zhangyafeikimi/word2vec-win32
在有一次实验中发现结果和原先的差别太大了,所以试试neohope的吧
有小于号的那段内容被过滤掉了…
#if _MSC_VER 小于 1900
struct timespec {
time_t tv_sec;
long tv_nsec;
};
#endif
2.在属性中修改 链接器…