原文章地址：
Word2Vec (Part 1): NLP With Deep Learning with Tensorflow (Skip-gram)
Word2Vec (Part 2): NLP With Deep Learning with Tensorflow (CBOW）
上一篇文章，也就是Skip-gram模型，点这里
下面带来CBOW模型的讲解：
CBOW是什么？
CBOW是什么呢？它的全称是 continuous bag-of-words ，中文是连续词袋模型。它的框架可以说就是将skip-gram模型倒转过来。在skip-gram模型中，是根据目标词预测上下文。而CBOW模型，则是根据上下文预测目标词。
为什么要使用CBOW模型？
既然我们已经有了skip-gram模型，为什么我们还要学习CBOW模型呢？原因就是CBOW模型的表现更加优秀。一部分的原因在于CBOW模型的 inputs 更加丰富。换句化说，假定如下句子：the dog barked at the mailman ， 在skip-gram模型中，输入输出为 (input:'dog',output:'barked') ，而在CBOW模型中，将有以下输入输出：(input:['the','barked','at'],output:'dog') 。可以看出在CBOW中，只有当 [the, barked, at] 等词准确地出现了，才会预测dog 出现，而不像skip-gram那样，只能预测出 dog 将出现在 barked 附近。
CBOW模型
CBOW的概念模型看起来就像倒过来的skip-gram模型一样。尽管看起来如此，但是CBOW模型和skip-gram模型并不是对称的。下面是模型的框架图。
注意到，因为实现框架图跟skip-gram的十分相似，所以没有给出来。如何把这个概念模型如转化成实现模型呢，我们要做的就是生成 (input, output) 的 batch。换句化说，对每一列每次处理处理 b 个（b - batch大小）词（比如b x word[t-2],b x word[t-1], b x word[t+1],b x word[t+2]）。
CBOW 背后的思想是，我们使用所有 input 词的平均词向量作为学习模型的输入。
数据生成
现在，生成数据的函数需要做一些小小的修改来适应CBOW模型。下面是修改后的代码：
def generate_batch(batch_size, skip_window): # skip window is the amount of words we're looking at from each side of a given word # creates a single batch global data_index assert skip_window%2==1 span = 2 * skip_window + 1 # [ skip_window target skip_window ] batch = np.ndarray(shape=(batch_size,span-1), dtype=np.int32) labels = np.ndarray(shape=(batch_size, 1), dtype=np.int32) # e.g if skip_window = 2 then span = 5 # span is the length of the whole frame we are considering for a single word (left + word + right) # skip_window is the length of one side # queue which add and pop at the end buffer = collections.deque(maxlen=span) #get words starting from index 0 to span for _ in range(span): buffer.append(data[data_index]) data_index = (data_index + 1) % len(data) # num_skips => # of times we select a random word within the span? # batch_size (8) and num_skips (2) (4 times) # batch_size (8) and num_skips (1) (8 times) for i in range(batch_size): target = skip_window # target label at the center of the buffer target_to_avoid = [ skip_window ] # we only need to know the words around a given word, not the word itself # do this num_skips (2 times) # do this (1 time) # add selected target to avoid_list for next time col_idx = 0 for j in range(span): if j==span//2: continue # e.g. i=0, j=0 => 0; i=0,j=1 => 1; i=1,j=0 => 2 batch[i,col_idx] = buffer[j] # [skip_window] => middle element col_idx += 1 labels[i, 0] = buffer[target] buffer.append(data[data_index]) data_index = (data_index + 1) % len(data) assert batch.shape[0]==batch_size and batch.shape[1]== span-1可以注意到，batch 的大小变为 (b x span-1)，而修改前为（b x 1）。并且去除了num_skips.因为我们将使用 span 中所有的词。直观地说，batch 的索引（i，j）可以理解为文档labels 中第 j 个词的 i-skip_window 偏移量（ i<skip_window 时）或 i-skip_window+1 个词 （i>=skip_window 时）。举个例子，假设 skip_window=1 ， 输入句子 the dog barked at the mailman，我们将会得到，
batch: [['the','barked'],['dog','at'],['barked','the'],['at','mailman']]
labels: ['dog','barked','at','the']
训练模型
同样，训练模型的阶段也需要做一些调整。但是这也没有很复杂，我们要做的就是把 data placholder 的大小做一些调整，并且为多个输入写
写入正确的符号操作以得到其平均值。考虑到训练过程比较重要，因此我将会把代码分成几个小片段，并且从中挑取重要的来讲解。
变量初始化
首先我们要把 train_dataset 的 placeholder 改变为 （b x 2*skip_window）（记住，span-1 = 2*skip_window）。其它保持不变。
if __name__ == '__main__': batch_size = 128 embedding_size = 128 # Dimension of the embedding vector. skip_window = 1 # How many words to consider left and right. num_skips = 2 # How many times to reuse an input to generate a label. valid_size = 16 # Random set of words to evaluate similarity on. valid_window = 100 # Only pick dev samples in the head of the distribution. # pick 16 samples from 100 valid_examples = np.array(random.sample(range(valid_window), valid_size//2)) valid_examples = np.append(valid_examples,random.sample(range(1000,1000+valid_window), valid_size//2)) num_sampled = 64 # Number of negative examples to sample. graph = tf.Graph() with graph.as_default(), tf.device('/cpu:0'): # Input data. train_dataset = tf.placeholder(tf.int32, shape=[batch_size,2*skip_window]) train_labels = tf.placeholder(tf.int32, shape=[batch_size, 1]) valid_dataset = tf.constant(valid_examples, dtype=tf.int32) # Variables. # embedding, vector for each word in the vocabulary embeddings = tf.Variable(tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0)) softmax_weights = tf.Variable(tf.truncated_normal([vocabulary_size, embedding_size], stddev=1.0 / math.sqrt(embedding_size))) softmax_biases = tf.Variable(tf.zeros([vocabulary_size]))向量查找与求平均
这里我们要做些大的变动。我们要重新编写 embedding lookup 并且正确地求出它们的平均值。总来的来说，我们要查找 train_dataset (大小为 b x 2*skip_window) 的每一行，查找行中词ID对应的向量。然后将这些向量保存在临时变量（ embedding_i ）中，在把这些向量连接起来称为复合向量（embeds）(大小为 2*skip_window x b x D)，进而在 axis 0 上求得 reduce mean 。最终我们可以对 data 的每个 batch 生成 train_labels 中词相应上下文的平均向量。
# Model. embeds = None for i in range(2*skip_window): embedding_i = tf.nn.embedding_lookup(embeddings, train_dataset[:,i]) print('embedding %d shape: %s'%(i,embedding_i.get_shape().as_list())) emb_x,emb_y = embedding_i.get_shape().as_list() if embeds is None: embeds = tf.reshape(embedding_i,[emb_x,emb_y,1]) else: embeds = tf.concat(2,[embeds,tf.reshape(embedding_i,[emb_x,emb_y,1])]) assert embeds.get_shape().as_list()[2]==2*skip_window print("Concat embedding size: %s"%embeds.get_shape().as_list()) avg_embed = tf.reduce_mean(embeds,2,keep_dims=False) print("Avg embedding size: %s"%avg_embed.get_shape().as_list()) Loss 函数以及优化现在相较于skip-gram模型，我们在CBOW模型的 sampled_softmax_loss 中使用了平均向量。代码方面没有很大的变动。
loss = tf.reduce_mean(tf.nn.sampled_softmax_loss(softmax_weights, softmax_biases, avg_embed, train_labels, num_sampled, vocabulary_size)) optimizer = tf.train.AdagradOptimizer(1.0).minimize(loss) # We use the cosine distance: norm = tf.sqrt(tf.reduce_sum(tf.square(embeddings), 1, keep_dims=True)) normalized_embeddings = embeddings / norm valid_embeddings = tf.nn.embedding_lookup(normalized_embeddings, valid_dataset) similarity = tf.matmul(valid_embeddings, tf.transpose(normalized_embeddings)) 运行程序最后我们要来让tensorflow跑起来。
with tf.Session(graph=graph) as session: tf.initialize_all_variables().run() print('Initialized') average_loss = 0 for step in range(num_steps): batch_data, batch_labels = generate_batch(batch_size, skip_window) feed_dict = {train_dataset : batch_data, train_labels : batch_labels} _, l = session.run([optimizer, loss], feed_dict=feed_dict) average_loss += l if step % 2000 == 0: if step > 0: average_loss = average_loss / 2000 # The average loss is an estimate of the loss over the last 2000 batches. print('Average loss at step %d: %f' % (step, average_loss)) average_loss = 0 # note that this is expensive (~20% slowdown if computed every 500 steps) if step % 10000 == 0: sim = similarity.eval() for i in range(valid_size): valid_word = reverse_dictionary[valid_examples[i]] top_k = 8 # number of nearest neighbors nearest = (-sim[i, :]).argsort()[1:top_k+1] log = 'Nearest to %s:' % valid_word for k in range(top_k): close_word = reverse_dictionary[nearest[k]] log = '%s %s,' % (log, close_word) print(log) final_embeddings = normalized_embeddings.eval() 结果最终我们得到的结果如下
Average loss at step 0: 7.687360 Nearest to he: annoying, menachem, publicize, unwise, skinny, attractors, devastating, declination, Nearest to is: iarc, agrarianism, revoluci, bachman, distinguish, schliemann, carbons, ne, Nearest to some: routed, oscillations, reverence, collaborating, invitational, murderous, mortimer, migratory, Nearest to only: walkway, loud, today, headshot, foundational, asceticism, tracked, hare, ... Nearest to i: intermediates, backed, techs, duly, inefficiencies, ibadi, creole, poured, Nearest to bbc: mprp, catching, slavic, mol, dorian, mining, inactivity, applet, Nearest to cost: cakes, voltages, halter, disappeared, poking, buttocks, talents, salle, Nearest to proposed: prisoners, ecuador, sorghum, complying, saturdays, positioned, probing, observables, Average loss at step 100000: 2.422888 Nearest to he: she, it, they, there, who, eventually, neighbors, theses, Nearest to is: was, has, became, remains, be, becomes, seems, cetacean, Nearest to some: many, several, certain, most, any, all, both, these, Nearest to only: settling, orchids, commutation, until, either, first, alcohols, rabba, ... Nearest to i: we, you, ii, iii, iv, they, t, lm, Nearest to bbc: news, corporation, coffers, inactivity, mprp, formatted, cara, pedestrian, Nearest to cost: cakes, length, completion, poking, measure, enforcers, parody, figurative, Nearest to proposed: introduced, discovered, foreground, suggested, dismissed, argued, ecuador, builder, 完整代码可以在这里下载： 5_word2vec_cbow.py好了，这两篇文章也翻译得差不多了，有问题欢迎留言讨论，相互学习，以后看到更多不错的blog我将会继续翻译的！～
