pytorch做seq2seq注意力模型的翻译
以下是对pytorch 1.0版本 的seq2seq+注意力模型做法语--英语翻译的理解（这个代码在pytorch0.4上也可以正常跑）：
1 # -*- coding: utf-8 -*-
2 """
3 Translation with a Sequence to Sequence Network and Attention
4 *************************************************************
5 **Author**: `Sean Robertson <https://github.com/spro/practical-pytorch>`_
6
7 In this project we will be teaching a neural network to translate from
8 French to English.
9
10 ::
11
12 [KEY: > input, = target, < output]
13
14 > il est en train de peindre un tableau .
15 = he is painting a picture .
16 < he is painting a picture .
17
18 > pourquoi ne pas essayer ce vin delicieux ?
19 = why not try that delicious wine ?
20 < why not try that delicious wine ?
21
22 > elle n est pas poete mais romanciere .
23 = she is not a poet but a novelist .
24 < she not not a poet but a novelist .
25
26 > vous etes trop maigre .
27 = you re too skinny .
28 < you re all alone .
29
30 ... to varying degrees of success.
31
32 This is made possible by the simple but powerful idea of the `sequence
33 to sequence network <http://arxiv.org/abs/1409.3215>`__, in which two
34 recurrent neural networks work together to transform one sequence to
35 another. An encoder network condenses an input sequence into a vector,
36 and a decoder network unfolds that vector into a new sequence.
37
38 .. figure:: /_static/img/seq-seq-images/seq2seq.png
39 :alt:
40
41 To improve upon this model we'll use an `attention
42 mechanism <https://arxiv.org/abs/1409.0473>`__, which lets the decoder
43 learn to focus over a specific range of the input sequence.
44
45 **Recommended Reading:**
46
47 I assume you have at least installed PyTorch, know Python, and
48 understand Tensors:
49
50 - https://pytorch.org/ For installation instructions
51 - :doc:`/beginner/deep_learning_60min_blitz` to get started with PyTorch in general
52 - :doc:`/beginner/pytorch_with_examples` for a wide and deep overview
53 - :doc:`/beginner/former_torchies_tutorial` if you are former Lua Torch user
54
55
56 It would also be useful to know about Sequence to Sequence networks and
57 how they work:
58
59 - `Learning Phrase Representations using RNN Encoder-Decoder for
60 Statistical Machine Translation <http://arxiv.org/abs/1406.1078>`__
61 - `Sequence to Sequence Learning with Neural
62 Networks <http://arxiv.org/abs/1409.3215>`__
63 - `Neural Machine Translation by Jointly Learning to Align and
64 Translate <https://arxiv.org/abs/1409.0473>`__
65 - `A Neural Conversational Model <http://arxiv.org/abs/1506.05869>`__
66
67 You will also find the previous tutorials on
68 :doc:`/intermediate/char_rnn_classification_tutorial`
69 and :doc:`/intermediate/char_rnn_generation_tutorial`
70 helpful as those concepts are very similar to the Encoder and Decoder
71 models, respectively.
72
73 And for more, read the papers that introduced these topics:
74
75 - `Learning Phrase Representations using RNN Encoder-Decoder for
76 Statistical Machine Translation <http://arxiv.org/abs/1406.1078>`__
77 - `Sequence to Sequence Learning with Neural
78 Networks <http://arxiv.org/abs/1409.3215>`__
79 - `Neural Machine Translation by Jointly Learning to Align and
80 Translate <https://arxiv.org/abs/1409.0473>`__
81 - `A Neural Conversational Model <http://arxiv.org/abs/1506.05869>`__
82
83
84 **Requirements**
85 """
86 from __future__ import unicode_literals, print_function, division
87 from io import open
88 import unicodedata
89 import string
90 import re
91 import random
92
93 import torch
94 import torch.nn as nn
95 from torch import optim
96 import torch.nn.functional as F
97
98 device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
99
100 ######################################################################
101 # Loading data files
102 # ==================
103 #
104 # The data for this project is a set of many thousands of English to
105 # French translation pairs.
106 #
107 # `This question on Open Data Stack
108 # Exchange <http://opendata.stackexchange.com/questions/3888/dataset-of-sentences-translated-into-many-languages>`__
109 # pointed me to the open translation site http://tatoeba.org/ which has
110 # downloads available at http://tatoeba.org/eng/downloads - and better
111 # yet, someone did the extra work of splitting language pairs into
112 # individual text files here: http://www.manythings.org/anki/
113 #
114 # The English to French pairs are too big to include in the repo, so
115 # download to ``data/eng-fra.txt`` before continuing. The file is a tab
116 # separated list of translation pairs:
117 #
118 # ::
119 #
120 # I am cold. J'ai froid.
121 #
122 # .. Note::
123 # Download the data from
124 # `here <https://download.pytorch.org/tutorial/data.zip>`_
125 # and extract it to the current directory.
126
127 ######################################################################
128 # Similar to the character encoding used in the character-level RNN
129 # tutorials, we will be representing each word in a language as a one-hot
130 # vector, or giant vector of zeros except for a single one (at the index
131 # of the word). Compared to the dozens of characters that might exist in a
132 # language, there are many many more words, so the encoding vector is much
133 # larger. We will however cheat a bit and trim the data to only use a few
134 # thousand words per language.
135 #
136 # .. figure:: /_static/img/seq-seq-images/word-encoding.png
137 # :alt:
138 #
139 #
140
141
142 ######################################################################
143 # We'll need a unique index per word to use as the inputs and targets of
144 # the networks later. To keep track of all this we will use a helper class
145 # called ``Lang`` which has word → index (``word2index``) and index → word
146 # (``index2word``) dictionaries, as well as a count of each word
147 # ``word2count`` to use to later replace rare words.
148 #
149
150 SOS_token = 0
151 EOS_token = 1
152
153
154 # 每个单词需要对应唯一的索引作为稍后的网络输入和目标.为了追踪这些索引
155 # 则使用一个帮助类 Lang ，类中有 词 → 索引 (word2index) 和 索引 → 词
156 # (index2word) 的字典, 以及每个词word2count 用来替换稀疏词汇.
157
158
159 # 此处创建的Lang 对象来表示源/目标语言，它包含三部分：word2index、
160 # index2word 和word2count，分别表示单词到id、id 到单词和单词的词频。
161 # word2count的作用是用于过滤一些低频词（把它变成unknown）
162
163 class Lang:
164 def __init__(self, name):
165 self.name = name
166 self.word2index = {}
167 self.word2count = {}
168 self.index2word = {0: "SOS", 1: "EOS"}
169 self.n_words = 2 # Count SOS and EOS
170
171 def addSentence(self, sentence):
172 for word in sentence.split(' '):
173 self.addWord(word) # 用于添加单词
174
175 def addWord(self, word):
176 if word not in self.word2index: # 是不是新的词
177 # 如果不在word2index里，则需要新的定义字典
178 self.word2index[word] = self.n_words
179 self.word2count[word] = 1
180 self.index2word[self.n_words] = word
181 self.n_words += 1 # 相当于每次index+1
182 else:
183 self.word2count[word] += 1 # 计算每次词的个数
184
185
186 ######################################################################
187 # The files are all in Unicode, to simplify we will turn Unicode
188 # characters to ASCII, make everything lowercase, and trim most
189 # punctuation.
190 #
191
192 # Turn a Unicode string to plain ASCII, thanks to
193 # http://stackoverflow.com/a/518232/2809427
194
195 # 此处是为了将Unicode字符串转换为纯ASCII
196 # 原文件是Unicode编码
197 def unicodeToAscii(s):
198 return ''.join(
199 c for c in unicodedata.normalize('NFD', s)
200 if unicodedata.category(c) != 'Mn'
201 )
202
203
204 # Lowercase, trim, and remove non-letter characters
205
206 # 小写,修剪和删除非字母字符
207 def normalizeString(s):
208 s = unicodeToAscii(s.lower().strip())
209 s = re.sub(r"([.!?])", r" \1", s)
210 s = re.sub(r"[^a-zA-Z.!?]+", r" ", s)
211 return s
212
213
214 ######################################################################
215 # To read the data file we will split the file into lines, and then split
216 # lines into pairs. The files are all English → Other Language, so if we
217 # want to translate from Other Language → English I added the ``reverse``
218 # flag to reverse the pairs.
219 #
220
221
222 # 要读取数据文件,我们将把文件分成行,然后将行成对分开. 这些文件
223 # 都是英文→其他语言,所以如果我们想从其他语言翻译→英文,我们添加了
224 # 翻转标志 reverse来翻转词语对.
225 def readLangs(lang1, lang2, reverse=False):
226 print("Reading lines...")
227
228 # Read the file and split into lines
229 # 读取文件并按行分开
230 lines = open('data/%s-%s.txt' % (lang1, lang2), encoding='utf-8'). \
231 read().strip().split('\n')
232
233 # Split every line into pairs and normalize
234 # 将每一行分成两列并进行标准化
235 pairs = [[normalizeString(s) for s in l.split('\t')] for l in lines]
236
237 # Reverse pairs, make Lang instances
238 # 翻转对,Lang实例化
239 if reverse:
240 pairs = [list(reversed(p)) for p in pairs]
241 input_lang = Lang(lang2)
242 output_lang = Lang(lang1)
243 else:
244 input_lang = Lang(lang1)
245 output_lang = Lang(lang2)
246
247 return input_lang, output_lang, pairs
248
249
250 ######################################################################
251 # Since there are a *lot* of example sentences and we want to train
252 # something quickly, we'll trim the data set to only relatively short and
253 # simple sentences. Here the maximum length is 10 words (that includes
254 # ending punctuation) and we're filtering to sentences that translate to
255 # the form "I am" or "He is" etc. (accounting for apostrophes replaced
256 # earlier).
257 #
258
259 # 由于例句较多,为了方便快速训练,则会将数据集裁剪为相对简短的句子.
260 # 这里的单词的最大长度是10词(包括结束标点符号),
261 # 保留”I am” 和”He is” 开头的数据
262
263 MAX_LENGTH = 10
264
265 eng_prefixes = (
266 "i am ", "i m ",
267 "he is", "he s ",
268 "she is", "she s",
269 "you are", "you re ",
270 "we are", "we re ",
271 "they are", "they re "
272 )
273
274
275 def filterPair(p):
276 return len(p[0].split(' ')) < MAX_LENGTH and \
277 len(p[1].split(' ')) < MAX_LENGTH and \
278 p[1].startswith(eng_prefixes)
279 # 是否满足长度
280
281
282 def filterPairs(pairs):
283 return [pair for pair in pairs if filterPair(pair)]
284
285
286 ######################################################################
287 # The full process for preparing the data is:
288 #
289 # - Read text file and split into lines, split lines into pairs
290 # - Normalize text, filter by length and content
291 # - Make word lists from sentences in pairs
292 #
293
294 def prepareData(lang1, lang2, reverse=False):
295 input_lang, output_lang, pairs = readLangs(lang1, lang2, reverse)
296 # 读入数据lang1,lang2,并翻转
297 print("Read %s sentence pairs" % len(pairs))
298 # 一共读入了多少对
299 pairs = filterPairs(pairs)
300 # 符合条件的配对有多少对
301 print("Trimmed to %s sentence pairs" % len(pairs))
302 print("Counting words...")
303 for pair in pairs:
304 input_lang.addSentence(pair[0])
305 output_lang.addSentence(pair[1])
306 print("Counted words:")
307 print(input_lang.name, input_lang.n_words)
308 print(output_lang.name, output_lang.n_words)
309 return input_lang, output_lang, pairs
310
311
312 # 对数据进行预处理
313 input_lang, output_lang, pairs = prepareData('eng', 'fra', True)
314 print(random.choice(pairs)) # 随机展示一对
315
316
317 ######################################################################
318 # The Seq2Seq Model
319 # =================
320 #
321 # A Recurrent Neural Network, or RNN, is a network that operates on a
322 # sequence and uses its own output as input for subsequent steps.
323 #
324 # A `Sequence to Sequence network <http://arxiv.org/abs/1409.3215>`__, or
325 # seq2seq network, or `Encoder Decoder
326 # network <https://arxiv.org/pdf/1406.1078v3.pdf>`__, is a model
327 # consisting of two RNNs called the encoder and decoder. The encoder reads
328 # an input sequence and outputs a single vector, and the decoder reads
329 # that vector to produce an output sequence.
330 #
331 # .. figure:: /_static/img/seq-seq-images/seq2seq.png
332 # :alt:
333 #
334 # Unlike sequence prediction with a single RNN, where every input
335 # corresponds to an output, the seq2seq model frees us from sequence
336 # length and order, which makes it ideal for translation between two
337 # languages.
338 #
339 # Consider the sentence "Je ne suis pas le chat noir" → "I am not the
340 # black cat". Most of the words in the input sentence have a direct
341 # translation in the output sentence, but are in slightly different
342 # orders, e.g. "chat noir" and "black cat". Because of the "ne/pas"
343 # construction there is also one more word in the input sentence. It would
344 # be difficult to produce a correct translation directly from the sequence
345 # of input words.
346 #
347 # With a seq2seq model the encoder creates a single vector which, in the
348 # ideal case, encodes the "meaning" of the input sequence into a single
349 # vector — a single point in some N dimensional space of sentences.
350 #
351
352
353 ######################################################################
354 # The Encoder
355 # -----------
356 #
357 # The encoder of a seq2seq network is a RNN that outputs some value for
358 # every word from the input sentence. For every input word the encoder
359 # outputs a vector and a hidden state, and uses the hidden state for the
360 # next input word.
361 #
362 # .. figure:: /_static/img/seq-seq-images/encoder-network.png
363 # :alt:
364 #
365 #
366
367 class EncoderRNN(nn.Module):
368 def __init__(self, input_size, hidden_size):
369 super(EncoderRNN, self).__init__()
370 self.hidden_size = hidden_size
371 # 定义隐藏层
372 self.embedding = nn.Embedding(input_size, hidden_size)
373 # word embedding的定义可以这么理解，例如nn.Embedding(2, 4)
374 # 2表示有2个词，4表示4维度，其实也就是一个2x4的矩阵，
375 # 如果有100个词，每个词10维，就可以写为nn.Embedding(100, 10)
376 # 注意这里的词向量的建立只是初始的词向量，并没有经过任何修改优化
377 # 需要建立神经网络通过learning的办法修改word embedding里面的参数
378 # 使得word embedding每一个词向量能够表示每一个不同的词。
379 self.gru = nn.GRU(hidden_size, hidden_size) # 用到了上面提到的GRU模型
380
381 def forward(self, input, hidden):
382 embedded = self.embedding(input).view(1, 1, -1) # -1是指自适应，view相当于reshape函数
383 output = embedded
384 output, hidden = self.gru(output, hidden)
385 return output, hidden
386
387 def initHidden(self): # 初始化
388 return torch.zeros(1, 1, self.hidden_size, device=device)
389
390
391 ######################################################################
392 # The Decoder
393 # -----------
394 #
395 # The decoder is another RNN that takes the encoder output vector(s) and
396 # outputs a sequence of words to create the translation.
397 #
398
399
400 ######################################################################
401 # Simple Decoder
402 # ^^^^^^^^^^^^^^
403 #
404 # In the simplest seq2seq decoder we use only last output of the encoder.
405 # This last output is sometimes called the *context vector* as it encodes
406 # context from the entire sequence. This context vector is used as the
407 # initial hidden state of the decoder.
408 #
409 # At every step of decoding, the decoder is given an input token and
410 # hidden state. The initial input token is the start-of-string ``<SOS>``
411 # token, and the first hidden state is the context vector (the encoder's
412 # last hidden state).
413 #
414 # .. figure:: /_static/img/seq-seq-images/decoder-network.png
415 # :alt:
416 #
417 #
418
419 class DecoderRNN(nn.Module):
420 # DecoderRNN与encoderRNN结构类似，结合图片即可搞清逻辑
421 def __init__(self, hidden_size, output_size):
422 super(DecoderRNN, self).__init__()
423 self.hidden_size = hidden_size
424
425 self.embedding = nn.Embedding(output_size, hidden_size)
426 self.gru = nn.GRU(hidden_size, hidden_size)
427 self.out = nn.Linear(hidden_size, output_size)
428 self.softmax = nn.LogSoftmax(dim=1)
429
430 def forward(self, input, hidden):
431 output = self.embedding(input).view(1, 1, -1) # -1是指自适应，view相当于reshape函数
432 output = F.relu(output)
433 output, hidden = self.gru(output, hidden) # 此处使用gru神经网络
434 # 对上述结果使用softmax,就是图片中左边倒数第二个
435 output = self.softmax(self.out(output[0]))
436 return output, hidden
437
438 def initHidden(self):
439 return torch.zeros(1, 1, self.hidden_size, device=device)
440
441
442 ######################################################################
443 # I encourage you to train and observe the results of this model, but to
444 # save space we'll be going straight for the gold and introducing the
445 # Attention Mechanism.
446 #
447
448
449 ######################################################################
450 # Attention Decoder
451 # ^^^^^^^^^^^^^^^^^
452 #
453 # If only the context vector is passed betweeen the encoder and decoder,
454 # that single vector carries the burden of encoding the entire sentence.
455 #
456 # Attention allows the decoder network to "focus" on a different part of
457 # the encoder's outputs for every step of the decoder's own outputs. First
458 # we calculate a set of *attention weights*. These will be multiplied by
459 # the encoder output vectors to create a weighted combination. The result
460 # (called ``attn_applied`` in the code) should contain information about
461 # that specific part of the input sequence, and thus help the decoder
462 # choose the right output words.
463 #
464 # .. figure:: https://i.imgur.com/1152PYf.png
465 # :alt:
466 #
467 # Calculating the attention weights is done with another feed-forward
468 # layer ``attn``, using the decoder's input and hidden state as inputs.
469 # Because there are sentences of all sizes in the training data, to
470 # actually create and train this layer we have to choose a maximum
471 # sentence length (input length, for encoder outputs) that it can apply
472 # to. Sentences of the maximum length will use all the attention weights,
473 # while shorter sentences will only use the first few.
474 #
475 # .. figure:: /_static/img/seq-seq-images/attention-decoder-network.png
476 # :alt:
477 #
478 #
479
480 class AttnDecoderRNN(nn.Module):
481 def __init__(self, hidden_size, output_size, dropout_p=0.1, max_length=MAX_LENGTH):
482 super(AttnDecoderRNN, self).__init__()
483 self.hidden_size = hidden_size
484 self.output_size = output_size
485 self.dropout_p = dropout_p
486 self.max_length = max_length
487
488 self.embedding = nn.Embedding(self.output_size, self.hidden_size)
489 self.attn = nn.Linear(self.hidden_size * 2, self.max_length)
490 self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)
491 self.dropout = nn.Dropout(self.dropout_p)
492 self.gru = nn.GRU(self.hidden_size, self.hidden_size)
493 self.out = nn.Linear(self.hidden_size, self.output_size)
494
495 def forward(self, input, hidden, encoder_outputs):
496 # 对于输入的input内容进行embedding和dropout操作
497 # dropout是指随机丢弃一些神经元
498 embedded = self.embedding(input).view(1, 1, -1)
499 embedded = self.dropout(embedded)
500
501 # 此处相当于学出来了attention的权重
502 # 需要注意的是torch的concatenate函数是torch.cat，是在已有的维度上拼接，
503 # 而stack是建立一个新的维度，然后再在该纬度上进行拼接。
504 attn_weights = F.softmax(
505 self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)
506
507 # 将attention权重作用在encoder_outputs上
508 # 对存储在两个批batch1和batch2内的矩阵进行批矩阵乘操作。
509 # batch1和 batch2都为包含相同数量矩阵的3维张量。
510 # 如果batch1是形为b×n×m的张量，batch1是形为b×m×p的张量，
511 # 则out和mat的形状都是n×p
512 attn_applied = torch.bmm(attn_weights.unsqueeze(0),
513 encoder_outputs.unsqueeze(0))
514 # 拼接操作，将embedded和attn_Applied拼接起来
515 output = torch.cat((embedded[0], attn_applied[0]), 1)
516 # 返回一个新的张量，对输入的制定位置插入维度 1
517 output = self.attn_combine(output).unsqueeze(0)
518
519 output = F.relu(output)
520 output, hidden = self.gru(output, hidden)
521
522 output = F.log_softmax(self.out(output[0]), dim=1)
523 return output, hidden, attn_weights
524
525 def initHidden(self):
526 return torch.zeros(1, 1, self.hidden_size, device=device)
527
528
529 ######################################################################
530 # .. note:: There are other forms of attention that work around the length
531 # limitation by using a relative position approach. Read about "local
532 # attention" in `Effective Approaches to Attention-based Neural Machine
533 # Translation <https://arxiv.org/abs/1508.04025>`__.
534 #
535 # Training
536 # ========
537 #
538 # Preparing Training Data
539 # -----------------------
540 #
541 # To train, for each pair we will need an input tensor (indexes of the
542 # words in the input sentence) and target tensor (indexes of the words in
543 # the target sentence). While creating these vectors we will append the
544 # EOS token to both sequences.
545 #
546
547 def indexesFromSentence(lang, sentence):
548 return [lang.word2index[word] for word in sentence.split(' ')]
549
550
551 def tensorFromSentence(lang, sentence):
552 # 获得词的索引
553 indexes = indexesFromSentence(lang, sentence)
554 # 将EOS标记添加到两个序列中
555 indexes.append(EOS_token)
556 return torch.tensor(indexes, dtype=torch.long, device=device).view(-1, 1)
557
558
559 def tensorsFromPair(pair):
560 # 每一对为需要输入的张量（输入句子中的词的索引）和目标张量
561 # （目标语句中的词的索引）
562 input_tensor = tensorFromSentence(input_lang, pair[0])
563 target_tensor = tensorFromSentence(output_lang, pair[1])
564 return (input_tensor, target_tensor)
565
566
567 ######################################################################
568 # Training the Model
569 # ------------------
570 #
571 # To train we run the input sentence through the encoder, and keep track
572 # of every output and the latest hidden state. Then the decoder is given
573 # the ``<SOS>`` token as its first input, and the last hidden state of the
574 # encoder as its first hidden state.
575 #
576 # "Teacher forcing" is the concept of using the real target outputs as
577 # each next input, instead of using the decoder's guess as the next input.
578 # Using teacher forcing causes it to converge faster but `when the trained
579 # network is exploited, it may exhibit
580 # instability <http://minds.jacobs-university.de/sites/default/files/uploads/papers/ESNTutorialRev.pdf>`__.
581 #
582 # You can observe outputs of teacher-forced networks that read with
583 # coherent grammar but wander far from the correct translation -
584 # intuitively it has learned to represent the output grammar and can "pick
585 # up" the meaning once the teacher tells it the first few words, but it
586 # has not properly learned how to create the sentence from the translation
587 # in the first place.
588 #
589 # Because of the freedom PyTorch's autograd gives us, we can randomly
590 # choose to use teacher forcing or not with a simple if statement. Turn
591 # ``teacher_forcing_ratio`` up to use more of it.
592 #
593
594 teacher_forcing_ratio = 0.5
595
596
597 # teacher forcing即指使用教师强迫其能够更快的收敛
598 # 不过当训练好的网络被利用时，容易表现出不稳定性
599 # teacher_forcing_ratio即指教师训练比率
600 # 用于训练的函数
601
602
603 def train(input_tensor, target_tensor, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion,
604 max_length=MAX_LENGTH):
605 # encoder即指EncoderRNN(input_lang.n_words, hidden_size)
606 # attn_decoder即指 AttnDecoderRNN(hidden_size, output_lang.n_words, dropout_p=0.1)
607 # hidden=256
608 encoder_hidden = encoder.initHidden()
609
610 # encoder_optimizer 即指optim.SGD(encoder.parameters(), lr=learning_rate)
611 # decoder_optimizer 即指optim.SGD(decoder.parameters(), lr=learning_rate)
612 # nn.Parameter()是Variable的一种，常被用于模块参数(module parameter)。
613 # Parameters 是 Variable 的子类。Paramenters和Modules一起使用的时候会有一些特殊的属性，
614 # 即：当Paramenters赋值给Module的属性的时候，他会自动的被加到 Module的 参数列表中
615 # (即：会出现在 parameters() 迭代器中)。将Varibale赋值给Module属性则不会有这样的影响。
616 # 这样做的原因是：我们有时候会需要缓存一些临时的状态(state), 比如：模型中RNN的最后一个隐状态。
617 # 如果没有Parameter这个类的话，那么这些临时变量也会注册成为模型变量。
618 encoder_optimizer.zero_grad()
619 decoder_optimizer.zero_grad()
620
621 # 得到长度
622 input_length = input_tensor.size(0)
623 target_length = target_tensor.size(0)
624
625 # 初始化outour值
626 encoder_outputs = torch.zeros(max_length, encoder.hidden_size, device=device)
627
628 loss = 0
629
630 # 以下循环是学习过程
631 for ei in range(input_length):
632 encoder_output, encoder_hidden = encoder(input_tensor[ei], encoder_hidden)
633 encoder_outputs[ei] = encoder_output[0, 0] # 这里为什么取 0,0
634
635 # 定义decoder的Input值
636 decoder_input = torch.tensor([[SOS_token]], device=device)
637
638 decoder_hidden = encoder_hidden
639
640 use_teacher_forcing = True if random.random() < teacher_forcing_ratio else False
641
642 if use_teacher_forcing:
643 # Teacher forcing: Feed the target as the next input
644 # 教师强制: 将目标作为下一个输入
645 # 你观察教师强迫网络的输出,这些网络是用连贯的语法阅读的,但却远离了正确的翻译 -
646 # 直观地来看它已经学会了代表输出语法,并且一旦老师告诉它前几个单词,就可以"拾取"它的意思,
647 # 但它没有适当地学会如何从翻译中创建句子.
648 for di in range(target_length):
649 # 通过decoder得到输出值
650 decoder_output, decoder_hidden, decoder_attention = decoder(
651 decoder_input, decoder_hidden, encoder_outputs)
652 # 定义损失函数并计算
653 loss += criterion(decoder_output, target_tensor[di])
654 decoder_input = target_tensor[di] # Teacher forcing
655
656 else:
657 # Without teacher forcing: use its own predictions as the next input
658 # 没有教师强迫: 使用自己的预测作为下一个输入
659 for di in range(target_length):
660 # 通过decoder得到输出值
661 decoder_output, decoder_hidden, decoder_attention = decoder(
662 decoder_input, decoder_hidden, encoder_outputs)
663
664 # topk：第k个最小元素,返回第k个最小元素
665 # 返回前k个最大元素，注意是前k个，largest=False，返回前k个最小元素
666 # 此函数的功能是求取1-D 或N-D Tensor的最低维度的前k个最大的值，返回值为两个Tuple
667 # 其中values是前k个最大值的Tuple，indices是对应的下标，默认返回结果是从大到小排序的。
668 topv, topi = decoder_output.topk(1)
669 decoder_input = topi.squeeze().detach() # detach from history as input
670
671 loss += criterion(decoder_output, target_tensor[di])
672 if decoder_input.item() == EOS_token:
673 break
674 # 反向传播
675 loss.backward()
676
677 # 更新参数
678 encoder_optimizer.step()
679 decoder_optimizer.step()
680
681 return loss.item() / target_length
682
683
684 ######################################################################
685 # This is a helper function to print time elapsed and estimated time
686 # remaining given the current time and progress %.
687 #
688
689 import time
690 import math
691
692
693 # 根据当前时间和进度百分比,这是一个帮助功能,用于打印经过的时间和估计的剩余时间.
694
695 def asMinutes(s):
696 m = math.floor(s / 60)
697 s -= m * 60
698 return '%dm %ds' % (m, s)
699
700
701 def timeSince(since, percent):
702 now = time.time()
703 s = now - since
704 es = s / (percent)
705 rs = es - s
706 return '%s (- %s)' % (asMinutes(s), asMinutes(rs))
707
708
709 ######################################################################
710 # The whole training process looks like this:
711 #
712 # - Start a timer
713 # - Initialize optimizers and criterion
714 # - Create set of training pairs
715 # - Start empty losses array for plotting
716 #
717 # Then we call ``train`` many times and occasionally print the progress (%
718 # of examples, time so far, estimated time) and average loss.
719 #
720
721 def trainIters(encoder, decoder, n_iters, print_every=1000, plot_every=100, learning_rate=0.01):
722 start = time.time()
723 plot_losses = []
724 print_loss_total = 0 # Reset every print_every
725 plot_loss_total = 0 # Reset every plot_every
726
727 encoder_optimizer = optim.SGD(encoder.parameters(), lr=learning_rate)
728 decoder_optimizer = optim.SGD(decoder.parameters(), lr=learning_rate)
729
730 # 获取训练的一对样本
731 training_pairs = [tensorsFromPair(random.choice(pairs))
732 for i in range(n_iters)]
733 # 定义出的损失函数
734 criterion = nn.NLLLoss()
735
736 for iter in range(1, n_iters + 1):
737 training_pair = training_pairs[iter - 1]
738 input_tensor = training_pair[0]
739 target_tensor = training_pair[1]
740
741 # 训练的过程并用于当损失函数
742 loss = train(input_tensor, target_tensor, encoder,
743 decoder, encoder_optimizer, decoder_optimizer, criterion)
744 print_loss_total += loss
745 plot_loss_total += loss
746
747 if iter % print_every == 0:
748 print_loss_avg = print_loss_total / print_every
749 print_loss_total = 0
750 # 打印进度(样本的百分比,到目前为止的时间,估计的时间)和平均损失.
751 print('%s (%d %d%%) %.4f' % (timeSince(start, iter / n_iters),
752 iter, iter / n_iters * 100, print_loss_avg))
753
754 if iter % plot_every == 0:
755 plot_loss_avg = plot_loss_total / plot_every
756 plot_losses.append(plot_loss_avg)
757 plot_loss_total = 0
758 # 绘制图像
759 showPlot(plot_losses)
760
761
762 ######################################################################
763 # Plotting results
764 # ----------------
765 #
766 # Plotting is done with matplotlib, using the array of loss values
767 # ``plot_losses`` saved while training.
768 #
769
770 import matplotlib.pyplot as plt
771
772 plt.switch_backend('agg')
773 import matplotlib.ticker as ticker
774 import numpy as np
775
776
777 # 使用matplotlib进行绘图，使用训练时保存的损失值plot_losses数组.
778 def showPlot(points):
779 plt.figure()
780 fig, ax = plt.subplots()
781 # this locator puts ticks at regular intervals
782 # 这个定位器会定期发出提示信息
783 loc = ticker.MultipleLocator(base=0.2)
784 ax.yaxis.set_major_locator(loc)
785 plt.plot(points)
786
787
788 ######################################################################
789 # Evaluation
790 # ==========
791 #
792 # Evaluation is mostly the same as training, but there are no targets so
793 # we simply feed the decoder's predictions back to itself for each step.
794 # Every time it predicts a word we add it to the output string, and if it
795 # predicts the EOS token we stop there. We also store the decoder's
796 # attention outputs for display later.
797 #
798
799 def evaluate(encoder, decoder, sentence, max_length=MAX_LENGTH):
800 with torch.no_grad():
801 # 从sentence中得到对应的变量
802 input_tensor = tensorFromSentence(input_lang, sentence)
803 # 长度
804 input_length = input_tensor.size()[0]
805
806 # encoder即指EncoderRNN(input_lang.n_words, hidden_size)
807 # attn_decoder即指 AttnDecoderRNN(hidden_size,
808 # output_lang.n_words, dropout_p=0.1)
809 # hidden=256
810 encoder_hidden = encoder.initHidden()
811
812 # 初始化outputs值
813 encoder_outputs = torch.zeros(max_length, encoder.hidden_size, device=device)
814
815 # 以下是学习过程
816 for ei in range(input_length):
817 encoder_output, encoder_hidden = encoder(input_tensor[ei],
818 encoder_hidden)
819 encoder_outputs[ei] += encoder_output[0, 0]
820
821 # 定义好decoder部分的input值
822 decoder_input = torch.tensor([[SOS_token]], device=device) # SOS
823
824 # 设置好隐藏层
825 decoder_hidden = encoder_hidden
826
827 decoded_words = []
828 decoder_attentions = torch.zeros(max_length, max_length)
829
830 for di in range(max_length):
831 # 得到结果
832 decoder_output, decoder_hidden, decoder_attention = decoder(decoder_input, decoder_hidden, encoder_outputs)
833
834 # attention部分的数据
835 decoder_attentions[di] = decoder_attention.data
836 # 选择output中的第一个值
837 topv, topi = decoder_output.data.topk(1)
838 if topi.item() == EOS_token:
839 decoded_words.append('<EOS>')
840 break
841 else:
842 decoded_words.append(output_lang.index2word[topi.item()]) # 将output_lang添加到decoded
843
844 decoder_input = topi.squeeze().detach()
845
846 return decoded_words, decoder_attentions[:di + 1]
847
848
849 ######################################################################
850 # We can evaluate random sentences from the training set and print out the
851 # input, target, and output to make some subjective quality judgements:
852 #
853
854 # 从训练集中评估随机的句子并打印出输入,目标和输出以作出一些主观质量判断
855 def evaluateRandomly(encoder, decoder, n=10):
856 for i in range(n):
857 pair = random.choice(pairs)
858 print('>', pair[0])
859 print('=', pair[1])
860 output_words, attentions = evaluate(encoder, decoder, pair[0])
861 output_sentence = ' '.join(output_words)
862 print('<', output_sentence)
863 print('')
864
865
866 ######################################################################
867 # Training and Evaluating
868 # =======================
869 #
870 # With all these helper functions in place (it looks like extra work, but
871 # it makes it easier to run multiple experiments) we can actually
872 # initialize a network and start training.
873 #
874 # Remember that the input sentences were heavily filtered. For this small
875 # dataset we can use relatively small networks of 256 hidden nodes and a
876 # single GRU layer. After about 40 minutes on a MacBook CPU we'll get some
877 # reasonable results.
878 #
879 # .. Note::
880 # If you run this notebook you can train, interrupt the kernel,
881 # evaluate, and continue training later. Comment out the lines where the
882 # encoder and decoder are initialized and run ``trainIters`` again.
883 #
884
885 hidden_size = 256
886 # 编码部分
887 encoder1 = EncoderRNN(input_lang.n_words, hidden_size).to(device)
888 # 加入了attention机制的解码部分
889 attn_decoder1 = AttnDecoderRNN(hidden_size, output_lang.n_words, dropout_p=0.1).to(device)
890 # 训练部分
891 trainIters(encoder1, attn_decoder1, 75000, print_every=5000)
892
893 ######################################################################
894 # 随机生成一组结果
895 evaluateRandomly(encoder1, attn_decoder1)
896
897 ######################################################################
898 # Visualizing Attention
899 # ---------------------
900 #
901 # A useful property of the attention mechanism is its highly interpretable
902 # outputs. Because it is used to weight specific encoder outputs of the
903 # input sequence, we can imagine looking where the network is focused most
904 # at each time step.
905 #
906 # You could simply run ``plt.matshow(attentions)`` to see attention output
907 # displayed as a matrix, with the columns being input steps and rows being
908 # output steps:
909 #
910
911 output_words, attentions = evaluate(encoder1, attn_decoder1, "je suis trop froid .")
912 plt.matshow(attentions.numpy())
913
914
915 ######################################################################
916 # For a better viewing experience we will do the extra work of adding axes
917 # and labels:
918
919 def showAttention(input_sentence, output_words, attentions):
920 # Set up figure with colorbar
921 fig = plt.figure()
922 ax = fig.add_subplot(111)
923 cax = ax.matshow(attentions.numpy(), cmap='bone')
924 fig.colorbar(cax)
925
926 # Set up axes
927 ax.set_xticklabels([''] + input_sentence.split(' ') +
928 ['<EOS>'], rotation=90)
929 ax.set_yticklabels([''] + output_words)
930
931 # Show label at every tick
932 ax.xaxis.set_major_locator(ticker.MultipleLocator(1))
933 ax.yaxis.set_major_locator(ticker.MultipleLocator(1))
934
935 plt.show()
936
937
938 def evaluateAndShowAttention(input_sentence):
939 output_words, attentions = evaluate(
940 encoder1, attn_decoder1, input_sentence)
941 print('input =', input_sentence)
942 print('output =', ' '.join(output_words))
943 showAttention(input_sentence, output_words, attentions)
944
945
946 evaluateAndShowAttention("elle a cinq ans de moins que moi .")
947 evaluateAndShowAttention("elle est trop petit .")
948 evaluateAndShowAttention("je ne crains pas de mourir .")
949 evaluateAndShowAttention("c est un jeune directeur plein de talent .")
950
951 ######################################################################
952 # Exercises
953 # =========
954 #
955 # - Try with a different dataset
956 #
957 # - Another language pair
958 # - Human → Machine (e.g. IOT commands)
959 # - Chat → Response
960 # - Question → Answer
961 #
962 # - Replace the embeddings with pre-trained word embeddings such as word2vec or
963 # GloVe
964 # - Try with more layers, more hidden units, and more sentences. Compare
965 # the training time and results.
966 # - If you use a translation file where pairs have two of the same phrase
967 # (``I am test \t I am test``), you can use this as an autoencoder. Try
968 # this:
969 #
970 # - Train as an autoencoder
971 # - Save only the Encoder network
972 # - Train a new Decoder for translation from there
973 #