Q2: Image Captioning with LSTMs (35 points)

作业代码已上传至我github: https://github.com/jingshuangliu22/cs231n,欢迎参考、讨论、指正。

LSTM_Captioning.ipynb

Load MS-COCO data

idx_to_word <type 'list'> 1004 train_captions <type 'numpy.ndarray'> (400135, 17) int32 val_captions <type 'numpy.ndarray'> (195954, 17) int32 train_image_idxs <type 'numpy.ndarray'> (400135,) int32 val_features <type 'numpy.ndarray'> (40504, 512) float32 val_image_idxs <type 'numpy.ndarray'> (195954,) int32 train_features <type 'numpy.ndarray'> (82783, 512) float32 train_urls <type 'numpy.ndarray'> (82783,) |S63 val_urls <type 'numpy.ndarray'> (40504,) |S63 word_to_idx <type 'dict'> 1004

LSTM: step forward

next_h error: 5.70541304045e-09 next_c error: 5.81431230888e-09

dx error: 6.6214710385e-09 dh error: 5.10364159425e-10 dc error: 1.46461864645e-08 dWx error: 9.92319895433e-09 dWh error: 6.53036889306e-08 db error: 2.05724967677e-09

LSTM: forward

h error: 8.61053745211e-08

dx error: 6.01572477862e-10 dh0 error: 6.01572477862e-10 dWx error: 6.01572477862e-10 dWh error: 6.01572477862e-10 db error: 6.01572477862e-10

loss: 9.82445935443 expected loss: 9.82445935443 difference: 2.26485497024e-12

Overfit LSTM captioning model

(Iteration 1 / 100) loss: 78.372126 (Iteration 11 / 100) loss: 44.292632 (Iteration 21 / 100) loss: 27.702264 (Iteration 31 / 100) loss: 10.905141 (Iteration 41 / 100) loss: 5.647874 (Iteration 51 / 100) loss: 1.872165 (Iteration 61 / 100) loss: 0.460800 (Iteration 71 / 100) loss: 0.196382 (Iteration 81 / 100) loss: 0.195097 (Iteration 91 / 100) loss: 0.120261

LSTM test-time sampling