[AMMAI] [Lecture 12] - "Text Understanding from Scratch"

Paper Information:
  Zhang, Xiang, and Yann LeCun. "Text Understanding from Scratch." arXiv preprint arXiv:1502.01710 (2015).

Motivation:
  ConvNet is quite successful in image domain; therefore, they want to apply it in text domain hoping that it can learn the relationship between text by character-level ConvNet.

Contributions:
  Demonstrating the ability of deep learning system in text understanding without embedding knowledge.

Technical summarization:
  ConvNet Model Design:
    
The convolution function is define as:
And the max-pooling function is define as:

Finally, they use ReLU as thresholding function.
For the network architecture, they design the ConvNets with the architecture with 6 conv and 3 fc.

  Data Augmentation:
    Just like the rotating, scaling and flipping in image recognition to achieve augmentation. They choose to adopt replacing synonyms as their method for invariance.

My comment:


  It's interesting that they compare quantization with Braille used for assisting blind reading. In this situation, ConvNet is just like a blind person tring to learn the binary encoding.

For the experiment, they evaluate on many dataset like DBpedia, Amazon and Yahoo! Answers. All the experiments have better result comparing to bag of words or word2vec.

  Besides, they also demonstrate the ability of dealing with Chinese on Sogou News corpus. This experiment shows it's generality in language. It's extremely amazing that the accuracy is high even crossing the language.