Contextual String Embeddings

This paper proposes a novel type of contextualized characterlevel word embedding which is hypothesized to combine the best attributes of the word-embeddings; namely, the ability to (1) pre-train on large unlabeled corpora, (2) capture word meaning in context and therefore produce different embeddings for polysemous words depending on their usage, and (3) model words and context fundamentally as sequences of characters, to both better handle rare and misspelled words as well as model subword structures such as prefixes and endings. It presents a method to generate such a contextualized embedding for any string of characters in a sentential context, and thus refers to the proposed representations as contextual string embeddings.