Aphasia is a language disorder resulting from a stroke or brain injury. Individuals with aphasia make paraphasias: speech errors consisting of an unintended word. This work focuses on automatically classifying semantic similarity of the paraphasia. We train a modern language model called BERT alongside a binary classifier to categorize each transcribed response to a PNT item as semantically similar to the target or not. We evaluate the accuracy of this classification compared with clinician scores. We also compare the results of this model to a previously used language model called word2vec.