Natural language processing is a broad field that encompasses several sub-tasks. One problem that has gained visibility over the past several years is that of Sentiment Analysis. This is the process of determining the attitude of an author towards some subject across some spectrum, typically “positive” or “negative,” by analyzing the textual information. Whereas the field started with simple counting of words with certain characteristics, it has grown in complexity with the advent of deep learning and neural network based language models. Typically, datasets used to train and evaluate these models consist of text with appropriate labels, such as movie reviews with an accompanied star rating. However, the applicability of those results to other scenarios, such as unstructured or natural text has not been clear. In this paper, we demonstrate a clear and simple case that shows that the problem of sentiment analysis is fundamentally unsuitable for natural text. We consider state-of-the-art black box models developed and hosted by 3 of the largest companies in this field: Amazon, Google and IBM.