Coronary artery disease (CAD) is the third most prominent cause of mortality in the world and is linked to 17.8 million deaths per year. Blockage of the coronary arteries leads to reduced blood supply to the heart muscles, causing complications such as myocardial infarction (MI), otherwise known as a heart attack. If left untreated, this can result in long-term damage. MRI or CT scans are typically performed to check for MI. Although it is challenging, diagnosis of CAD using echocardiography could be possible, which is relatively inexpensive and quicker than other imaging modalities. Furthermore, automation of this process can help reduce the burden on cardiologists, and even make diagnosis accessible where clinical experts are not present. In this work, we present an end-to-end deep learning approach for classification of MI in echocardiography videos. We show how our automatic method outperforms existing published work for automatic MI classification (+1.8% F1-score) and shows slightly lower results compared to the best performing semi-automatic method (F1-scores of 82.90 vs. 85.71). Our work has been developed using multiple publicly available datasets and was evaluated on an MI classification dataset with an F1-score of 87.09%. Our code is available at (https://github.com/BioMedIA-MBZUAI/mi-classification).