Recent advancements in Artificial Intelligence have enabled impressive gains in tasks once thought able to be performed only by humans, or even by trained professionals. Given enough information about the problem at hand and many cases where we know the answer, AI methods can automatically identify patterns and learn complicated functions that accurately predict the outcome. Recent deep learning-based models have achieved expert-level performance in tasks like identifying tumors in magnetic resonance imaging scans, detecting subject positional anomalies, such as falls, or predicting disease risk from biomarkers. However, the complexity of some of these functions make it difficult to understand on what basis the ultimate decision is made for a given problem, reducing their utility in the real world. In my talk, I will discuss the importance of interpretability for machine learning and AI models, present some ongoing efforts in this area, and give some examples of work we are doing in my lab in the biomedical and AI space.