Sarcasm detection has been an interesting and difficult task in NLP. Sarcasm in memes is special in that the it's usually demonstrated by the contrast between the captions and the image. For our project, we use a dataset of 7000 memes labeled as "sarcastic" or "not sarcastic". We took inspirations from OPEN AI's CLIP model and implemented a multi-modal binary classifier for sarcasm detection. Our model achieved a 74% accuracy with a 0.80 AUC-ROC score.