The purpose of this trial is to assess if ifinatamab deruxtecan (I-DXd) can treat esophageal squamous cell carcinoma (ESCC). I-DXd is an antibody-drug conjugate (ADC). An ADC attaches to a protein on cancer cells and delivers treatment to destroy those cells. The goal of this trial is to learn how many participants who receive I-DXd have the cancer respond, which means the cancer gets smaller or goes away.