The purpose of this study is to assess the safety and tolerability of zilovertamab vedotin as monotherapy and in combination in participants with select B-cell lymphomas including mantle cell lymphoma (MCL), Richter's transformation lymphoma (RTL), follicular lymphoma (FL), and chronic lymphocytic leukemia (CLL). This study will also evaluate zilovertamab vedotin as monotherapy and in combination with respect to objective response rate. * Cohort A: Participants with relapsed or refractory MCL relapsed or refractory disease after at least 2 prior systemic therapies including a Bruton's tyrosine kinase inhibition/inhibitor (BTKi), and post therapy chimeric antigen receptor T (CAR-T) cell therapy or ineligible for CAR-T cell therapy * Cohort B: Participants with relapsed or refractory RT disease after at least 1 prior systemic therapy * Cohort C: Participants with relapsed or refractory MCL relapsed or refractory disease after at least 1 prior systemic therapy and no prior exposure to a non-covalent BTKi * Cohort D: Participants with relapsed or refractory FL and CLL relapsed or refractory disease after at least 2 prior systemic therapies and have no other available therapy * Cohort E: Participants with relapsed or refractory FL after at least 2 prior systemic therapies and have no other available therapy The primary study hypothesis is that zilovertamab vedotin monotherapy has an increased Objective Response Rate (ORR) per Lugano Response Criteria as assessed by blinded independent central review (BICR). As of Amendment 07, Cohort D is closed to enrollment of participants with CLL and enrollment of participants into Arm 2 (zilovertamab vedotin at Dose 2 on Days 1 \& 8 of each 3 Week Cycle (Q2/3W)).