We report on the effect of oxygen disorder in the CuO chains on the superconducting properties of oxygen depleted YBa$_2$Cu$_3$O$_x$ (YBCO). While moderate disorder, induced thermally, depresses the critical temperature as a result of a reduced carrier concentration, strong oxygen disorder and chain fragmentation induced by low energy (80 keV) He$^+$ irradiation suppresses the critical temperature beyond the values expected from the reduction of the carrier density. This provides an experimental evidence of pair breaking by chain disorder and outlines the importance of chain states in the pairing mechanism in the YBCO family.