Electron-phonon coupling in Yb$^{3+}$:LiNbO$_3$ is studied by comparing the optical (absorption and emission) low-temperature spectra with the room temperature Raman spectrum of undoped LiNbO$_3$. As a result, the whole vibrational structure of the optical spectra can be explained by a coupling of the optical phonons of LiNbO$_3$ and electronic transitions of Yb$^{3+}$.