Oriented single crystals of $R$Fe$_3$(BO$_3$)$_4$, with $R$=La or Nd, have been studied by Raman spectroscopy. Spectra with the relevant polarization configurations have been recorded in order to obtain the symmetry of the observed phonons. The factor group analysis and the correlation with the free (BO$_3$)$^{3-}$ ion are done in order to identify most of the phonons associated with the two different types of (BO$_3$)$^{3-}$ ion present in the crystal. The number and symmetries of the optical Raman active modes are $7 A_1 + 19 E$, among which $4 A_1 + 8 E$ can be assigned as mostly due to (BO$_3$)$^{3-}$ vibrations. $7 A_1 + 18 E$ modes were observed. The highest energy peaks have been assigned to the regular planar (BO$_3$)$^{3-}$ and to the three irregular (BO$_3$)$^{3-}$ groups. The two lowest energy peaks of $A_1$ symmetry (around 180 and 300 cm$^{-1}$) are very probably related to the BO$_3$ rotatory mode and to Fe displacements. $R$ ions do not participate in $A_1$ symmetry modes. The $E$ mode around 90 cm$^{-1}$ (the lowest frequency mode) is probably due to the $R$ ions which have the longest bonds and are the heaviest ions.