This paper is concerned with the numerical solution of the singularly perturbed differential-difference equations with small shifts called delay and advanced parameters. A fourth order finite difference method with a fitting factor is proposed for the solution of the singularly perturbed differential-difference equations with mixed shifts. The delay and advanced shifts are managed by Taylor series and an asymptotically equivalent singularly perturbed two-point boundary value problem is obtained. A fitting factor is introduced in the fourth order finite difference scheme for the problem which takes care of the small values of the perturbation parameter. This fitting factor is obtained from the asymptotic solution of singular perturbations. Thomas algorithm is used to solve the discrete system of the difference scheme. Convergence of the proposed method is analyzed. Maximum absolute errors in comparison with the several numerical experiments aretabulated to illustrate the proposed method.