The coupling oscillation problems between shafting torsional vibration system and speed control system have often occurred due to the increasingly complex marine propulsion system. The propulsion shafting is always assumed to be rigid and the diesel engine is usually modelled as mean value model for general simulation of diesel speed control system. To investigate the stability characteristics between these two systems, a frequency-domain model considering the coupling effects is proposed. In such a coupling model, the shafting was modelled as deformable rather than rigid and the actual cylinder excitations were considered while modelling the diesel engine. The proposed model was verified by test results and they agree with each other very well. The results illustrated that the stability of the coupling system obtained by the proposed model is worse than that obtained by the mean value model, and the stability margin of the coupling system is negatively correlated with the gain coefficients of the speed control system. What's more, an instability region has been found due to the inappropriate integral coefficient of speed controller, which should be avoided in practical engineering applications.