Focusing on the control of chatter vibration in a milling process and the optimization of machining parameters in practical engineering, this paper presented theoretical model of dynamic milling force for a two-degree-of-freedom milling system. And then the milling chatter stability domain for the system was investigated by using frequency-domain analysis and semi-discretization methods. With software MATLAB as development tool, the simulation algorithms for predicting chatter stability domain of the milling process above-mentioned were constructed and the corresponding stability lobe diagrams can be obtained. For further clarification, a visual software interface based on MATLAB-GUI module was developed as well, and the results derived from two prediction methods were shown clearly. By selecting typical machining parameters and simulation parameters, the effects of modal parameters, milling force coefficients, tool parameters, and machining parameters on milling chatter stability domain were obtained. The stability lobe diagrams obtained in this work were completely consistent with experimental results, which further verified the predictive accuracy and practicability of the simulation system in this work.