A high-power 808 nm diode-directly-pumped passively mode-locked 1064nm Nd:YVO4 laser was demonstrated with a semiconductor saturable absorber mirror (SESAM). Through theory analysis and carefully experimental optimization of laser mode radii in the laser crystal and on the SESAM. An average output power of 3.39 W and single pulse energy of 0.115 μJ was obtained with a pulse width of 16.3ps and a repetition rate of 29.36 MHz under a pump power of 8.92 W, corresponding to an optic-optic efficiency of 38.0％ and the high beam quality in the x, y direction was 1.12 and 1.07, respectively.