Sizing a prop isn't that hard to do . If it isn't turning up the rpm's then it is too big or too much pitch or both . It is probably too much pitch .
In a perfect world a 12" prop pitch would try to advance the boat one foot per shaft rpm ( not engine ) .
So if you are trying to hit 6 knots ( 36480' per hour ) with a 12" prop , you need to make 608 shaft rpm's ( with no slip ) .
I had that motor, I think, 10hp Yanmar and it would not swing a 9x9 prop .