I've had my ham radio license for almost 40 years and been asked by friends to help them fix their transmitter installations for most of that time. Usually it's been along the lines of adding more power because they didn't have enough range. In all those years, I've never had to add more power to get them the range increase that they wanted. The problem was always in the antenna system/installation.
First, as the folks above mention, get that antenna as high up as possible.
Next, get/make an antenna that's optimized for the band you're operating on. Otherwise, it's just going dissipate the radio energy as heat. A good antenna doesn't need to be expensive. I made one that I was able to contact the Mir space station with out of coat hangers. See the ARRL Amateur Radio Handbook for designs and instructions.
Then, there's the coax and the connectors. For many people that's just an afterthought, like an extension cable. The wrong choices here can easily eat up 3/4 of the power coming from your radio, effectively turning your 10 watt transmitter into a 2.5 watt device. Use the highest quality coax and connectors you can afford. You really do get what you pay for. Buy them through a radio distributor instead of a marine supply place so that you're paying for the quality and not the "marine" markup.
Next, match all the impedances. If the radio has a 50-ohm output, use 50-ohm coax, connectors and antenna. Same for 75-ohm. Unmatched impedances will cause the radio energy to bounce around in the antenna system, eventually dissipating as heat. If too much is reflected back into the transmitter, it can be damaged or it may automatically reduce the power to avoid damage. Either way, it's not going to be getting out to the other boats where it's needed.
You can go the extra mile and install matching networks between the antenna and coax and coax and radio to get a perfect end-to-end match. That way you'll end up losing almost no power and your transmitter will operate full power.
Now that you've spent all this time, money and trouble - protect it. Make sure that everything is watertight and weatherproof. It should not be bothered by a knockdown that puts the masthead into the water. It should stand up to a driving rain and and all-day fog. It should be protected from fresh and salt water, as well as sunlight and bird crap. Spots of corrosion can act as non-linear mixers with the result that instead of transmitting on the marine bands as you intend, you'll be accidently transmitting on the aircraft bands as well. Not only will your signal to the Coast Guard be weaker, you could be covering up air traffic control instructions to a plane. So make sure that there's no corrosion in the system, ever.
By making your sysem this efficient, you can get away with a low-power transmitter, especially in the VHF bands. Not only can you use a hand-held, as Michael does, but you'll be able to run it on low power. It also works both ways - a good transmit antenna is a good receive antenna. You'll be able to hear weaker signals more clearly.
So, my advice is to spend the money on the antenna system and save it in the transmitter and power system.
Have fun,
Laszlo