Pay Back America Act of 2004

Pay Back America Act of 2004 - Amends the Internal Revenue Code to impose a per passenger tax on covered international voyages of commercial passenger vessels.

Establishes the Caribbean Ports and Infrastructure Protection Trust Fund (Caribbean Fund) and the Water and Marine Wildlife Protection Trust Fund (Wildlife Fund) in the Treasury. Appropriates to those funds a percentage of the above-referenced taxes.

Authorizes the Secretary of of the Treasury to make specified payments from the Caribbean Fund to the Bahamas, the Cayman Islands, Barbados, Jamaica, Saint Lucia, Grenada, Antigua and Barbuda, Belize, the British Virgin Islands, Dominica, Guyana, Haiti, Montserrat, Saint Kitts and Nevis, Saint Vincent and the Grenadines, Suriname, Trinidad and Tobago, Anguilla, Bermuda, and the Turks and Caicos Islands.

Requires amounts in the Wildlife Fund to be used for ocean waste cleanup or enforcement of restrictions on ocean dumping.

Directs the Secretary of the department in which the Coast Guard is operating to issue regulations requiring cruise vessels entering U.S. ports or places to have: (1) posted notice that passengers may report illegal dumping from the vessel by calling a toll-free number; and (2) included such notice on each ticket sold.

Prohibits: (1) vessels from entering U.S. ports or places absent properly functioning pollution prevention equipment or compliance with notice requirements; (2) false entries in a vessel's oil record book or record of shore-side disposal; (3) operation of cruise vessels in U.S. foreign or domestic commerce that fail to comply with notice requirements.

Requires biannual inspections of vessels.