The rules mention several times the player affecting planets they "influence". This means Captains can only spend Fortune on planets on which they've had adventures.