It might not be too much of an issue, depending on how you build your Trees, especially if you primarily use Hunter as your subclass, since Fury Stance Hunter is one of the best subclasses for Fighter, Ranger, Gunner, Tecther, Braver, and Bouncer. You have to make some sacrifices in support Skills that you would normally have for a subclass for the sake of having Hunter weapon Focus Skills and Guard Skills, but it shouldn't be too bad.
From there it just leaves your subclasses for Hunter, Force, and Summoner. Fighter is a generally one of the best choices for all three, but you basically lose out on up to 15 SP just from Overload and Critical Strike since they only apply when Fighter is your main class, and even then the builds for Fighter for each of those classes varies because of their playstyle. Force also has the option of Tecther as a subclass, but a Techter sub Tree is usually built very differently from a main Tree.