So I just failed a rank 3 buster quest because a giant laser from the arms of the Castrum swept over the towers and destroyed them effortlessly. Did we do something wrong or are we being punished by garbage encounter design?

The cannons can be destroyed; you have to run back toward the Castle to attack them, so it becomes a matter of balancing defending the towers and going to destroy the cannons before they can fire.

also not all players need to deal with those cannons, 2 players can easily take then down before they shoot while the rest defends the towers