This is usually a bug that relates to a player being disconnected, having considerable lag, or moving into an area that the game doesn't expect the player to move to. This causes enemies to not path-find correctly or not spawn correctly, and inevitably causes the game to run out the timer of the quest. If the blue health bar at the top is higher than the red health bar however, the game will treat it as a win, and you will still receive item drop rewards in the tent, however not as many as you would if you defeat the castrum demonica. Nor will you get as many Buster Points.
This issue is mostly resolved in the upcoming non-buster Level-Up versions of this quest where there are no actual phases besides standard add/boss kill phases.
Please make sure to report this bug on the contact page of the support website.