Your issue might be with your console. What model is it? Also, it could have something to do with how it is set up. Your internet modem might be configured differently than your friends' also. If it is just a case of the console configuration, restoring everything to default should help. If it is your modem, it could be going out or just not properly receiving packets from the PSO2 servers. If resetting the model to default settings doesn't work, you might need to set up port forwards for the server to your XBOX.
Thanks for the reply. I have whatever the base/original model of the Xbox is called. One of the people I play with is my brother and he has had his since launch; he also has the same gateway-thing from Verizon as his modem/router and he hasn't had any issues (another also has the same modem/router, though he's playing on a One X). I haven't messed with router settings other than to lower the firewall setting just in case and to set my Xbox to a static IP (this is all only after experiencing issues with this game and these obviously haven't helped).
Edit: I think I've correctly forwarded ports for the Xbox now, but whether that in anyway helped will be tough to say since like I said, I've had issues on and off and haven't had problems since Friday night when I last posted (it is now more than 2 days later).