Both my VPS's cannot connect to the API!

[tykoon]

This is weird.

My dedi and a shared hosting account are both fine.

But my MFPMU sites and other API sites on both my VPS's cannot get any results from the API!

The API script I have just returns blank and all the MFPMU sites give a 500 server error.

When I look at my Prosperent stats my impressions /clicks died around 1pm on the 20th.

Any ideas why this is happening?

Worpress blogs and the MFPMU control panel on these VPS's work fine, but the API is not working and I have done nothing to these servers at all.

[$imon]

There's some scheduled maintenance by SL today?

http://community.prosperent.com/showthread.php?283-Scheduled-maintenance-Thursday-July-21-2011-%2807-21-2011%29

[tykoon]

Yes, I don't think it is that.

My dedi and shared return results fine.

But both my VPS's stopped accessing the API yesterday around 1 pm...

[AcidRaZor]

AFAIK there's an issue between Turnkey and SL in terms of routing. Could be that?

[tykoon]

These are the errors I get in my logs on both servers..


PHP Warning: curl_errno(): 14 is not a valid cURL handle resource in /home/domains/domain.org/www/Prosperent_Api.php on line 350
PHP Fatal error: Uncaught exception 'Exception' in /home/domains/domain.org/www/Prosperent_Api.php:350\nStack trace:\n#0 /home/domains/domain.org/www/Prosperent_Api.php(308): Prosperent_Api->makeRequest('http://api.pros...')\n#1 /home/domains/domain.org/www/inc_functions.php(728): Prosperent_Api->fetch()\n#2 /home/domains/domain.org/www/inc_functions.php(277): getFeed()\n#3 /home/domains/domain.org/www/inc_functions.php(37): doFeedPage()\n#4 /home/domains/domain.org/www/index.php(10): getQueryInfo()\n#5 {main}\n thrown in /home/domains/domain.org/www/Prosperent_Api.php on line 350

[tykoon]

Yes, it is Turnkey I believe via a re-seller!

What to do?

[AcidRaZor]

Yes, it is Turnkey I believe via a re-seller!

What to do?

Brian is following this up with the engineers, but it wouldn't hurt just sending him a quick PM for an update?

[tykoon]

Yeah, I Pm'ed him already.

[prosperent brian]

Contact turnkey. I worked last night with softlayer engineers and there are no issues we could find anywhere in the softlayer network. When running a test from a turnkey server to softlayer, it fails right at turnkey's network.

[prosperent brian]

Conversation with the engineer last night:


Here are the results from one of the users:

user@srv01:~$ sudo mtr -c 100 --report 173.192.23.139
HOST: srv01 Loss% Snt Last Avg Best Wrst StDev
1. 208-85-5-97.STATIC.turnkeyin 0.0% 100 0.7 0.8 0.7 1.7 0.2
2. ??? 100.0 100 0.0 0.0 0.0 0.0 0.0



Employee Response - 2011-Jul-20 23:52 (GMT-0600) [Update 6]

Hi,

Thank you for the update.

The mtr result provided by your customer is not complete. From the snippet that he provided, I cold see that the connections are not even going past their ISP. I would suggest to disable the firewall in the local machine or to contact the corresponding ISP for a resolution.

I ran an mtr test to the IP address "173.192.23.139" from the following URL.

===========
http://www.websitepulse.com/help/testtools.mtr-test.html
===========

The result of this test is not showing any packet loss in our network.

Please let me know if your require any further assistance.

Best Regards,
Dias D
SoftLayer Support

[tykoon]

I am waiting for my VPS's support to get back to me on this issue.

Thanks

[prosperent brian]

Keep me updated. Seems odd that is is only isolated to turnkey servers. If you run the test I posted, shoot me the results. If they look the same as what I quoted in my last post, it would seem there is some issue routing from turnkey to softlayer. If you get a different result, let me know so I can forward it to an engineer on my end and see if there is an issue there.

[tykoon]

Well I just tried to ping that IP and my Putty script just hung on the first ping...

[prosperent brian]

Yeah, that's not good. I got one packet to go through from our end, but it shoes 95 percent packet loss once it hits the turnkey network:


mtr -c 100 --report 208.85.5.99
app02.prosperent.com Snt: 100 Loss% Last Avg Best Wrst StDev
173.192.23.129-static.reverse.softlayer.com 0.0% 0.4 5.2 0.3 190.8 24.9
po105.dar01.sr01.dal01.networklayer.com 0.0% 0.3 5.1 0.2 193.3 24.0
ae6.bbr01.eq01.dal03.networklayer.com 0.0% 0.2 1.3 0.2 45.4 5.6
be-101-pe01.1950stemmons.tx.ibone.comcast.ne 0.0% 9.8 6.9 0.4 16.0 3.8
dal2-pr2-ae0-403.us.twtelecom.net 0.0% 0.3 3.2 0.3 98.3 11.8
aby1-ar3-xe-0-0-0-0.us.twtelecom.net 0.0% 43.5 46.0 43.5 102.0 9.8
jm20-ny1-ge-1-3-0.STATIC.turnkeyinternet.net 95.0% 46.4 46.3 44.3 47.3 1.2
c6509-ny1-i5ge50.turnkeyinternet.net 0.0% 43.9 44.4 43.9 74.9 3.3
208-85-5-99.STATIC.turnkeyinternet.net 0.0% 42.2 42.2 42.0 47.1 0.6

[tykoon]

sudo mtr -c 100 --report 173.192.23.139

I tried that and I get "sudo: mtr: command not found"

Did I type it wrong?

[prosperent brian]

This appears to be where the packetloss is taking place if you want to let their support know: jm20-ny1-ge-1-3-0.STATIC.turnkeyinternet.net

[tykoon]

My VPS support said this:



Yes, there appears to be a network issue on Softlayer's side. Traceroutes from the main VPS node drop off here:

10 po2.slr01.sr01.dal01.networklayer.com (66.228.118.142) 41.306 ms 41.311 ms 40.912 ms
11 * * *

There is nothing we can do about this, because it is on their side. You'll need to contact them about why it's being blocked/dropped.

[AcidRaZor]

My VPS support said this:

That's funny, because the SL dude said the same thing about it being on turnkey's side :D

[tykoon]

I know. Extremely frustrating. They are both blaming each other. Meanwhile my sites are all down...

[prosperent brian]

Let me forward this info to softlayer again and see what we can come up with.

[$imon]

If there was a problem at SL's end wouldn't that affect other server providers too rather than just Turnkey?

[prosperent brian]

That's what I would think as well.

[Zaphod]

This seem ok atm... I was having probs from TK as well as a New Jersey based host I use - I call it The Situation Hosting.

My traffic stats drop sharply to 1/5th normal between 11 AM and 12 PM yetserday.

[tykoon]

Mine just came back now!

I'd like to know who fixed what! ;)

[prosperent brian]

As would I.

[$imon]

Lol @ The Situation.

[tykoon]

Just got this back from my VPS suppport...


From what I understand, Turnkey disabled a certain type of packet reply to fight off a DDOS attack yesterday. Softlayer apparently requires this packet type during their internal routing, so the packets got dropped. As soon as TI re-enabled it, Softlayer allowed packets through again.

[prosperent brian]

Alright, so turnkey was blocking the requests. Makes sense. No one at softlayer could find anything wrong.

[Zaphod]

Good idea Turnkey.

Ensign Ricky: "Hmm.. we seem to be under attack, sir."

Turnkey Ship's Captin: "Might as well go ahead and raise shields and block traffic to and from a major ISP!"