Author: Maurice Ruckman

Posted on

Secondary Flow Outage Google CDN Failure

Short Answer:

Yesterday’s Secondary Flow Outage was a UI issue, and not a DSSF issue.  Some of the AngularJS files for SF come from external sources which became unavailable.  There was a Google load balancing issue yesterday.

Long Answer:

I have a Splunk alert that fires whenever there have been zero secondary flow purchases within a 15 minute period.  It normally fires a few false alarms at night but some of the alarms started firing in the morning.  I sent the first email saying something might be wrong at 8:46am, and reached Rohith on HipChat.

Since the 500 alerts were not firing, the first guess was maybe the logs stop rolling.   That was not the case, so I just went to WWW and clicked the link the Secondary Flow, and the logon portal did not appear.  I opened an incident for anything that Rohith was doing could be logged.

This looks exactly like the symptom of a 500 error, but it was not.  Secondary Flow team took over the digging and Lenny and others found that the UI was not loading properly.

Load Balance Issue with Google:

https://status.cloud.google.com/incident/cloud-networking/17002

Example of incident reported by Google

Aug 30, 2017 09:30

We are experiencing an issue with a subset of Network Load Balance. The configuration change to mitigate this issue has been rolled out and we are working on further measures to completely resolve the issue. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 10:30 US/Pacific with current details.

Examples of CDN’s Used:

https://ajax.googleapis.com/ajax/libs/angularjs/1.5.0/angular.min.js

https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css

Posted on

Looking for open file counts

Logon to a server and get the PID for an application

$ ps -aux | grep -i <application>

Count of Files
$ sudo lsof -a -p <PID> | wc -l

List of Files
$ sudo lsof -a -p <PID> | more

From: "Mamidi, Sundeep (Capgemini)" <SMamidi@hollandamerica.com>
Subject: RE: Splunk Alert: MLR - Secondary Flow API 500 Errors Watch
Date: August 15, 2017 at 10:38:16 AM PDT

It’s the same on p013 as well (4096).

Here’s the command :

lsof –a –p <pid> | wc –l à This gives the number of connected files
lsof –a –p <pid> à This gives the list of files that are connected

As of now, it shows around 700-750 on both servers. What I’m hoping is the file connections are being stalled on one of the servers, reaching the max connections limit. When we do a restart, it refreshes. As I said, I could increase the number of max.connections. However, that’s not the best way of handling this issue.

What possibly could happen, if we increase the max connections is, the application would consume all of those (even 10000) and give us the same error again.

There should be a bigger issue, somewhere on the backend that’s causing the stale connections. Looking at the AppD, I see some errors on hal-porta as well, at the same time.

Thanks,
Sundeep

Posted on

WordPress Unable to Upload Files

Refer: https://goo.gl/iZ56R3

  • Log into your WordPress admin dashboard
  • Go to Media Settings menu (Settings -> Media)
  • Enter wp-content/uploads in the “Store uploads in this folder” text box
  • Save the changes by hitting the “Save Changes” button and you are done