Browsers failing with inline JavaScript Issues

10:19 AM] Rohith Poreddy: These are the changes I made to apache config

# X-FRAME Options
Header set X-Frame-Options SAMEORIGIN

# X-XSS-Protection:
Header always set X-XSS-Protection "1; mode=block"

# X-Content-Type-Options:
Header always set X-Content-Type-Options: nosniff

# Content-Security-Policy
Header set Content-Security-Policy "script-src 'self'; object-src 'self'"

[10:20 AM] Rohith Poreddy: In /etc/httpd/conf/httpd.conf

Secondary Flow Outage Google CDN Failure

Short Answer:

Yesterday’s Secondary Flow Outage was a UI issue, and not a DSSF issue.  Some of the AngularJS files for SF come from external sources which became unavailable.  There was a Google load balancing issue yesterday.

Long Answer:

I have a Splunk alert that fires whenever there have been zero secondary flow purchases within a 15 minute period.  It normally fires a few false alarms at night but some of the alarms started firing in the morning.  I sent the first email saying something might be wrong at 8:46am, and reached Rohith on HipChat.

Since the 500 alerts were not firing, the first guess was maybe the logs stop rolling.   That was not the case, so I just went to WWW and clicked the link the Secondary Flow, and the logon portal did not appear.  I opened an incident for anything that Rohith was doing could be logged.

This looks exactly like the symptom of a 500 error, but it was not.  Secondary Flow team took over the digging and Lenny and others found that the UI was not loading properly.

Load Balance Issue with Google:

https://status.cloud.google.com/incident/cloud-networking/17002

Example of incident reported by Google

Aug 30, 2017 09:30

We are experiencing an issue with a subset of Network Load Balance. The configuration change to mitigate this issue has been rolled out and we are working on further measures to completely resolve the issue. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 10:30 US/Pacific with current details.

Examples of CDN’s Used:

https://ajax.googleapis.com/ajax/libs/angularjs/1.5.0/angular.min.js

https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css

Looking for open file counts

Logon to a server and get the PID for an application

$ ps -aux | grep -i <application>

Count of Files
$ sudo lsof -a -p <PID> | wc -l

List of Files
$ sudo lsof -a -p <PID> | more

From: "Mamidi, Sundeep (Capgemini)" <SMamidi@hollandamerica.com>
Subject: RE: Splunk Alert: MLR - Secondary Flow API 500 Errors Watch
Date: August 15, 2017 at 10:38:16 AM PDT

It’s the same on p013 as well (4096).

Here’s the command :

lsof –a –p <pid> | wc –l à This gives the number of connected files
lsof –a –p <pid> à This gives the list of files that are connected

As of now, it shows around 700-750 on both servers. What I’m hoping is the file connections are being stalled on one of the servers, reaching the max connections limit. When we do a restart, it refreshes. As I said, I could increase the number of max.connections. However, that’s not the best way of handling this issue.

What possibly could happen, if we increase the max connections is, the application would consume all of those (even 10000) and give us the same error again.

There should be a bigger issue, somewhere on the backend that’s causing the stale connections. Looking at the AppD, I see some errors on hal-porta as well, at the same time.

Thanks,
Sundeep

Install HAL Certs for IntelliJ

Copy the HALIntellJCerts.zip (Download) to here:
/usr/lib/jvm/java-8-oracle/jre/lib/security

Trust All for Maven - When you have NO Valid Certs
mvn install -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -Dmaven.wagon.http.ssl.ignore.validity.dates=true

Import certs into java keystore

# Copy the certificate into the directory Java_home\Jre\Lib\Security
# Change your directory to Java_home\Jre\Lib\Security>
# Import the certificate to a trust store.

CD to your security folder on Ubuntu
$ cd /usr/lib/jvm/java-8-oracle/jre/lib/security

Copy all the certifcates there
Downloads/Certs/AddTrustExternalCARoot.cer
Downloads/Certs/COMODOAddTrustExternalCARoot.cer
Downloads/Certs/COMODORSACertAuthority.cer
Downloads/Certs/New_Princess_COMODO_RSA_Organization_Validation_Secure_Server_CA.cer

$ keytool -import -alias comodoaddtrustexternalcaroot -file COMODOAddTrustExternalCARoot.cer -keystore cacerts -storepass changeit
$ keytool -import -alias comodorsacertauthority -file COMODORSACertAuthority.cer -keystore cacerts -storepass changeit
$ keytool -import -alias addtrustexternalca -file AddTrustExternalCARoot.cer -keystore cacerts -storepass changeit

Trust this certificate: [Yes]

Mac Java Location
echo $(/usr/libexec/java_home)
/Library/Java/JavaVirtualMachines/jdk1.8.0_71.jdk/Contents/Home

/Library/Java/JavaVirtualMachines/jdk1.8.0_71.jdk/Contents/Home/jre/lib/security

 

Akamai Error Lookup

Logon to Luna Control Center

Support Link

More Tools...

Translate Error String

Paste in the string to translate