From: Augustine, Anila (HAL)
Sent: Monday, July 30, 2012 8:16 AM
To: Guillaume Radde
Cc: Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL); Thompson, Sonya (HAL)
Subject: RE: Looking for Thread Deadlocks on JBoss
Cool! That’s easy J. I mentioned this to Don on Friday too. Good news is we will be getting another linux admin (possibly this week) – woohoo! That should free up 40 hours for Don.
From: Guillaume Radde [mailto:guillaume.radde@redhat.com]
Sent: Friday, July 27, 2012 10:01 AM
To: Augustine, Anila (HAL)
Cc: Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL); Thompson, Sonya (HAL)
Subject: Re: Looking for Thread Deadlocks on JBoss
Around 40 man-hour. Depends on how many environments we want to monitor.
http://www.redhat.com/v/swf/jboss_on/2122_RedHat_JONOverview.html
From: "Anila Augustine (HAL)" <AAugustine@HollandAmerica.com>
To: "Guillaume Radde" <guillaume.radde@redhat.com>
Cc: "Maurice Ruckman (HAL)" <MRuckman@HollandAmerica.com>, "Eric Fillman (HAL)" <EFillman@HollandAmerica.com>, "Brian Klein (HAL)" <BKlein@HollandAmerica.com>, "Sonya Thompson (HAL)" <SThompson@HollandAmerica.com>
Sent: Friday, July 27, 2012 9:11:27 AM
Subject: RE: Looking for Thread Deadlocks on JBoss
So what would it take to deploy JON?
From: Guillaume Radde [mailto:guillaume.radde@redhat.com]
Sent: Thursday, July 26, 2012 6:46 PM
To: Augustine, Anila (HAL)
Cc: Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL); Thompson, Sonya (HAL)
Subject: Re: Looking for Thread Deadlocks on JBoss
I will add jmx-console. JBoss Operation Network is a separate monitoring tool (kind of like Foglight) that we did not deploy.
From: "Anila Augustine (HAL)" <AAugustine@HollandAmerica.com>
To: "Guillaume Radde" <guillaume.radde@redhat.com>, "Sonya Thompson (HAL)" <SThompson@HollandAmerica.com>
Cc: "Maurice Ruckman (HAL)" <MRuckman@HollandAmerica.com>, "Eric Fillman (HAL)" <EFillman@HollandAmerica.com>, "Brian Klein (HAL)" <BKlein@HollandAmerica.com>
Sent: Thursday, July 26, 2012 2:52:34 PM
Subject: RE: Looking for Thread Deadlocks on JBoss
Guillaume - in your documentation, please add JBoss Operation Network - I think the team would benefit from a show & tell, if that’s not done already.
From: Guillaume Radde [mailto:guillaume.radde@redhat.com]
Sent: Thursday, July 26, 2012 10:31 AM
To: Thompson, Sonya (HAL)
Cc: Augustine, Anila (HAL); Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL)
Subject: Re: Looking for Thread Deadlocks on JBoss
There are a lot of things that can be seen through the jmx-console. Globally, it can be used to review the internal of a JBoss server while it's running. The downside is that it's hard to find what you're looking for. I usually just google what I'm looking for with "jmx-console"
From: Augustine, Anila (HAL)
Sent: Monday, July 30, 2012 8:16 AM
To: Guillaume Radde
Cc: Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL); Thompson, Sonya (HAL)
Subject: RE: Looking for Thread Deadlocks on JBoss
Cool! That’s easy J. I mentioned this to Don on Friday too. Good news is we will be getting another linux admin (possibly this week) – woohoo! That should free up 40 hours for Don ;-)
From: Guillaume Radde [mailto:guillaume.radde@redhat.com]
Sent: Friday, July 27, 2012 10:01 AM
To: Augustine, Anila (HAL)
Cc: Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL); Thompson, Sonya (HAL)
Subject: Re: Looking for Thread Deadlocks on JBoss
Around 40 man-hour. Depends on how many environments we want to monitor.
http://www.redhat.com/v/swf/jboss_on/2122_RedHat_JONOverview.html
From: "Anila Augustine (HAL)" <AAugustine@HollandAmerica.com>
To: "Guillaume Radde" <guillaume.radde@redhat.com>
Cc: "Maurice Ruckman (HAL)" <MRuckman@HollandAmerica.com>, "Eric Fillman (HAL)" <EFillman@HollandAmerica.com>, "Brian Klein (HAL)" <BKlein@HollandAmerica.com>, "Sonya Thompson (HAL)" <SThompson@HollandAmerica.com>
Sent: Friday, July 27, 2012 9:11:27 AM
Subject: RE: Looking for Thread Deadlocks on JBoss
So what would it take to deploy JON?
From: Guillaume Radde [mailto:guillaume.radde@redhat.com]
Sent: Thursday, July 26, 2012 6:46 PM
To: Augustine, Anila (HAL)
Cc: Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL); Thompson, Sonya (HAL)
Subject: Re: Looking for Thread Deadlocks on JBoss
I will add jmx-console. JBoss Operation Network is a separate monitoring tool (kind of like Foglight) that we did not deploy.
From: "Anila Augustine (HAL)" <AAugustine@HollandAmerica.com>
To: "Guillaume Radde" <guillaume.radde@redhat.com>, "Sonya Thompson (HAL)" <SThompson@HollandAmerica.com>
Cc: "Maurice Ruckman (HAL)" <MRuckman@HollandAmerica.com>, "Eric Fillman (HAL)" <EFillman@HollandAmerica.com>, "Brian Klein (HAL)" <BKlein@HollandAmerica.com>
Sent: Thursday, July 26, 2012 2:52:34 PM
Subject: RE: Looking for Thread Deadlocks on JBoss
Guillaume – in your documentation, please add JBoss Operation Network - I think the team would benefit from a show & tell, if that’s not done already.
From: Guillaume Radde [mailto:guillaume.radde@redhat.com]
Sent: Thursday, July 26, 2012 10:31 AM
To: Thompson, Sonya (HAL)
Cc: Augustine, Anila (HAL); Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL)
Subject: Re: Looking for Thread Deadlocks on JBoss
There are a lot of things that can be seen through the jmx-console. Globally, it can be used to review the internal of a JBoss server while it's running. The downside is that it's hard to find what you're looking for. I usually just google what I'm looking for with "jmx-console"
From: "Sonya Thompson (HAL)" <SThompson@HollandAmerica.com>
To: "Maurice Ruckman (HAL)" <MRuckman@HollandAmerica.com>, "Eric Fillman (HAL)" <EFillman@HollandAmerica.com>, "Brian Klein (HAL)" <BKlein@HollandAmerica.com>
Cc: "Anila Augustine (HAL)" <AAugustine@HollandAmerica.com>, "Guillaume Radde (guillaume.radde@redhat.com) (guillaume.radde@redhat.com)" <guillaume.radde@redhat.com>
Sent: Thursday, July 26, 2012 9:09:58 AM
Subject: RE: Looking for Thread Deadlocks on JBoss
Guillaume & Maurice,
Thanks so much for sharing this info with us!
~Sonya
From: Ruckman, Maurice (HAL)
Sent: Thursday, July 26, 2012 7:49 AM
To: Fillman, Eric (HAL); Thompson, Sonya (HAL); Klein, Brian (HAL)
Cc: Augustine, Anila (HAL); Guillaume Radde (guillaume.radde@redhat.com) (guillaume.radde@redhat.com)
Subject: RE: Looking for Thread Deadlocks on JBoss
Link to Console
haldevjbsXX.hq.halw.com/jmx-console/
admin/admin
List of sessions:
jboss.web
host=localhost,path=/,type=Manager
listSessionIds
Looking for a thread deadlock, how to
jboss.system
type=ServerInfo
listThreadCpuUtilization
Look for a thread taking a huge amount of CPU
Such as ajp-haldevjbs04%2F10.194.50.211-8009-3
and go back to type=ServerInfo
listThreadDump
and then search for thread found previously
From: Fillman, Eric (HAL)
Sent: Thursday, July 26, 2012 6:33 AM
To: Ruckman, Maurice (HAL); Thompson, Sonya (HAL); Klein, Brian (HAL)
Cc: Augustine, Anila (HAL); Guillaume Radde (guillaume.radde@redhat.com) (guillaume.radde@redhat.com)
Subject: RE: Looking for Thread Deadlocks on JBoss
I’m not following this.
Are you supposed to type this stuff in on command line or something or in the JBoss console?
From: Ruckman, Maurice (HAL)
Sent: Wednesday, July 25, 2012 4:47 PM
To: Fillman, Eric (HAL); Thompson, Sonya (HAL); Klein, Brian (HAL)
Cc: Augustine, Anila (HAL); Guillaume Radde (guillaume.radde@redhat.com) (guillaume.radde@redhat.com)
Subject: Looking for Thread Deadlocks on JBoss
Guillaume showed me how he found the deadlocked thread. Here are the notes:
Looking for a thread deadlock, how to
jboss.web
host=localhost,path=/,type=Manager
Looking for deadlocks
jboss.system
type=ServerInfo
listThreadCpuUtilization
Look for a thread taking a huge amount of CPU
Such as ajp-haldevjbs04%2F10.194.50.211-8009-3
and go back to type=ServerInfo
listThreadDump
and then search for thread
Thanks,
From: Augustine, Anila (HAL)
Sent: Monday, July 30, 2012 8:16 AM
To: Guillaume Radde
Cc: Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL); Thompson, Sonya (HAL)
Subject: RE: Looking for Thread Deadlocks on JBoss
Cool! That’s easy J. I mentioned this to Don on Friday too. Good news is we will be getting another linux admin (possibly this week) – woohoo! That should free up 40 hours for Don ;-)
From: Guillaume Radde [mailto:guillaume.radde@redhat.com]
Sent: Friday, July 27, 2012 10:01 AM
To: Augustine, Anila (HAL)
Cc: Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL); Thompson, Sonya (HAL)
Subject: Re: Looking for Thread Deadlocks on JBoss
Around 40 man-hour. Depends on how many environments we want to monitor.
http://www.redhat.com/v/swf/jboss_on/2122_RedHat_JONOverview.html
From: "Anila Augustine (HAL)" <AAugustine@HollandAmerica.com>
To: "Guillaume Radde" <guillaume.radde@redhat.com>
Cc: "Maurice Ruckman (HAL)" <MRuckman@HollandAmerica.com>, "Eric Fillman (HAL)" <EFillman@HollandAmerica.com>, "Brian Klein (HAL)" <BKlein@HollandAmerica.com>, "Sonya Thompson (HAL)" <SThompson@HollandAmerica.com>
Sent: Friday, July 27, 2012 9:11:27 AM
Subject: RE: Looking for Thread Deadlocks on JBoss
So what would it take to deploy JON?
From: Guillaume Radde [mailto:guillaume.radde@redhat.com]
Sent: Thursday, July 26, 2012 6:46 PM
To: Augustine, Anila (HAL)
Cc: Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL); Thompson, Sonya (HAL)
Subject: Re: Looking for Thread Deadlocks on JBoss
I will add jmx-console. JBoss Operation Network is a separate monitoring tool (kind of like Foglight) that we did not deploy.
From: "Anila Augustine (HAL)" <AAugustine@HollandAmerica.com>
To: "Guillaume Radde" <guillaume.radde@redhat.com>, "Sonya Thompson (HAL)" <SThompson@HollandAmerica.com>
Cc: "Maurice Ruckman (HAL)" <MRuckman@HollandAmerica.com>, "Eric Fillman (HAL)" <EFillman@HollandAmerica.com>, "Brian Klein (HAL)" <BKlein@HollandAmerica.com>
Sent: Thursday, July 26, 2012 2:52:34 PM
Subject: RE: Looking for Thread Deadlocks on JBoss
Guillaume – in your documentation, please add JBoss Operation Network - I think the team would benefit from a show & tell, if that’s not done already.
From: Guillaume Radde [mailto:guillaume.radde@redhat.com]
Sent: Thursday, July 26, 2012 10:31 AM
To: Thompson, Sonya (HAL)
Cc: Augustine, Anila (HAL); Ruckman, Maurice (HAL); Fillman, Eric (HAL); Klein, Brian (HAL)
Subject: Re: Looking for Thread Deadlocks on JBoss
There are a lot of things that can be seen through the jmx-console. Globally, it can be used to review the internal of a JBoss server while it's running. The downside is that it's hard to find what you're looking for. I usually just google what I'm looking for with "jmx-console"
From: "Sonya Thompson (HAL)" <SThompson@HollandAmerica.com>
To: "Maurice Ruckman (HAL)" <MRuckman@HollandAmerica.com>, "Eric Fillman (HAL)" <EFillman@HollandAmerica.com>, "Brian Klein (HAL)" <BKlein@HollandAmerica.com>
Cc: "Anila Augustine (HAL)" <AAugustine@HollandAmerica.com>, "Guillaume Radde (guillaume.radde@redhat.com) (guillaume.radde@redhat.com)" <guillaume.radde@redhat.com>
Sent: Thursday, July 26, 2012 9:09:58 AM
Subject: RE: Looking for Thread Deadlocks on JBoss
Guillaume & Maurice,
Thanks so much for sharing this info with us!
~Sonya
From: Ruckman, Maurice (HAL)
Sent: Thursday, July 26, 2012 7:49 AM
To: Fillman, Eric (HAL); Thompson, Sonya (HAL); Klein, Brian (HAL)
Cc: Augustine, Anila (HAL); Guillaume Radde (guillaume.radde@redhat.com) (guillaume.radde@redhat.com)
Subject: RE: Looking for Thread Deadlocks on JBoss
Link to Console
haldevjbsXX.hq.halw.com/jmx-console/
admin/admin
List of sessions:
jboss.web
host=localhost,path=/,type=Manager
listSessionIds
Looking for a thread deadlock, how to
jboss.system
type=ServerInfo
listThreadCpuUtilization
Look for a thread taking a huge amount of CPU
Such as ajp-haldevjbs04%2F10.194.50.211-8009-3
and go back to type=ServerInfo
listThreadDump
and then search for thread found previously
From: Fillman, Eric (HAL)
Sent: Thursday, July 26, 2012 6:33 AM
To: Ruckman, Maurice (HAL); Thompson, Sonya (HAL); Klein, Brian (HAL)
Cc: Augustine, Anila (HAL); Guillaume Radde (guillaume.radde@redhat.com) (guillaume.radde@redhat.com)
Subject: RE: Looking for Thread Deadlocks on JBoss
I’m not following this.
Are you supposed to type this stuff in on command line or something or in the JBoss console?
From: Ruckman, Maurice (HAL)
Sent: Wednesday, July 25, 2012 4:47 PM
To: Fillman, Eric (HAL); Thompson, Sonya (HAL); Klein, Brian (HAL)
Cc: Augustine, Anila (HAL); Guillaume Radde (guillaume.radde@redhat.com) (guillaume.radde@redhat.com)
Subject: Looking for Thread Deadlocks on JBoss
Guillaume showed me how he found the deadlocked thread. Here are the notes:
Looking for a thread deadlock, how to
jboss.web
host=localhost,path=/,type=Manager
Looking for deadlocks
jboss.system
type=ServerInfo
listThreadCpuUtilization
Look for a thread taking a huge amount of CPU
Such as ajp-haldevjbs04%2F10.194.50.211-8009-3
and go back to type=ServerInfo
listThreadDump
and then search for thread
Thanks,
Maurice Ruckman
Voice: (206) 626-7497 x67497
Cellular: (206) 650-7576
Maurice Ruckman
Voice: (206) 626-7497 x67497
Cellular: (206) 650-7576