HPOM v9 Config Sync Issue Workaround

I saw an issue the other day that suggested the automatic configuration synchronisation between primary and DR HPOM v9 management servers (using opccfgsync) wasn't working entirely as HP have documented.

We had a space issue on a DR server and on closer examination I saw that despite the config being copied across as expected, the cleanup variables (as described in the Admin Ref guide) were being ignored).  Subsequently, the old sync files were not being deleted and therefore the filesystem was filling up.

On investigation into the sync logfiles, I noticed a couple of errors listed in some files, so my theory is that because of this error the sync process just flips out and doesn't exit cleanly and continue with the cleanup operation.  You'd think there would be an error reported, but no.  Especially when I was running the sync as part of an HPOM schedule policy.

After a lot of head scratching, I eventually started playing with the scenario file and tweaking the original one suggested by HP to not include various settings, just to see if that made any difference.

Bingo!

In the end by omitting the FORWARDING_MANAGER and RESPONSIBLE_MANAGER lines (I am guessing the names as I am not in the office now) that worked and the config was copied (as indeed it seemed to have been anyway), but critically, the housekeeping now worked as designed and the number of sync files reduced to 3, deleting all of the older ones (as I wanted).

Hurrah!

Hope this helps anyone else having issues with the sync process & housekeeping.  If you have any questions just fire me a message or check me out on FB at https://www.facebook.com/EnterpriseMonitoring

All the best,

Dave Gerrish

www.protocolsoftware.com - new website coming soon

twitter - @openview

HPOM Migration to OMi - Preparation

Just starting to take a look at OMi to plan migration from HPOM v9 - looks like a completely different and re-worked front-end.

First thing I need to do is work out all the main priorities before I can plan this in more detail - for example working out where the equivalent config items from HPOM v9 are in OMi - then I can check where functionality has been superseded in the new tool, or where any rework is required.

I have a rough plan jotted down already, so will share this with you in due course.

If you've migrated to OMi I'd appreciate your views on approach/problems etc., or if you are considering the migration yourself please sign-up and then I can keep you updated as to progress, and hopefully help you out also!

Have a great day,
Dave

Synchronising 2 NNMi Servers (Production and DR)

Synchronising 2 NNMi servers is not simple and straightforward as you would hope/expect.

Unlike HPOM (OML) for example, where you can simple download configuration and upload it again (or in the case of v9.2+ where you can automatically synchronise 2 systems), NNMi doesn't have a neat way to achieve this.

The situation is over complicated with the 2 database options used by NNMi - Oracle and the embedded Postgres database.

According to HP, the Postgres/embedded version has better options for exporting and importing the data to a 2nd server for DR purposes, but using Oracle the solution is far from simple or elegant.  I can't even get a clearly-defined process from HP support which is worrying.

At the moment I am experimenting with the export and import tools, which seems to cover all of the config settings, but at the moment it's reporting errors importing the trap definitions into the DR NNMi server.  This is critical of course, because if the DR server doesn't have the correct trap definitions configured then the alerts won't look the same on the server if we failover or go into DR (disaster recovery) mode.

I'm working with HP support to resolve this, but once again it highlights the poor development of these tools and the lack of adequate and sufficient error reporting.

I'll let you know how I get on.

Dave

Alerts owned by NNMi stuck in HPOM

I previously mentioned on the blog (http://tiny.cc/protocolblog) about a strange problem where I see alerts owned in...

Posted by Protocol - Infrastructure Monitoring on Wednesday, 27 January 2016

NNMi alerts owned by opc_adm in HPOM

Have a current weird problem, where I see alerts owned in HPOM by opc_adm due to the web services integration of NNMi with HPOM.

I can't see it documented anywhere (as usual so often with these things!).  HP support are investigating.

If anyone else has any experience of this please share your tips.

Thanks,
Dave

Logfile Monitoring Delays?

Monitoring logfiles using a policy with hundreds and hundreds of conditions can cause HPOM's logfile encapsulator (opcle) to run into problems.

I have seen delays in message receive time on the HPOM manager in numerous HPOM installations, sometimes hours behind the original logfile alert time, which is obviously unacceptable in production monitoring environments.

HP support couldn't offer much help, but a few key principles can help reduce this problem and hopefully eradicate it completely.

1. Try and ensure the logfile being monitored is named specifically, or if using a command to generate a list of logfiles, try and make sure this list is small
2. Try and keep the number of policy conditions to a minimum where possible
3. Try and ensure as much suppression is placed at the top of the logfile monitoring policy to ensure opcle matches and drops unncessary text as soon as possible (thereby freeing it up to parse additional lines)
4. Consider changing the environment variable which allows opcle to read multiple lines at a time - for example OPC_LE_MAX_LINES_READ
5. Review the logfile polling period

Hope this helps.
Dave

MSI Issues after 9.2x upgrade of HP Operations Manager

Bear in mind if you use the MSI (Message Stream Interface) feature within HP Operations Manager HPOM) and have upgraded to v9.2 you could witness this problem.

If you have an ECS (event correlation circuit) that takes all messages from the MSI, and performs some action on some or all of the alerts, you could see behaviour as if the message was being copied to the MSI rather than being diverted to the MSI.  This can cause unexpected behavour such as duplicate messages, or active messages where you expected suppression (on further investigation you will probably see that duplication is causing a message to go via the MSI and get suppressed, but the copy is not sent to the MSI and is therefore not suppressed).

Solution:

Since OML 9.2x there are several MSI applications provided by HPOM itself - opcesf, opchcd, opcecm.  If you don't set their order in msiconf, all of them have the same priority if they are enabled.  Because of that all of them get the same messages at the same time and it appears as copy method would be used.

I checked again your msiconf file, and it looks like :
HealthCheck 1
esf 11

Please change it to become :
HealthCheck 1
opcesf 11
opcecm 21

After that, please restart opcecm :
# ovc -restart opcecm

You can then test to ensure the changes have been successful and the message behaviour is as expected.

Dave

Bottleneck Alert Probability 100% Annoying?

If you don't want to see the default (annoying!) Performance related alerts in HPOM, the ones similar to this...

Network Bottleneck Probability 100%

... then I hope you realise you can turn them off!

I saw one client who forget to do this, and due to the messages not being seen by any user profile or account, the quantity of these alerts was HUGE, and took a long time to get cleared-down manually, wasting time & effort.

I'll write more about these alerts in the future, and how they CAN be of use, but they need to be configured carefully and with planning, thought and care, as they are not intuitive or helpful out-of-the-box.

Here's how you check the current status of the performance agent:

/opt/perf/bin/agsysdb –l

And this is the command to turn off the sending of performance alerts to HPOM:

/opt/perf/bin/agsysdb –ovo off

Bear in mind that agent upgrades can turn this feature back on, so you might want to ensure your procedures include steps to turn it off again if required.

Alternatively, you could create a monitor to alert you if it gets enabled!  Now there's a thought...

Dave

Java GUI Issues in HPOM

I've seen issues in the past where Java GUI alerts in HP Operations Manager (HPOM) v9.x don't get owned or acknowledged when you trigger those actions.

If you reload the Java GUI console then the action you previously performed works, i.e. the alert is owned or acknowledged, but it's not being done automatically.

HP support came up with this fix - which seemed to work:

# /opt/OV/bin/ovc -stop SERVER
# rm -rf /var/opt/OV/share/tmp/OpC/mgmt_sv/WAP*
# rm -rf /var/opt/OV/share/tmp/OpC/mgmt_sv/WWW*
# rm -rf /var/opt/OV/share/tmp/OpC/ui/*
# /opt/OV/bin/ovc -start

I have also seen the Java GUI hang on the "Loading Messages" screen, and we think this was due to different versions of the Java GUI being used, so I strongly recommend if you see this issue keep an eye on the versions being used (use the listguis command) and try to align all users to the latest version.

Dave

Welcome

Hi,

Welcome to my new blog - this is where I am going to share comments, experience, tips and anything else I think of that might be of use to you, in and around the area of IT infrastructure monitoring.

I am very experienced in using HP Operations Manager (HPOM) software (formerly known as HP OpenView), but will try and also make sure these posts are of use to anyone working in the area of IT monitoring systems in general, as many of the issues, problems and goals remain the same, regardless of the tools used.

Thanks,
Dave