[Sticky] 3 Weeks To Cycle One NHI Password

So why did it take 3 weeks to cycle one NHI password?

This is a tale of hardcoded passwords, lack of environment segregation, password cycling, monitoring controls, sharing of credentials and human use of NHIs.

At a large financial institution I get pulled into help deal with a major business impacting production issue by my CISO. All I was told initially, it was believed to be related to a NHI.

On initial investigation in turned out someone in IT was trying to make a change in a QA database, but accidentally connected to production.

• Issue 1 - the IT staff member was inappropriately using a NHI account to access and update a QA DB.
• Issue 2 - the NHI account in question had the same password in both production and non-production, so a clear issue around lack of environment segregation.

So given the magnitude of the business impact, I get told we need to get the password for this DB NHI account cycled asap.

In speaking to the application team responsible for the account and database, the flagged a few issues.

• Issue 3 - the team did not know all the places where the NHI was referenced in code and scripts, the password was hardcoded in plain text and not in a secrets vault.
• Issue 4 - the team knew the password had been shared to some friend and family application teams also connecting to the DB.

So our long journey began on why it took 3 weeks to cycle the NHI password.

Whilst the application team knew some applications that we using/sharing their NHI, they did not know all of them, given change in staff in the team, so some of the history on when credentials were shared were unknown.

• Issue 5 - there was a lack of monitoring controls.

I had to work with our DBA engineers, to turn on some crude form of monitoring controls on the DB in question. After a few days we were able to work out all the IP addresses where connections were coming in using the specific NHI in question.

We then went on a long journey contacting, working with all the application teams that were sharing the one production NHI DB account and for each team we had to create a new NHI DB account for them, and ask them to move over to this new NHI dedicated for their application.

As you can imagine for each application team to move to a new NHI meant they needed to look at all their code/scripts to see where the shared NHI was being used and then making coding changes to migrate to the new NHI. They first however needed to test this out with a new NHI account on a QA DB, check everything was working and then repeat the same process in production, performing multiple production releases, to ensure any operational impact was minimised.

Clearly given the risk of operational impact to production, given so many applications, scripts needing changing, this whole effort took approximately 3 weeks to complete, just to cycle one production NHI !

As a final belt and braces verification, we had to check the custom monitoring control solution we had implemented to ensure the original NHI DB account in question, was only receiving incoming connections from the application that owned the NHI from known servers it uses and no other connections were coming in.

• Issue 6 - lack of password cycling - whilst cycling is hard, if we had this control requirement in place, we would have picked many of the above risks which would have been mitigated, and may not had lead to the original incident in first place, or at least minimised the impact or resolution time.

After this significant business impacting incident my CISO and head of IAM realised how big the NHI problem was and immediately asked me (Mr NHI) to start a huge global NHI program.

In conclusion, this real example is a very stark reminder of the huge risks around NHIs and hope this helps enforce the message this is one risk exposure you need to get under control within your organisation.