At Analytics8 we leverage AWS EC2 for many of our testing environments. These testing environments are intended to be very temporary, so only the bare resources necessary are required to use the server setup. Recently, I ran into an issue with one such server – here’s how I fixed the problem.
I installed a Qlik Sense server a number of weeks ago that was used for a client specific demo. When the demo was over, I powered off the EC2 instance to save costs. Now, the client needed something off the server and asked to power it on again. This is a fairly common scenario and works regularly without a problem.
Booting up the Qlik Sense server, I RDP’d in and clicked on the desktop icon to open QMC where I was greeted with a Page Not Found error. My first assumption was that the services hadn’t started. Qlik Sense services start on a delay so it’s not uncommon for me to try to connect to the to the QMC before it’s ready. Checking the services though, I found them running.
The Qlik Sense server was set up initially to use the https://<ComputerName>.a8test.com. This is a good best practice in AWS since the internal computer name doesn’t change. (The guide to install on AWS can be found here.) However, desperate times call for desperate measures so I tried using localhost, 127.0.0.1, the external DNS, all with no luck.
At this point I had to start digging deep. Remember, this machine was turned off weeks ago when it was working and now it wasn’t. I double checked to make sure the Windows firewall was off (don’t do this in a production environment). I verified ports were open and listening. Then I checked the logs and kicked myself for not checking there first.
In C:\ProgramData\Qlik\Sense\Log\Repository\Trace there should be 3 log files; Audit, Security and System. In my folder there were 30+ files with an appended date. The historical log files should get written to the repository so seeing that many of them tells me something is wrong, but not what. In the System Repository log I a Fatal Exception during bootstrap error indicating the Keyset does not exist. I wish I could say I know exactly what that means but I don’t. What I do know is that that particular error is telling me that something is likely wrong with the certificates that were set up during the Qlik Sense install.
I had run into this issue before when a client had changed the hostname of the Qlik Sense Server. I reached directly out to Qlik support and they provided a link to a support article describing the steps to change the hostname and reinstall the certificates. I didn’t change the hostname on the AWS instance but it seems possible that Amazon moved my virtual instance from one physical host to another while it was off and therefor changing some deep memory address that was being used for the certificates. This would make sense based on the little bit I know about Amazon EC2 instances. For those of you who have an Instance that stays on all the time, or, only goes off in the evenings, I doubt this would be an issue. But for those of us who let a powered off instance sit for a while, I can imagine Amazon moving it to optimize the use of their cloud computing platform. Regardless of what happened, I decided to use the support article to try to reinstall the certificates.
After backing up all the Qlik Sense related certificates in Local Computer Personal, Local Computer TRCA, Current User Personal and Current User TRCA, I deleted them. I also deleted the %ProgramData%\Qlik\Sense\Repository\Exported Certificates\.Local Certificates folder. Lastly, I ran “C:\Program Files\Qlik\Sense\Repository\Repository.exe” -bootstrap -iscentral -restorehostname command using using an elevated command prompt. Once it finished I started the Qlik Sense services and voila, everything worked again.
The referenced support article gives a lot more detailed information that you should reference if trying this at home. Also, I would be remiss if I led you to believe I figured out all the above on my own. Qlik Support was very helpful in getting to the bottom of this. If you’re experiencing the same issue, I hope this helps a bit.
To thrive with your data, your people, processes, and technology must all be data-focused. This may sound daunting, but we can help you get there. Sign up to meet with one of our analytics experts who will review your data struggles and help map out steps to achieve data-driven decision making.