Aptible recently introduced ElasticSearch 5.0 as one of two supported ElasticSearch versions. With this new version, Aptible also released an update to their Kibana docker repo allowing you to use Kibana 5.0 with ElasticSearch 5.0. The upgrade to Kibana 5 from Kibana 4 involved some much needed hefty changes and is a very welcomed upgrade! Check out the changes here.
Assumptions
This guide assumes the following:
- You’re using Aptible’s docker-kibana repository
- You’re using Aptible’s ElasticSearch 2.2 database
If you aren’t using Aptible’s repository, you can still use this guide for migrating your indices from ElasticSearch 2.2 to ElasticSearch 5.0. You can follow a standard upgrade path from Kibana 4.2 or later without performing a reindex. If you’re upgrading from 4.1 and earlier, you’ll need to perform a reindex. A migration guide for this can be found here.
Prerequisites
In order to perform the upgrade with this guide, you will need Aptible Toolbelt, Node JS, NPM, and ElasticDump. Follow the instructions below to perform the install.
Aptible Toolbelt
Install the Aptible Toolbelt from here. Some options in the guide can be performed via the Aptible dashboard at https://dashboard.aptible.com.
Node JS
Get the package and install accordingly. The following link will provide you the latest version of node for your operating system.
https://nodejs.org/en/
Alternatively, you can use almost any package manager for your OS, such as homebrew for macOS.
brew install node
NPM
NPM is Node JS’s package manager, which allows you to install nifty tools to assist you with development or complicated tasks. Once you have node installed, you should already have npm. It’s recommended to upgrade npm to the latest so you don’t run into any issues.
sudo npm install npm -g
ElasticDump
ElasticDump is the tool we’re going to use to make transferring your indices from 2.2 to 5.0 a breeze. You can install this globally using the -g flag or you can install it to the current directory you’re on without providing the flag. For simplicity, we’ll install globally so it can be executed from everywhere and the guide is easier to follow.
npm install elasticdump -g
The Upgrade Process
Our process here is to create a new Aptible ElasticSearch 5.0 database, migrate the indices from the old to the newly created database, and then update our configuration parameters, resulting in an upgrade to Kibana 5.0. The process is actually very easy, but involves a few commands and some time to migrate the data.
However, before we begin this process, you need to consider the impact. If you are upgrading a live application, you have to consider whether or not any of the following are acceptable: downtime, duplicate log entries, and missing log entries. All can be avoided if you have time to prepare and are willing to put in some extra time and effort – and the majority of the risk falls into how you have your indices labeled! Click here for more explanation and strategies for upgrading active databases.
The guide below assumes that some log duplication is acceptable. Let’s begin.
Create ElasticSearch 5.0 Database
Create a new ElasticSearch 5.0 database via the Aptible dashboard or via the toolbelt with the following command:
aptible db:create [ES 5.0 Handle] --type=elasticsearch --size=[Size] --environment=[Environment]
- [ES 5.0 Handle] - The handle of the ElasticSearch 5.0 database.
- [Size] - The size of the database in gigabytes in 10 GB increments.
- [Environment] - The environment the database should exist in. This parameter is optional if you only have one environment or have defaults set up.
The last line returned from the above db:create call will return your database connection string. Save the connection string (URL) as we’ll use it at the very last step in this guide for upgrading Kibana. We’ll reference it later as [ES 5.0 URL].
Create Log Drain
Now that the ElasticSearch 5.0 database is created, we’ll want to submit all new logs into it. To do this, you’ll need to log into your Aptible Dashboard.
- Log in to your Aptible Dashboard
- Select the environment you created your ElasticSearch 5.0 database in
- Select the “Logging” tab from the middle of the screenn
- Click the “Create Log Drain” button at the bottom of the screen
- Click the “Self-Hosted Aptible Elasticsearch” button
- Enter in your new database’s handle, [ES 5.0 Handle]
- Select your database from the drop down at the bottom of the popup
- Click the “Save Log Drain” button
Once this log drain finishes setting up, your logs will be directed to your new database. If you haven’t deleted your previous log drain that submits logs to your ElasticSearch 2.2 database, you will begin tracking the logs in both ElasticSearch databases. It is recommended to wait to delete your old log drain until your migration is successful. This will lead to some log duplication, but will make it very easy for you to start over if needed. If your application(s) are not active at the time of the migration, your duplication will be minimal and mostly system logs.
Create Tunnel to Both ElasticSearch Databases
In separate processes, tunnel into both of the ElasticSearch 2.2 and 5.0 databases.
aptible db:tunnel [ES 2.2 Handle]
aptible db:tunnel [ES 5.0 Handle]
Each tunnel will return an output like the one below. Save the URL (https://…) for the next steps.
Creating elasticsearch tunnel to [ES 2.2 Handle]...
Connect at https://aptible:09lWzWjX2sK@localhost.aptible.in:64551/
Or, use the following arguments:
* Host: localhost.aptible.in
* Port: 64551
* Username: aptible
* Password: 09lWzWjX2sK
Transfer Indices
Using the URL printed out from the previous tunnel commands, get the indices of your ElasticSearch 2.2 database. These indices are what is copied over to your new ElasticSearch 5.0 database.
curl [ES 2.2 Tunnel URL]/_cat/indices?h=i
This will print out on a new line each index you have on your ElasticSearch 2.2 database, like the following:
logstash-2017.01.26
logstash-2017.01.25
logstash-2017.01.24
.kibana
You will not need the .kibana index unless the Kibana property index.mapper.dynamic (default: true) was changed to false. When we change the configuration for the Kibana app container and Kibana 5.0 starts, it will create it’s own necessary indices.
For each index in the ElasticSearch 2.2 database print out from the last command, we need to execute the following commands:
elasticdump --input=[ES 2.2 Tunnel URL]/[Index] --output=[ES 5.0 Tunnel URL]/[Index] --type=analyzer
elasticdump --input=[ES 2.2 Tunnel URL]/[Index] --output=[ES 5.0 Tunnel URL]/[Index] --type=mapping
elasticdump --input=[ES 2.2 Tunnel URL]/[Index] --output=[ES 5.0 Tunnel URL]/[Index] --type=data
Input/Output arguments should look like this:
--input=https://aptible:09lWzWjX2sK@localhost.aptible.in:64551/logstash-2017.01.26
Depending on how many indices you have and how large they are, this can take awhile. It’s recommended that if you are transferring many gigabytes to use the ElasticDump with the scroll API to reduce risk, though not covered by this guide.
At this time, your migration from ElasticSearch 2.2 to ElasticSearch 5.0 is completed! Next up is upgrading Kibana to 5.0.
Upgrade Kibana
To upgrade Kibana, you’ll just need to pull the latest from Aptible’s docker-kibana repo and update the app’s config. Unless you’ve customized your fork, the pull should work, flawlessly.
git pull https://github.com/aptible/docker-kibana.git master
Then push the merge back up to your app’s repository on Aptible. Once the deploy finishes, you’ll need to set the config variables to use your new ElasticSearch 5.0 database.
aptible config:set KIBANA_ACTIVE_VERSION=5 DATABASE_URL=[ES 5.0 URL]
If you don’t have the [ES 5.0 URL] handy, you can look it up in your Aptible Dashboard. Once the above command finishes, you’ll need to log into your Kibana instance. If prompted to set an indexing column, choose the one you used prior to upgrading. If you don’t remember it, choose the most appropriate one for yourself moving forward.
You’re all set, upgraded, and ready to go!
If you haven’t deleted your old log drain for ElasticSearch 2.2, you are still submitting logs into both ElasticSearch databases. If you no longer need the ElasticSearch 2.2 database collecting logs, now would be a great time to delete that old log drain, and if desired, deprovision the ElasticSearch 2.2 database. Even if you deprovision a database, Aptible keeps the backups for their normal retention period. By deprovisioning, you will need to submit a support ticket if you need to restore from a backup as the dashboard will no longer have it available for you to do it yourself.
Kibana 5 Links
Kibana 5 Introduction on YouTube.
Kibana 5 Getting Started Guide.
Active Database Upgrade Strategies
As mentioned earlier in the guide, when and how you upgrade can have an impact on your active ElasticSearch databases. The strategies and outcomes are below.
Minimal Log Duplication
This guide was developed for this strategy. By having two log drains active at the same time, one for ElasticSearch 2.2 and another for ElasticSearch 5.0, both databases are receiving the log entries. By performing the migration during your dead time or off-peak hours, the only log entries you should receive are those coming from your applications that are not generated by your users.
No Log Duplication, But Log Loss
If duplication is a problem but minor loss is not, especially those entries generated by off-peak traffic, you should delete the old log drain prior to creating the new log drain. This will most likely result in loss of log entries, but you will not have any duplication. Please beware, if your indices are large, they will take hours to migrate over and you will not be collecting logs during the entire migration process.
Indices by Dates and Times with No Duplication or Loss
If your indices are by date, such as: logstash-2017.01.24, you are in luck! You can easily migrate over all of your indices that reference dates prior to you performing the migration. Once you are done with the migration, it’s a matter of cleaning up the loose ends. The process we used with great success was:
- Keep the old and new log drains active during and after the initial migration. The duplication will be removed.
- Migrate over all indices for dates in the past from ElasticSearch 2.2 to ElasticSearch 5.0, not to include the index labeled for the day you’re doing the initial migration.
- The next day, delete the index with the date of the initial migration on the ElasticSearch 5.0 database.
- After deleting the index the next day, migrate over the last index from ElasticSearch 2.2 – the day of the initial migration.
- Delete the old log drain and deprovision the ElasticSearch 2.2 database if desired.