I’m pretty sure now that the root cause for my recent Atlassian clusterfail, which support had us work around by replacing hostnames with IP addresses in both setenv.sh and cluster.properties, was that the hostnames were too long.

Not the fully-qualified domain name, just the hostname component. We have a fairly verbose naming convention, where you can search for things like “jira-appserver” and see all environments, or “sandbox-jira-app” and see just the ones in a single cluster (both of these are useful for clustershell incantations). Since we’re doing an A/B cutover for this upgrade, the new sandbox machines also got the application version added to their hostnames, pushing them to 31 characters.

The official limit is 63, so either someone wasn’t reading the standard, or they’re internally using a double-byte encoding like UTF-16, possibly for Windows compatibility reasons.

(Komi-san wants a second opinion)

The annoying symptom for this failure is that a bunch of the core add-ons simply refuse to load, taking a full 5 minutes to time out during startup. This is also the way it fails if you leave out the allegedly-optional ehcache.* options in cluster.properties, even when upgrading the first node.

By the way, when upgrading a Jira cluster, the node that comes up first (which does the heavy lifting for the upgrade) must be active in the load-balancer that answers to the official base URL, because when the other nodes come up, that’s the hostname they’ll contact to ask about the upgrade status, even though the cluster members are stored in the database so they know how to reach it directly. If you don’t want users to try to log in during the upgrade, you have to tinker with your load-balancer config so that only the other nodes get directed to the upgraded one, while everyone else still gets a “go away” page.

(pictures are related, in the sense of “every time I shoot down one problem with this upgrade, three more appear out of nowhere”)

Comments via Isso

Markdown formatting and simple HTML accepted.

Sometimes you have to double-click to enter text in the form (interaction between Isso and Bootstrap?). Tab is more reliable.