announcements

Annie Lowney

How to Secure Event Sponsorship

Holger Blank of JSConf EU shares his sponsorship tips

Holger Blank, pictured below, is the sponsorship maestro for JSConf EU, a not for profit JavaScript conference in Berlin. JSConf EU started back in 2009. Their top sponsorship slots bring in €14K. Since the inception of the event they have been in the enviable position of having sponsors approach them… something that most event organisers can only dream about.


holgerblank-2016.png

JSConf EU is a desirable event for sponsors because it brings together a group of people that have a skillset that is in demand. But don’t let their niche appeal discourage you from figuring out how to secure event sponsorship. In this article, Holger provides an interesting take on the role of sponsors. We have also created a sponsorship proposal calculator that you can use. The link to it is at the end of the article.

 COMPARE YOUR EVENT TO OTHERS

JsConf2017_Day1_JuliaLukaLilaNitzschke_Zoooi-029.jpg


When you compare your life to your Facebook friends' you’ll probably cry yourself to sleep. That’s unless you have a Facebook friend that prattles on about how healthy they are… You see their photo of kale and raise it a photo of vanilla ice-cream with a kale garnish, but I digress.

If you’re in the first year of launching your event, research established events that are similar to yours. This is a strategy that Holger employed:

“The JSConf conference in the United States was quite successful, so as a result sponsors were keen to get behind the first European JavaScript conference due to the connections that JSConf made. As a result we try to reciprocate the help we got when we first started out by sharing our connections with similar first time events.”

This approach allows you to do two things:

  1. Mirror the type of sponsors your peers in the industry have.
  2. Present the potential growth of your event, therefore you make it an appealing prospect.

A SPONSOR SHOULD INTEGRATE SEAMLESSLY INTO THE EVENT

JsConf2017_Day1_JuliaLukaLilaNitzschke_Zoooi-104.jpg


Every event organiser has different sponsorship marketing thresholds. Some tolerate a pull-up banner with a low growl, and others go apocalyptic at the mere suggestion that they should hand over their attendees details to a sponsor. You need to know how far you’re willing to go to secure sponsorship. As with any event, the attendee experience should be foremost in your mind.

For Holger, sponsors know their exact boundaries for JSConf EU:

We aim not to sell any speaking slots at our conference, at least not for products. We aren’t big fans of product companies pitching their product. Of course, we have some exceptions to that rule. From time to time a huge international tech company will ask us to let them on stage to promote their latest technology that has been made open source, but usually it's a good thing if a huge international tech company wants to talk because they have to say something that will matter to our attendees.”

You can’t blame a sponsor for suggesting ideas if you don’t set out clearly your expectations from the start, as Holger explains, “we try to play everything really open with our sponsors. We share our point of view of what is really meaningful for our attendees and what is not. And JSConf is a community driven conference, so it's quite expensive, but it's expensive because we want to deliver great quality for our attendees like the best food you can have at a conference, the best atmosphere and setting."

This great article by Jonathan Grupp details his attendee experience of JSConf EU 2017.

"We believe it's better to stay independent in how we curate our program, that includes how we invite speakers and how we sell tickets. We want to do it our way. We don't want to be disturbed by big companies who will try to force us to do things they want to do and we don't want to do.”

On the flip side sponsors need to feel like they are getting an ROI. Sponsorship of events is notoriously hard to track. Sponsor managers need to justify giving away a chunk of their budget to their bosses, it’s simply Giving Money Away: Expectations 101. Holger’s approach is interesting,

“We sell them the fact that sponsoring makes the conference awesome. So be part of the awesomeness. Ticket costs will cover the basics, and every penny they spend on our conference will make it even more awesome. We’re selling visibility to them. We prefer a non-invasive visibility. We try to sell them ideas like, build a networking area. Don’t have big booths, don’t have too much signage, just integrate yourself and get in touch with the community. Don’t be aggressive!”

It’s an interesting take on sponsorship. Most sponsors are sidelined to the fringes of an event like penned in animals, some get a pet and others just yearn for some attention like this guy.

Key takeaways:

JsConf2017_Day1_JuliaLukaLilaNitzschke_Zoooi-116.jpg
  • Before you even reach out to a sponsor, establish what you are willing to trade in exchange for sponsorship… don’t be rash; really think it through. What is best for your attendees? They're your long-term investment.

  • Encourage your sponsors to think of new ways to engage with attendees that don’t involve standing passively at a booth. For example, they could create a networking space. Help them brainstorm something that will add value to the event.

  • A sponsorship manager, most of the time, will need some type of return on investment. Sponsorship is very difficult to measure in definites. It shouldn’t be seen as a marketing activity that will yield that. It is an awareness piece. The measurement that could be employed is for the sponsor to allow for a 1-2 month lag time from the date of the event to see if there was a spike in traffic to their site and/or sales growth.

CREATING YOUR PITCH

JsConf2017_Day1_JuliaLukaLilaNitzschke_Zoooi-118.jpg

Crafting a sponsorship pitch is a skill in and of itself. A trap a lot of people fall into is mimicking the tone of pitches that they see online. As our teacher used to say, “there is only one of you in the world because you’re very special, Tito.” Figure out what your event's voice is. Is it serious but approachable? Knowledgeable? Humourous? Only then can you start to craft your email pitch.

If you can meet the sponsorship manager in person or if you can call them, do. If you’re calling them cold don’t launch into your pitch, figure out whether they're hesitant or warn, and forewarn them that you plan to send them a sponsorship pitch email/deck and ask if it’s okay to do so. Don’t underestimate the art of conversation; it adds some humanity to the relationship.

Maybe get creative and surprise them with a basket of apples if your conference is inspired by Apple like our co-founder's conference. It will leave them intrigued and curious. It’s a great way to start a conversation.

Key takeaways:

JsConf2017_Day1_JuliaLukaLilaNitzschke_Zoooi-042.jpg
  • Keep the email succinct or get in touch via Twitter.

  • Highlight the important stuff:
    • Your event’s mission.
    • The type of audience you plan on having. 
    • Attendance numbers, noteworthy speakers, and past sponsors.
    • Examples of social media reach.
    • Images of the event into the email to help them visualise.
  • Money money money money, money! It’s always tricky to broach the topic of money. Do you say it in the pitch email or do you wait till your pitch email get’s a bite and then send them the pitch deck? There're pros and cons to both and it goes back to your style. Just be flexible with sponsorship amounts, especially if you are a new event. It’s great to have a price list but you don’t want it to be a barrier to involvement either. Emphasize that you’re open to discussing what they can afford.

  • Try to speak in person or on the phone to the sponsorship manager before you email or tweet them.

  • Maybe send them something creative that is tied to your event as a means to starting a conversation.

  • Use the sponsorship calculator to estimate how much money you need to secure in order for your event to run.

IN CLOSING

JsConf2017_Day1_JuliaLukaLilaNitzschke_Zoooi-156.jpg

Don’t be intimidated by securing sponsorship. If your event is a good fit for that company, it will happen. Don’t do a scatter gun approach. Focus in on the companies that you know are a good fit for your event. Don’t sacrifice the integrity of your event for cash. Be very clear from the outset about what type of relationship you want the sponsor to have with your event. Brainstorm with the sponsor and come up with some unique but valuable activities for both stakeholders.

Try to establish a personable relationship with the sponsor before you pitch them. Don’t underestimate the power of a conversation. You both have goals, it’s just up to you to show the sponsorship manager that you can and want to help them achieve their goals.

how-to-process-your-event-payments
Maria Keenan

Marketing Has to Get Better

A Bit of Background

I was a telemarketer. I used to call Americans to ask them if they wanted white papers from tech hardware companies, and if they’d be willing to answer surveys about their businesses in exchange for them.

If they agreed, my job was to get as much information out of them as I could. Then at the end of the call I'd tell them a sales rep from the hardware company would be in touch and hang up before they could object. Gross, yes, but I was just doing what I was told to do.

It was a pretty ineffective strategy, hence why I used to call 300 people a day. There was a 90% staff turnover in that company every year because the targets were becoming harder and harder and the tactics less and less effective.

Innevitably, the successful calls were with those who were bored or lonely. They spent 2 or 3 minutes after the business bit of the call asking about my day and talking about the somehow ever-present family connection that every American individual has with Ireland.

After a while, these conversations stopped making me sad, which I think was the hardest part. At that point, I became used to exploiting another human's need for compansionship and I knew I couldn't continue.

I left to find something else. 

There Had to Be Something Better

As my next step, I fancied single-handedly cutting the phone lines where I used to work. But, they used a VoIP. So I couldn't.

Instead, I channelled my energy into finding something better than cold calling. Marketing has to get better than that. I believed in something better than that. So, I looked around and found HubSpot.

If you came here for an exposé about HubSpot, I’m sorry to disappoint you. I don’t like gossip mongers, and I’m not that tacky.

They do inbound marketing. Inbound as a strategy had a lot of what I wanted: being able to help people, putting use to skilful writing, and being able to prove that there’s integrity in what you’re doing while still being able to measure its effectiveness.

If you want to get visual, here’s what the inbound marketing methodology looks like:

HubSpot-Inbound-Methodology.png

But then I left. I left because, even though I bought the methodology of inbound, I couldn’t practice much of it at HubSpot in sales other than telling Finnish people (whose surnames were really hard to pronounce) to buy software based on a great idea. So, again, I left to make the marketing world a better place. 

I had learned a lot. I was a little older, wiser and ready to put the best parts of that experience to use.

How I’m Trying to Make Marketing Better This Time

Marketing is hard. Attracting people to your website by writing about their struggles seems innocent, but some people have taken that idea and made a numbers game out of it. They’ve left behind the principal that made the concept so good in the first place: being helpful.

For example, people have started manipulating inbound marketing by picking popular online search terms, stealing other people’s ideas that they read on blogs elsewhere, and making a listicle of those ideas so that they’re seen as the expert despite not doing much work at all. Sure, it works in the short term, but there isn't much value in that.

Those people are as helpful as I was when I was calling people up and promising them the world. The only difference is that people gaming the marketing sector don’t use a phone.

I don’t want to be the person who shouts the loudest in a crowded room. I want to be the person that says the one sentence you remember to tell a friend when you leave.

So, here I am at Tito. I’m currently sitting in a room that’s so quiet I can hear my keyboard clacks reverberate off the walls, and yet we’ve helped to sell over €200 million worth of tickets.

From here, I want our marketing to focus on two things: 1) good writing and 2) being good to event organizers.

I’m committed to publishing articles based on how insightful they are, not how many random clicks they’ll get. That said, clicks are important in their own way, which is where 2) comes in. You’ll notice some of our blogs have buttons at the bottom of them now. As part of our strategy, we have some gated content; a fancy way of saying that we'll ask for your email address before giving you access to certain files. 

To explain, I’m going to use how well that button does to decide how helpful we’re being. If more people are willing to trust me to send them specific, long-form information, then that information must be resonating with them. But, if they don’t interact with the information when it gets in their inboxes, what’s the point in sending them more things? We’ll get ignored and they’ll get annoyed. Simple.

I hope this hints at the fact that we’re only at the beginning. I’m full of ideas and can’t wait to see our readers as excited as I am. You’ll see the first iterations very, very soon. If you want, you can even see one by clicking the button I mentioned below. I personally look forward to seeing what you think, but I certainly hope you enjoy.

how-to-process-your-event-payments

Paul Campbell

Welcoming Maria to Team Tito

“And we’ve done it all without any marketing”, is a phrase that I commonly reach for after talking about Tito numbers, growth or sales figures. It’s a bit disingenuous though. Without marketing, we wouldn't have a website, no blog, and we’d never have gone to any conferences to talk to potential customers.

It’s more true to say that all the marketing that we have done is without anyone working on it full time. This month that has changed, and we’re delighted to welcome Maria Keenan to the team full time to help us build a marketing plan that represents who we are, and develop tools to help us make better decisions based on what our customers want from us. Be sure to check out her first post.

I believe that the core of great marketing is a great product. But this goes both ways. Marketing that is honest and full of integrity is as much part of the product as the other benefits and values that a product provides. The message around a product, the voice, the culture and the sense of belonging that customers feel are all intrinsic to the experience of the product.

These are things that we’ve hired Maria to help us build on. Maria is the fifth full-time member of the team. It’s been deliberate that we’ve brought someone in early to help us with language, tone, writing, spirit and ultimately: feeling.

As we have signed up more and more interesting customers, we’ve become increasingly aware that there is well of knowledge ripe to be shared. One of Maria’s first projects has been talking to customers and building out a series of articles and posts and tools that we think will be useful to Tito customers and beyond.

Welcome Maria.

Maria Keenan

Why We Care About Sponsorship

It’s with pleasure that I tell you what our first theme for the revamped Tito blog is going to be: Sponsorship.

Why We Chose Sponsorship

We surveyed some of our customers, used some third-party analysis of the events sector, and swore aggressively at Excel until we came through with something concrete.

Sponsorship turned out to be something that was well-documented online, but in industries that were irrelevant to Tito customers. There is a thousand articles about Formula One sponsorship deals for every one about sponsorship for tech events. Given that, we chose to collect as much good stuff as we could to help folks who run tech events to find what they need.

Why Sponsorship is Something to Care About

I Googled “why you should care about sponsorship” and got an article on why you should care about persecuted Christians. That’s not very helpful for the task at hand.

Instead, here’s what we found throughout writing about this blog topic. They answer the question more accurately:

  1. Partnering with an interesting, relevant brand makes your event visible to their market. Chances are that partner is already present and well-known there. You probably want to work with their potential customers as they’re also good fits for your event. And vice-versa for the sponsor. Win-win.
  2. Sponsorship has long since gone beyond sticking a logo on a brochure. Creativity and sponsorship are now symbiotic, and creative events specifically have begun reinventing how they integrate their sponsor. These events now aim to create engaging spectacles at their events that delight their attendees as well as their sponsors. We’ll give you some examples in a future post.
  3. The most successful sponsorship partnerships are those that last beyond the first year of the event. Because of this, modern sponsorship is a source of recurring revenue that can help event organizers to expand the future-proofed cash flow they have for planning the next iteration of their event.

If you want to dive in straight away, the button below leads to a Sponsorship Proposal Calculator to tackle the ever-present struggle of figuring out how much you should ask your sponsors for based on the unique circumstances at your event.

We’re politely asking for your email address to download it, because if you’re interested in this, we think you’ll be interested in some other great stuff we’ve been working on. If you’re not ready just yet, there are more than enough pieces on the way to help you reap the above benefits. Stay tuned.

how-to-process-your-event-payments

 

Doc Parsons

Hello.

Don’t compare your backstage to other people’s onstage.

— Patrick Rothfuss

We’ll get into the awkward subject of why there have been no blog posts in the last ten months but first, an aside... Podcasts.

For various reasons, 2016 was the year I went from listening to a lot of music to listening to a lot of podcasts. I’m late to the game, I know, but for some reason as last year’s news got worse—Brexit, Trump, losing a loved one—I began to find great comfort and distraction in listening to other people’s voices in my head. Not necessarily content that tried to make sense of the madness happening in the world, but subjects that helped to ground me, like tech and productivity. It was an audible hug for my brain.

Along my podcast journey, I was reacquainted with the voice of Merlin Mann who spoke at dConstruct in 2010; my first ever web conference. He and John Gruber recorded a rather special edition of The Talk Show—which normally covers commentary on the Apple ecosystem—but for this episode mainly skirted around the US election result, having been recorded the day after. I thought it was a nice light through the fog and I began digging into some of Merlin’s other shows.

Back to Work popped up on my radar which is a show Merlin co-hosts with Dan Benjamin where they discuss productivity, communication, work, barriers, constraints, tools, and more. I took particular interest in an episode where they were discussing this notion of backstage versus onstage.

299: Unhook from the Comparison — Back to Work — Overcast

It got me thinking a lot.

tl;dr — the things you generally see on the web, on Instagram, on TV—these can all be seen as someone else’s onstage. It’s the public version. The polished version. And as content creators and makers of things it’s very hard not to be affected by the success of other people’s work. Or by the latest and greatest technique, or method, or framework. Sometimes these things can have a positive effect, but sometimes they can simply make you feel like crap. Like you’re forever trying to catch up. It’s a loud world, and only getting louder.

And then I realised that we at Tito are the kind of individuals—and by association, the kind of company—that try not to get distracted too much by outside influences. We keep our heads down and focus on the things we have to do. “Fuck all that noise” is a statement you’ll hear occasionally echoing through the office.

As with everything, this approach has it advantages but also comes with some drawbacks. We’re not natural showpeople. Marketing ourselves and talking about what we’re up to isn’t something that comes very naturally to us.

We’ve done a lot in the last 10 months, but our default behaviour isn’t to shout about it. Instead, we just hunker down and move on to the next thing. And, when you’re a small team, there’s always a big pile of next things.

Is that a good approach for increasing engagement? Nope.
Is this helping us to scale beyond our wildest dreams? Nope.

We’re perhaps a little too humble for our own good, but we’d prefer that to the alternative. Because the alternative can come across as brash and abrasive. Or maybe it’s ego.

I have a feeling that a lot of our customers prefer our lack of jazz hands. What they get from Tito isn’t a lot of heavy promotion and attention grabbing headlines. They get an honest product with a team dedicated to improving their experience.

With all that said, there needs to be a balance. If you came to our blog looking for a pulse, you’d be forgiven for thinking that all was not well. In fact, quite the opposite is true. We’ve been very busy. We’ve even hired a few great folks to help us improve our *onstage*.

We’re also hiring a few folks to help us improve our bus number.

Not to mention the fact that we haven’t been idle during the last year, and we have plenty of exciting updates that we’ll share soon.

So hello, we’re back. Not that we ever left, but you’ll be hearing a lot more from us from now on.

Karl Mc Carthy

Renting Conference Venues and the Mistakes I Made

“As the banquet manager began rearranging our carefully laid out AV equipment 30 minutes before the doors opened I knew we had made a major mistake in picking the venue.”

I found myself in the situation above not so long ago and it was certainly not fun. Watching our plans and run of show ripped apart broke the heart. It was one incident of many in what was a fraught relationship with the venue. The warning signs were there and a combination of enthusiasm and limited choice landed us with a venue we probably should have passed on. So to avoid such a traumatizing experience here are a few tips I have picked up over the years to help find you that perfect spot.

Big Corporations = Big Paperwork

The venue may be highly reputable but keep an eye out for a truckload of rules and regulations. I have dealt with some major venues and the larger they are the more paperwork there is. An example of this is when I worked with a large hotel chain and by the end of it we needed a team of legals to sift through all the paperwork. Sometimes the smaller lesser known venue is the way to go, especially if you are just starting out.

Remember, you’re the customer!

Unfortunately, with the increase of events, conference space is scarce on the ground and at times it feels like you are the vendor and not the customer. Try and keep the venue working for you and make sure to have plenty of backup options and let the venue know you have them. In the past, I have arrived with printed quotes from competitor venues and I am never shy to show them. If the venue has a bar and you expect your attendees to make use of it then you should ask for a discount. A quick calculation can tell you how much they will make and in most cases a percentage can be deducted from the rental fee. At a recent event, I presented bar revenue stats from previous events and managed to have the entire venue rental fee written off so having a track record certainly helps in this respect.

Hidden Costs

Ever got stuck with €3,500 electricity bill after an event? I did, it wasn’t fun. By failing to read some fine print I ended up on the hook for the cost of lighting the floodlights in the very large venue car park all week. Lesson learned in a big way. Be aware of what is and what is not included in your rental fee, who is liable for security costs, how many bar staff are available, does the venue fee cover cleaning, who does the cleaning, can I hire my own or do I have to use venues, the list goes on and on and on.

Exit Clause

I have spent many an hour waiting for the staging to be cleared out of a venue, a lot of venues have strict penalty clauses so make sure you are on-site and not in the bar toasting your events success. It can be a real anti-climax packing everything up but it’s better than paying for an extra day rental. Make sure your third-party vendors are aware of your timelines and insert clauses where required to cover your liability.

AV OK?

Acoustics are very important and one of the first things on your checklist, there are few things worse than not being able to hear the speaker and it’s a surefire way of losing the room quickly. When I look at a venue I always ring round to the local AV firm as they are normally the best source of information and well worth a call as they have probably kitted out the room in the past. If the venue is providing the equipment double check its quality and add to it if required.

Think Big, Plan Small

We all think we will sell out, unfortunately, according to recent stats less than a quarter of events come close to full capacity. It is a lot better to have a sold out event rather than a half empty venue, all you have to do is look at the most recent Olympics to get the picture. Ambience is an important factor for production so try and err on the side of caution.

Karl Mc Carthy

Sponsorship Sales Sluggish? Seven Suggestions for Success

At Tito I speak with conference organisers every day and one of the main talking points is sponsorship sales. Be it a well established tech event or a one off charity function, get a big sponsorship client on-board early on and you can ride the momentum all the way till opening day. Get it wrong and you’ll be flying home cattle class with nothing but shattered dreams and a suitcase full of unused lanyards. With that in mind here are a few tips I have picked up over the years that will help you land that perfect patron.

1. No más to PDFs

For my mind placing a sponsorship PDF on your site does not set the pulse racing nor engage the prospect. It feels impersonal and gives me the vibe you are not willing to go the extra mile. Same goes for embedded forms where the user must fill out a form like a regular Joe. Your potential sponsor is a VIP. Treat them as such by publishing the name and email of your sponsorship lead, a good example of this can be found here from the team behind Voxburner and the successful YMS conference series.

2. Get me a real doctor

Sponsorship sales is a job for the grown ups. Handing over prospects to junior staff will lead to nothing more than a burned lead list. I have found there is a certain level of experience required to speak with C-level type prospects so organisers should be hitting the phones and stalking LinkedIn profiles. Only professional sadists enjoy making cold calls but unfortunately it is part and parcel of running a successful event. If you are expecting the sponsor to open the cheque book then the organisers should be directly involved. Embrace the challenge, it is only going to make you a stronger person, plus you’ll finally appreciate Glengarry Glen Ross.

3. You can have any color, as long as it’s black

Perusing the sponsorship landing page should be a stress free experience so try and keep the sponsorship options nice and simple. I get confused when I see lists of package levels from platinum level, gold level all the way down to pig iron level. By curating your options the sponsor can focus on the core offerings of your conference. Ideally you will have a tonne of secret add-ons that personalize the package and make the sponsor feel like a million bucks.

4. Logo on your website, no dice

Following on from the above, it is time to get creative with how the sponsors interact with your attendees. Banging a logo on your event website is great but for me I am interested in something more. Think how your sponsor can engage with the attendee, preferably by getting their product/service in a real use case. It conveys a sense of legitimacy and character for your sponsors rather than the in your face hard sell some organisers pitch sponsors at. Heroku pulled this off with great success very effectively by gifting credit to attendees to add to their accounts, whether they were existing customers or not.

5. When it comes to data analysis, I Excel

For organisers with a couple events under their belt a bank of attendee data is there to be analysed. There is no need to be overly clever, I will always be interested in the standard metrics such as demographics, social media footprint and potential reach. Testimonials of previous sponsors alongside ROI stats goes a long way to sealing the deal. The organisers at NEXT Conference layout wonderful visuals which sum up their data and who their attendees are.

6. It’s cool to be in-kind

Sponsorship is not always all about a cash injection or giveaway. Your typical conference involves an army of third party vendors and there may be a cross over of target markets well worth exploring. With Tito I am always open to discussing sponsorship opportunities and find it a cost efficient marketing play. It also means I can support fantastic community led conferences that need a helping hand, same too goes for charities.

7. Noah’s Arcade presents Wayne’s World

Quality not quantity is the mantra here. Although it may hurt your pockets in the short term it will serve you well in the long run. Your sponsors have to share common interests with the attendees otherwise they both lose out. Common sense yes but a mistake that is made time and again by organisers.

Karl McCarthy - Tito, Head of Expansion

Paul Campbell

The Tito Profile Series, Volume 1, Interview 1: Marc Thiele

As our customer base grows, it is becoming very clear to us that our customers have a lot to share with each other (and everyone else) about their approach to building events, business and improving their communities.

To celebrate this and to share the fruits of any insight from these interesting folks, we’re running a short interview series of some interesting Tito customers. We have commissioned an initial run of 6 interviews by Aoife Reaper-Reynolds, and we’ll be publishing them here over the next few weeks.

The first interview is with Marc Thiele, who became a very early Tito customer a few years ago. Marc’s conference “beyond tellerrand”, has gone from strength to strength not only to achieve a solid reputation in the design community, but has become a successful business in its own right for Marc.

We’re excited about launching this profile series, and we hope that you enjoy it and find the profiles valuable.

Read Marc’s interview here

Paul Campbell

Tito  WWDC

If you’re in and around the SF area during the week of WWDC this year, be sure to check out some of the awesome fringe events. Tickets to AltConf and Layers are still available.

In Apple’s words:

In addition to WWDC, a variety of other exciting developer events will take place throughout the week in San Francisco. Follow the links below to find even more opportunities for learning, fun, and networking.

It’s our particular delight that 4/5 of those links eventually lead to Tito. <3 Thanks to Rob, Jessie, Jim and John for choosing Tito!

https://developer.apple.com/wwdc/attending/

Paul Campbell

Downtime Incident Report, March 25 2016

This morning at 5am, Tito became unavailable for all customers. We were made aware of this at around 8am, investigated, pushed an update and resolved the issue by 8:45am.

As a service that has enjoyed 100% uptime in the last six months (and 99.99% uptime in the last year), this was very frustrating. We have had outages in the past that were out of our control, where our downtime was caused by a third-party service (along with many others), but this time it was due to a configuration issue on our end.

The background

In 2014, one of our caching (redis) servers went offline for maintenance and we didn’t have auto-failover. Since then, we implemented a master-slave redis setup with auto-failover via Amazon’s Elasticache service.

What happened?

For both their redis and memcached offerings on Elasticache, Amazon offer a “master” DNS endpoint that gets updated automatically if there is a failover event. Tito is set up to use that master or “Configuration endpoint” for memcached, but the redis setup looks slightly different. When it was set up originally, our assumption was that the write master endpoint would stay the same in the event of a failover. Unfortunately, we failed to realise at the time that Elasticache provides a “Primary Endpoint”, and pointed Tito’s redis configuration to the write master.

At 5am this morning, Elasticache initiated a failover event that resulted in the write master being replaced by the read slave, the slave being promoted to write master, and the old master being replaced with a new slave. Since Tito was pointing to the old master, which was now a read-only slave, any attempts to write failed, causing the downtime.

Updating Tito to use the “Primary endpoint” for redis solved the issue.

What steps are we taking?

There were two main issues here.

1) The configuration issue itself 2) The three hour response time

For the issue itself, Tito is now pointing to the correct endpoint, and should a failover event occur in the future, there would be minimal, if any, downtime.

For the second issue, we were lucky in a way that this happened at 5am, which is the quietest time of day for us, and usually the time we run any database-related patches or restarts.

As Tito grows, it will become more manageable for us to detect and address issues like these to the point that they go mostly unnoticed. In this case, 3 hours is disappointing, but hopefully acceptable given our six month track record.

We’ve worked hard to ensure that Tito is a solid platform and that when things do go wrong, that we have planned for it and configured for it. Unfortunately this time, our configuration let us down.

If you have any questions about any of this, please feel free to send them to support@tito.io.