Look at them cleverly placing themselves next to prominent systems like the air traffic controller, while toting their millions of lines like it's some sort of accomplishment. For a business that really only needs to accomplish two things to make their customers happy: 1) print tickets and 2) stop scalpers, they do absolutely terrible at those 2 things. I don't mean terrible as in they do the job poorly, it's just that they care 0% about the user experience during the entire ticketing process from end-to-end.
Ticketmaster is essentially enterprise software-as-a-service: the people buying it (the venues) are not the people using it. From that perspective, complaints are not surprising at all.
Except a large part of the Ticketmaster system is back of the office stuff that the consumer never uses. It isn't only the system used for the end user to purchase tickets, but also the system that the venue uses for things like inventory and access management. Odds are if you ask people working at the venue, they will have just as many complaints about Ticketmaster as you do. The problem is that none of their big competitors (that I have seen at least) have really put together a product as complete as one that Ticketmaster can offer a venue.
Possibly a reference to a supposed ancient Chinese saying "May you live in interesting times". "Interesting" here means dangerous and troubling in contrast to "uninteresting" or peaceful and stable.
They used to have their own operating systen or something according to an article I read about Terry Davis though that time is probably long gone. Edit: found the article: http://motherboard.vice.com/read/gods-lonely-programmer
So Denny Denker, the Ticketmaster chief architect, wrote a VAX Assembly language realtime ticketing operating system. He wrote it in the 70s, before things like TCP/IP were standardized, and they managed to virtualize it on Linux using a hacked up vax emulator[1]. They've tried many many times (when I was there around 2007ish) to rewrite it using normal things like C++ and mysql, yet couldn't manage to get it faster than a specialized realtime operating system. It was kind of hilarious when we moved it to NFS, as the operating system timed writes to the speed of the PDP-11 drive spin rate. When we moved it to NFS, the IO was 1000x faster, and the entire thing crashed. No end of jokes resulted from these sorts of "quirks" in our systems.
Last I've heard, they've been slowly moving functionality out of the VAX operating system into a more traditional stack.
Source: I used to work on the team responsible for the Linux servers that ran these virtualized vaxen, check the resume linked in my profile.
Once I have bought tickets, traveled 500 kilometers, stayed at hotel near venue. Once I got there, there was dozens of people confused about place being shut.
It turned out that the gig as cancelled and Ticketmaster forgot to notify buyers about it.
This was for one of the well known US bands, so there was a lot of people there.
The reason of cancellation was sickness of one of the members of the band that was announced on other site also selling tickets a week earlier so only Ticketmaster customers were affected.
Imagine disappointment. Also, I didn't get refund for tickets either despite requesting it.
Actually, to make their customers happy, they need to provide high quality inventory and revenue management tools that tightly integrate with third party databases and systems, and allow their users to make strong business and pricing decisions based on their data.
Ticketing systems are products sold to a presenter or promoter (or venue) and facilitate their business.
> For a business that really only needs to accomplish two things to make their customers happy: 1) print tickets and 2) stop scalpers, they do absolutely terrible at those 2 things.
Their job isn't really to do either of those things, and I'm not being glib. Let me explain:
First, TicketMaster (TM) doesn't really want to print your tickets. They would much rather just digitally manage them, and let you use your phone to go the the venue. Non-digital media costs real money, and TM would prefer to charge you for the convenience of managing your tickets, or sending you a PDF. As a bonus to them, this lets you easily resell your tickets as a fan on their TM+ exchange.
Secondly, TM doesn't really want to stop scalpers, at least not completely. This one is a bit complex, because portions of TM are definitely not scalper (i.e. broker) friendly, while other portions are. TM runs their own secondary market, called TM+, which is a "fan resale" market, in which they list ticket that were sold on the primary market (i.e. TM) for resale, with the benefit that since they were the original seller, they actually reissue the tickets with new barcodes and they are guaranteed to be accurate and work for entry. They are quite happy to work with the broker industry on this side, and will integrate with larger brokers to allow listings to be easily updated.
Personally, I think the bad names brokers get as a whole are often unwarranted, as there is a lot of benefit to a secondary market that people often ignore when they see extremely expensive tickets, such as extremely underpriced tickets for events that bomb (or another date is added), and the ability to find tickets for an event long after it's sold out (without a resale market, you would have little recourse to see an event that sold out minutes after it went on sale unless you got your tickets right then). That said, there are some bad actors in this arena, which brings us to the following...
> I don't mean terrible as in they do the job poorly, it's just that they care 0% about the user experience during the entire ticketing process from end-to-end.
TM has a really hard job. I'm not sure you understand the scale of traffic they get on a popular event's main sale. They can literally sell out a 30,000 seat venue in seconds. Keep in mind, on a sale that's busy like that, there are often brokers (the bad actors referenced before) that are running industry software that leverages AWS/GCE/Linode/Rackspace, etc to access the main sale page tens to hundreds of times a second per instance during that sale to get CAPTCHA words and answer them. Since the CAPTCHA success rate is relatively low, they brute force it with many, many parallel requests. Consider that during peak sale season, there may be multiple of these large sales per hour, all set to open on the hour. TM is constantly battling these brokers by changing subtly how their ticketing system functions. It's not that they don't care about customer experience (although they do have enough of the market to not have to worry too much), but that it's only part of an equation that must also be balanced with insane levels of load balancing and queuing to handle the crazy load they see.
Now, I don't want to give the impression that TM is without fault. They've definitely got their fingers in multiple industries here which historically have had quite a bit of animosity, and they are fairly good at playing the media and the industries against one each other while secretly assuaging them in the background. The fact of the matter is that you as a customer aren't their only concern and aren't their only business. They are actively balancing the needs of the promoters, artists, venue and brokers, because their business caters to all of them.
Edit: Clarified scope of virtual server usage slightly.
Yeah, I'm beating on them. But I worked at a Ticketmaster outlet in college, and the convenience fee was only $2. That paid for the special ticket printing machine, the leased line to their network, and the blank ticket stock & ink rolls. These days, they don't have those costs as the customers are the ones supplying the hardware, internet connection, and doing the printing. So there's no reason for the high fees.
Story time: Our best customers? Grateful Dead fans. They were happy to sit anywhere. Worst customers? Sports fans. "Can you show me that other seat? Can I see better from there?" "I'm sorry, that seat was sold to someone else."
>These days, they don't have those costs as the customers are the ones supplying the hardware, internet connection, and doing the printing. So there's no reason for the high fees.
Unless you buy tickets online, as the vast majority of people do.
I think you misunderstand. The customers (people buying tickets) are using their own computers, their own internet connection, and their own printers (with that oh-so-cheap ink) to buy and print their event tickets these days. So Ticketmaster is no longer supplying all that specialized equipment to the stores and no longer has those costs.
For some venues and ticket types, you can just let them scan the PDF on your phone. For a subset of those, there's an actual mobile ticket type where the only way to get in is to show the ticket from the TicketMaster/LiveNation app, and there is no PDF.
Ticketfly is doing a good job. They do charge some fees, but not to the egregious extent of Ticketmaster, and have a much better interface that holds up against traffic.
Unfortunately, Ticketmaster has a habit of just buying/merging with competitors (including Live Nation, in a merger that should have never been allowed to go through).
You can't disrupt Ticketmaster's service fees because it's the venues and the peformers that are also profiting from those service fees.
Apparently, Ticketmaster's business model is based on taking the negative public relations hit for charging higher prices on behalf of their customers.
If you create a startup called CheaperTickets.com with the noble intention of not charging those hated service fees, you won't get any customers. (Keep in mind it's the venues & the artists that are the real customers of Ticketmaster and not the ticket buyers). The venues want the lucrative cut of the service fees while Ticketmaster gladly takes the heat.
I don't know TM's numbers, but I know it is correct that processing fees are not always going 100% to the ticketing company. My company is in the music business and one of our products is for internal ticket sales. We collect fees on behalf of the seller. The sellers portion varies, but is usually about 75% of the fee - meaning that we, the processor, are only getting 25% of the fee that the customer sees.
Similarly, I've worked for an organization that charges a per ticket service fee. They take a 100% cut off this, when you consider their ticketing contract was negotiated as a fixed annual cost, not a per-ticket or %-of-revenue.
This just tells me that the original venues were not charging enough for tickets. If the tickets were priced ideally then StubHub would not be able to do this; the margins would be too low for the amount of risk involved.
Most business don't maximize short term revenue in that way. Compare it to any new tech gadget, especially ones with hard supply limitations like the limitations that exist for live events (thus preventing a true market meeting point of supply and demand). I'm sure Apple could sell a lot of iPhones at $1500 until supply stabilizes, reduce the price $200, then rinse and repeat. That might reduce the secondary market and be the ideal pricing strategy, but it would be a bad business decision especially from a public relations standpoint. Just like there will always be people waiting in line at Apple stores on day 1 to grab a device and throw in on eBay, there will always be ticket brokers doing the same.
Do you have proof of StubHub actually doing that? I buy/sell a lot on the secondary ticket market and don't think StubHub is purchasing tickets themselves (except to cover a gurantee.)
Now individual/corporate ticket brokers are for sure buying up bulk tickets and putting them on the secondary market through StubHub... but StubHub doesn't have anything to do with that besides being the platform utilized.
This business doesn't need disruption in the technical sense; there are plenty of alternatives, which in some cases, are even very easy to switch to.
The barriers are the very typical ones that big companies (as a matter of fact, I assume that the venue with the greatest exposure are in fact big companies) face towards innovation: bureaucracy, lock-in, inertia, PHBs, and so on.
Not that TM doesn't have any positive side - it's just a very typical example of the business workings of large companies (not all, of course).
Seems nobody has reviewed the actual article... So, I will: it's great. I could see this model, esp illustrations, getting somewhere with lay management that run things by the numbers and ROI. I'd like to try it on bankers given they're experts on getting people stuck in debt. Last thing they want to do is be caught in their own type of trap. I'd like to see more work like this, esp improving manual measurement areas.
I'll agree with you there, I like seeing the thought process that fed into this. We've had similar experiences having difficulty explaining why certain things are painful and then quantifying "how painful" and why. This seems to really try to address that, at least as a first step.
Can you summarize how they quantitatively measure aspects of tech debt? Because that seems to be a hard problem and I kind of don't believe that they have pulled off this feat
It's described on the page along with qualitative measurements they use. They used each where it made sense to produce something pretty solid for management to work with. The alternatives are currently... I don't know any...
So, until I see them, I think they did a good job going from using a vague concept to practical metrics. Baby steps, for sure, but with immediate, payoff potential.
Firstly I'll admit, I don't buy many tickets in general. However the times I used Ticketmaster I've had a neutral to positive experience. I'm always a bit annoyed at the service fees. But it takes little time to buy them, and the tickets go straight into my inbox. Whatever. I think there are more pressing issues to complain about
If you've ever tried to purchase tickets for a concert that you know would be sold out, the process is horrendous.
I mean, they know when they'll have a surge in customers (there's a freakin countdown clock to the purchase time) and yet they still can't set up something more elastic that actually allows me to load the page?
How an individual business has more lines of code than the largest and most complex scientific experiment ever conceived by human beings is...I dunno, embarrassing?
Strange how no-one here seems to have actually read the article. Do people have thoughts on how they handle technical debt and manage code quality? This was a well thought out way of structurally improving code quality in a code base that supports a big business driving 10billion$+ transaction volume. Do people have a better solution to what they should be doing?
RE:disrupting their business model, why is charging money for a service wrong? You cannot keep things for free by burning VC money. That only lasts so long...
52 comments
[ 5.3 ms ] story [ 152 ms ] threadIt was "interesting" in the Chinese sense but an absolute PITA to work with.
And that's not mention the WAN linking the cinemas based on packet radio that blocked :-)
What's that?
[0] https://en.wikipedia.org/wiki/May_you_live_in_interesting_ti...
It's actually neither Chinese nor ancient: http://www.phrases.org.uk/meanings/may-you-live-in-interesti...
Last I've heard, they've been slowly moving functionality out of the VAX operating system into a more traditional stack.
Source: I used to work on the team responsible for the Linux servers that ran these virtualized vaxen, check the resume linked in my profile.
[1] http://simh.trailing-edge.com/
Once I have bought tickets, traveled 500 kilometers, stayed at hotel near venue. Once I got there, there was dozens of people confused about place being shut. It turned out that the gig as cancelled and Ticketmaster forgot to notify buyers about it. This was for one of the well known US bands, so there was a lot of people there. The reason of cancellation was sickness of one of the members of the band that was announced on other site also selling tickets a week earlier so only Ticketmaster customers were affected.
Imagine disappointment. Also, I didn't get refund for tickets either despite requesting it.
Ticketing systems are products sold to a presenter or promoter (or venue) and facilitate their business.
Their job isn't really to do either of those things, and I'm not being glib. Let me explain:
First, TicketMaster (TM) doesn't really want to print your tickets. They would much rather just digitally manage them, and let you use your phone to go the the venue. Non-digital media costs real money, and TM would prefer to charge you for the convenience of managing your tickets, or sending you a PDF. As a bonus to them, this lets you easily resell your tickets as a fan on their TM+ exchange.
Secondly, TM doesn't really want to stop scalpers, at least not completely. This one is a bit complex, because portions of TM are definitely not scalper (i.e. broker) friendly, while other portions are. TM runs their own secondary market, called TM+, which is a "fan resale" market, in which they list ticket that were sold on the primary market (i.e. TM) for resale, with the benefit that since they were the original seller, they actually reissue the tickets with new barcodes and they are guaranteed to be accurate and work for entry. They are quite happy to work with the broker industry on this side, and will integrate with larger brokers to allow listings to be easily updated.
Personally, I think the bad names brokers get as a whole are often unwarranted, as there is a lot of benefit to a secondary market that people often ignore when they see extremely expensive tickets, such as extremely underpriced tickets for events that bomb (or another date is added), and the ability to find tickets for an event long after it's sold out (without a resale market, you would have little recourse to see an event that sold out minutes after it went on sale unless you got your tickets right then). That said, there are some bad actors in this arena, which brings us to the following...
> I don't mean terrible as in they do the job poorly, it's just that they care 0% about the user experience during the entire ticketing process from end-to-end.
TM has a really hard job. I'm not sure you understand the scale of traffic they get on a popular event's main sale. They can literally sell out a 30,000 seat venue in seconds. Keep in mind, on a sale that's busy like that, there are often brokers (the bad actors referenced before) that are running industry software that leverages AWS/GCE/Linode/Rackspace, etc to access the main sale page tens to hundreds of times a second per instance during that sale to get CAPTCHA words and answer them. Since the CAPTCHA success rate is relatively low, they brute force it with many, many parallel requests. Consider that during peak sale season, there may be multiple of these large sales per hour, all set to open on the hour. TM is constantly battling these brokers by changing subtly how their ticketing system functions. It's not that they don't care about customer experience (although they do have enough of the market to not have to worry too much), but that it's only part of an equation that must also be balanced with insane levels of load balancing and queuing to handle the crazy load they see.
Now, I don't want to give the impression that TM is without fault. They've definitely got their fingers in multiple industries here which historically have had quite a bit of animosity, and they are fairly good at playing the media and the industries against one each other while secretly assuaging them in the background. The fact of the matter is that you as a customer aren't their only concern and aren't their only business. They are actively balancing the needs of the promoters, artists, venue and brokers, because their business caters to all of them.
Edit: Clarified scope of virtual server usage slightly.
Basically nothing.
Yeah, I'm beating on them. But I worked at a Ticketmaster outlet in college, and the convenience fee was only $2. That paid for the special ticket printing machine, the leased line to their network, and the blank ticket stock & ink rolls. These days, they don't have those costs as the customers are the ones supplying the hardware, internet connection, and doing the printing. So there's no reason for the high fees.
Story time: Our best customers? Grateful Dead fans. They were happy to sit anywhere. Worst customers? Sports fans. "Can you show me that other seat? Can I see better from there?" "I'm sorry, that seat was sold to someone else."
Unless you buy tickets online, as the vast majority of people do.
Unfortunately, Ticketmaster has a habit of just buying/merging with competitors (including Live Nation, in a merger that should have never been allowed to go through).
Plus, Live Nation actually owns a lot of venues, so you'll never see them use another ticketing provider: https://en.wikipedia.org/wiki/Live_Nation_Entertainment#Venu...
Apparently, Ticketmaster's business model is based on taking the negative public relations hit for charging higher prices on behalf of their customers.
If you create a startup called CheaperTickets.com with the noble intention of not charging those hated service fees, you won't get any customers. (Keep in mind it's the venues & the artists that are the real customers of Ticketmaster and not the ticket buyers). The venues want the lucrative cut of the service fees while Ticketmaster gladly takes the heat.
[1]http://www.laweekly.com/music/ticketmaster-and-servants-band...
[2]http://www.reddit.com/r/todayilearned/comments/1xlt0i/til_ti...
Pearl Jam tried to stand up to Ticketmaster with a boycott and lost.
Ticketmaster is practically a saint next to them in my eyes.
Ticketmaster owns a Stubhub competitor. https://en.wikipedia.org/wiki/TicketsNow
Now individual/corporate ticket brokers are for sure buying up bulk tickets and putting them on the secondary market through StubHub... but StubHub doesn't have anything to do with that besides being the platform utilized.
The barriers are the very typical ones that big companies (as a matter of fact, I assume that the venue with the greatest exposure are in fact big companies) face towards innovation: bureaucracy, lock-in, inertia, PHBs, and so on.
Not that TM doesn't have any positive side - it's just a very typical example of the business workings of large companies (not all, of course).
So, until I see them, I think they did a good job going from using a vague concept to practical metrics. Baby steps, for sure, but with immediate, payoff potential.
I mean, they know when they'll have a surge in customers (there's a freakin countdown clock to the purchase time) and yet they still can't set up something more elastic that actually allows me to load the page?
Problem solved, no?
RE:disrupting their business model, why is charging money for a service wrong? You cannot keep things for free by burning VC money. That only lasts so long...