Source Code: Your daily look at what matters in tech.

enterpriseenterpriseauthorTom KrazitNoneAre you keeping up with the latest cloud developments? Get Tom Krazit and Joe Williams' newsletter every Monday and Thursday.d3d5b92349
×

Get access to Protocol

Your information will be used in accordance with our Privacy Policy

I’m already a subscriber
Protocol | Enterprise

How Bloomberg’s data center operations move the market

Bloomberg built its own network to serve its Wall Street customers, and while it works closely with cloud providers to serve certain customers, the secret sauce behind "the Terminal" is all in-house.

Outside the Stock Exchange

The Terminal is basically the popular shorthand for Bloomberg Professional Services, the arm of the company that sources real-time trading data from stock exchanges around the world and feeds it to traders and investors desperate for an edge.

Photo: Wikimedia Commons

There aren't a lot of companies that can credibly claim to be world-class tech infrastructure builders on par with the cloud giants. But Bloomberg, at the heart of the world's financial system, wouldn't have it any other way.

The bulk of the services offered by Bloomberg's financial empire run inside company-managed data centers filled with "highly tuned" Linux servers refined over the years to deliver a huge amount of real-time data to its customers, said Shawn Edwards, Bloomberg's chief technology officer in a recent interview with Protocol. It built its own private network to serve those customers — institutional investors, hedge funds, large banks — long before the cloud providers were operating at their current scale, because "some things don't lend themselves to the public web," he said.

But that doesn't mean Bloomberg is ignoring modern enterprise computing trends. It has worked extensively with AWS to offer services through the cloud giant to customers that want to work on the cloud, and has embraced new ideas such as containers and Kubernetes alongside systems that have been running for over a decade.

"We're big believers in evolution, instead of always having to write something brand new and have to take it over," Edwards said. The result is a sprawling array of enterprise tech that gives over 6,000 Bloomberg engineers the infrastructure to manage 200 billion messages a day containing market-moving information.

A cloud before the cloud

Bloomberg rolled its own technology from the very beginning.

"Bloomberg always had a modern architecture," Edwards said. "Chuck Zegar and Tom Secunda, two of the founders along with Mike Bloomberg, they had built kind of a web model before the web."

First released in 1982 for Merrill Lynch, Bloomberg's centralized computing model in New York powered what we now know as "the Terminal," the source of the lion's share of Bloomberg's revenue and profit. The Terminal is basically the popular shorthand for Bloomberg Professional Services, the arm of the company that sources real-time trading data from stock exchanges around the world and feeds it to traders and investors desperate for an edge.

That original system was written in Fortran, a programming language that predates COBOL, but modified in C++ around the time Edwards joined the company 17 years ago, he said.

Around that time, Bloomberg made several other changes to its underlying infrastructure that would eventually become blueprints for future distribution computing applications. It built a new user interface in JavaScript that ran server-side with "a lightweight toolkit" running on the client side, years before node.js simplified the process of running JavaScript — the most popular programming language running the last eight years, according to Stack Overflow — on both the server side and client side of an application, Edwards said.

It also began to organize its applications around a service-oriented architecture, building its own middleware that was similar to an open-source called gRPC released by Google in 2015 that brought the concept of microservices to a larger audience.

And on the hardware side, Bloomberg made a big bet on OpenStack, which over the years has become an example of how not to run an open standards organization. OpenStack was a response to some of the cloud computing concepts pioneered by AWS, but designed for and by tech vendors catering to companies that thought they still wanted to manage their own data centers.

Over time, it became clear to a lot of those end users and vendors that nothing was going to stop AWS, and support for OpenStack fizzled. But it still provided a solid blueprint for companies like Bloomberg that had already invested a ton of money in data center infrastructure, and Bloomberg continues to run much of its operation on that combination of hardware and software design principles.

Teaching the machines

Today, Bloomberg is still operating much of the same technology but with a few modern flourishes here and there.

OpenStack can't solve all its needs, and Bloomberg does run "purpose-built dedicated hardware for things that require it," Edwards said, such as "real-time data that doesn't belong on [virtual machines]."

It doesn't actually build its own servers like the major cloud companies do, but it is very picky about the hardware it puts into its environment and tweaks the Linux kernel running on those servers around its unique needs. The company has "experimented" with special-purpose chips like FPGAs (field programmable gate arrays) and hardware accelerators, Edwards said, but for the most part relies on off-the-shelf hardware customized by its engineering team.

Bloomberg does use public cloud services for what Edwards called the "dot-com" parts of its business, such as the media properties like Bloomberg News. It also meets customers where they are; if Bloomberg customers are running their own servers in public clouds, the company has worked with AWS and Google to link its own infrastructure with the public cloud servers used by its customers, he said.

Edwards is currently focused on improving Bloomberg's use of machine-learning techniques to improve its services.

The company thinks it has one of the best optical-character recognition systems in the world, which allows it to pull text and objects like tables out of company filings and financial statements and present it in a readable format. It's using machine learning to help predict the price of a bond, which is more complicated than it might sound because bonds trade far less frequently than stocks, and it's also using these techniques to help traders prioritize incoming messages from clients and capitalize on trade opportunities.

"We're a big data company, but we don't have nearly the amount of bytes in storage that Instagram has. But we have an embarrassingly large amount of heterogeneous data sets," Edwards said. "So when we say we understand documents, it's because of the 30 years of putting this together."

Power

The video game industry is bracing for its Netflix and Spotify moment

Subscription gaming promises to upend gaming. The jury's out on whether that's a good thing.

It's not clear what might fall through the cracks if most of the biggest game studios transition away from selling individual games and instead embrace a mix of free-to-play and subscription bundling.

Image: Christopher T. Fong/Protocol

Subscription services are coming for the game industry, and the shift could shake up the largest and most lucrative entertainment sector in the world. These services started as small, closed offerings typically available on only a handful of hardware platforms. Now, they're expanding to mobile phones and smart TVs, and promising to radically change the economics of how games are funded, developed and distributed.

Of the biggest companies in gaming today, Amazon, Apple, Electronic Arts, Google, Microsoft, Nintendo, Nvidia, Sony and Ubisoft all operate some form of game subscription. Far and away the most ambitious of them is Microsoft's Xbox Game Pass, featuring more than 100 games for $9.99 a month and including even brand-new titles the day they release. As of January, Game Pass had more than 18 million subscribers, and Microsoft's aggressive investment in a subscription future has become a catalyst for an industrywide reckoning on the likelihood and viability of such a model becoming standard.

Keep Reading Show less
Nick Statt
Nick Statt is Protocol's video game reporter. Prior to joining Protocol, he was news editor at The Verge covering the gaming industry, mobile apps and antitrust out of San Francisco, in addition to managing coverage of Silicon Valley tech giants and startups. He now resides in Rochester, New York, home of the garbage plate and, completely coincidentally, the World Video Game Hall of Fame. He can be reached at nstatt@protocol.com.

Over the last year, financial institutions have experienced unprecedented demand from their customers for exposure to cryptocurrency, and we've seen an inflow of institutional dollars driving bitcoin and other cryptocurrencies to record prices. Some banks have already launched cryptocurrency programs, but many more are evaluating the market.

That's why we've created the Crypto Maturity Model: an iterative roadmap for cryptocurrency product rollout, enabling financial institutions to evaluate market opportunities while addressing compliance requirements.

Keep Reading Show less
Caitlin Barnett, Chainanalysis
Caitlin’s legal and compliance experience encompasses both cryptocurrency and traditional finance. As Director of Regulation and Compliance at Chainalysis, she helps leading financial institutions strategize and build compliance programs in order to adopt cryptocurrencies and offer new products to their customers. In addition, Caitlin helps facilitate dialogue with regulators and the industry on key policy issues within the cryptocurrency industry.
Protocol | Policy

Lina Khan wants to hear from you

The new FTC chair is trying to get herself, and the sometimes timid tech-regulating agency she oversees, up to speed while she still can.

Lina Khan is trying to push the FTC to corral tech companies

Photo: Graeme Jennings/AFP via Getty Images

"When you're in D.C., it's very easy to lose connection with the very real issues that people are facing," said Lina Khan, the FTC's new chair.

Khan made her debut as chair before the press on Wednesday, showing up to a media event carrying an old maroon book from the agency's library and calling herself a "huge nerd" on FTC history. She launched into explaining how much she enjoys the open commission meetings she's pioneered since taking over in June. That's especially true of the marathon public comment sessions that have wrapped up each of the two meetings so far.

Keep Reading Show less
Ben Brody

Ben Brody (@ BenBrodyDC) is a senior reporter at Protocol focusing on how Congress, courts and agencies affect the online world we live in. He formerly covered tech policy and lobbying (including antitrust, Section 230 and privacy) at Bloomberg News, where he previously reported on the influence industry, government ethics and the 2016 presidential election. Before that, Ben covered business news at CNNMoney and AdAge, and all manner of stories in and around New York. He still loves appearing on the New York news radio he grew up with.

Protocol | Fintech

Beyond Robinhood: Stock exchange rebates are under scrutiny too

Some critics have compared the way exchanges attract orders from customers to the payment for order flow system that has enriched retail brokers.

The New York Stock Exchange is now owned by the Intercontinental Exchange.

Photo: Aditya Vyas/Unsplash

As questions pile up about how powerful and little-known Wall Street entities rake in profits from stock trading, the exchanges that handle vast portions of everyday trading are being scrutinized for how they make money, too.

One mechanism in particular — exchange rebates, or payments from the exchanges for getting certain trades routed to them — has raised concerns with regulators and members of Congress.

Keep Reading Show less
Tomio Geron

Tomio Geron ( @tomiogeron) is a San Francisco-based reporter covering fintech. He was previously a reporter and editor at The Wall Street Journal, covering venture capital and startups. Before that, he worked as a staff writer at Forbes, covering social media and venture capital, and also edited the Midas List of top tech investors. He has also worked at newspapers covering crime, courts, health and other topics. He can be reached at tgeron@protocol.com or tgeron@protonmail.com.

Protocol | Workplace

The Activision Blizzard lawsuit has opened the floodgates

An employee walkout, a tumbling stock price and damning new reports of misconduct.

Activision Blizzard is being sued for widespread sexism, harassment and discrimination.

Photo: Bloomberg/Getty Images

Activision Blizzard is in crisis mode. The World of Warcraft publisher was the subject of a shocking lawsuit filed by California's Department of Fair Employment and Housing last week over claims of widespread sexism, harassment and discrimination against female employees. The resulting fallout has only intensified by the day, culminating in a 500-person walkout at the headquarters of Blizzard Entertainment in Irvine on Wednesday.

The company's stock price has tumbled nearly 10% this week, and CEO Bobby Kotick acknowledged in a message to employees Tuesday that Activision Blizzard's initial response was "tone deaf." Meanwhile, there has been a continuous stream of new reports unearthing horrendous misconduct as more and more former and current employees speak out about the working conditions and alleged rampant misogyny at one of the video game industry's largest and most powerful employers.

Keep Reading Show less
Nick Statt
Nick Statt is Protocol's video game reporter. Prior to joining Protocol, he was news editor at The Verge covering the gaming industry, mobile apps and antitrust out of San Francisco, in addition to managing coverage of Silicon Valley tech giants and startups. He now resides in Rochester, New York, home of the garbage plate and, completely coincidentally, the World Video Game Hall of Fame. He can be reached at nstatt@protocol.com.
Latest Stories