Enterprise

How Bloomberg’s data center operations move the market

Bloomberg built its own network to serve its Wall Street customers, and while it works closely with cloud providers to serve certain customers, the secret sauce behind "the Terminal" is all in-house.

Outside the Stock Exchange

The Terminal is basically the popular shorthand for Bloomberg Professional Services, the arm of the company that sources real-time trading data from stock exchanges around the world and feeds it to traders and investors desperate for an edge.

Photo: Wikimedia Commons

There aren't a lot of companies that can credibly claim to be world-class tech infrastructure builders on par with the cloud giants. But Bloomberg, at the heart of the world's financial system, wouldn't have it any other way.

The bulk of the services offered by Bloomberg's financial empire run inside company-managed data centers filled with "highly tuned" Linux servers refined over the years to deliver a huge amount of real-time data to its customers, said Shawn Edwards, Bloomberg's chief technology officer in a recent interview with Protocol. It built its own private network to serve those customers — institutional investors, hedge funds, large banks — long before the cloud providers were operating at their current scale, because "some things don't lend themselves to the public web," he said.

But that doesn't mean Bloomberg is ignoring modern enterprise computing trends. It has worked extensively with AWS to offer services through the cloud giant to customers that want to work on the cloud, and has embraced new ideas such as containers and Kubernetes alongside systems that have been running for over a decade.

"We're big believers in evolution, instead of always having to write something brand new and have to take it over," Edwards said. The result is a sprawling array of enterprise tech that gives over 6,000 Bloomberg engineers the infrastructure to manage 200 billion messages a day containing market-moving information.

A cloud before the cloud

Bloomberg rolled its own technology from the very beginning.

"Bloomberg always had a modern architecture," Edwards said. "Chuck Zegar and Tom Secunda, two of the founders along with Mike Bloomberg, they had built kind of a web model before the web."

First released in 1982 for Merrill Lynch, Bloomberg's centralized computing model in New York powered what we now know as "the Terminal," the source of the lion's share of Bloomberg's revenue and profit. The Terminal is basically the popular shorthand for Bloomberg Professional Services, the arm of the company that sources real-time trading data from stock exchanges around the world and feeds it to traders and investors desperate for an edge.

That original system was written in Fortran, a programming language that predates COBOL, but modified in C++ around the time Edwards joined the company 17 years ago, he said.

Around that time, Bloomberg made several other changes to its underlying infrastructure that would eventually become blueprints for future distribution computing applications. It built a new user interface in JavaScript that ran server-side with "a lightweight toolkit" running on the client side, years before node.js simplified the process of running JavaScript — the most popular programming language running the last eight years, according to Stack Overflow — on both the server side and client side of an application, Edwards said.

It also began to organize its applications around a service-oriented architecture, building its own middleware that was similar to an open-source called gRPC released by Google in 2015 that brought the concept of microservices to a larger audience.

And on the hardware side, Bloomberg made a big bet on OpenStack, which over the years has become an example of how not to run an open standards organization. OpenStack was a response to some of the cloud computing concepts pioneered by AWS, but designed for and by tech vendors catering to companies that thought they still wanted to manage their own data centers.

Over time, it became clear to a lot of those end users and vendors that nothing was going to stop AWS, and support for OpenStack fizzled. But it still provided a solid blueprint for companies like Bloomberg that had already invested a ton of money in data center infrastructure, and Bloomberg continues to run much of its operation on that combination of hardware and software design principles.

Teaching the machines

Today, Bloomberg is still operating much of the same technology but with a few modern flourishes here and there.

OpenStack can't solve all its needs, and Bloomberg does run "purpose-built dedicated hardware for things that require it," Edwards said, such as "real-time data that doesn't belong on [virtual machines]."

It doesn't actually build its own servers like the major cloud companies do, but it is very picky about the hardware it puts into its environment and tweaks the Linux kernel running on those servers around its unique needs. The company has "experimented" with special-purpose chips like FPGAs (field programmable gate arrays) and hardware accelerators, Edwards said, but for the most part relies on off-the-shelf hardware customized by its engineering team.

Bloomberg does use public cloud services for what Edwards called the "dot-com" parts of its business, such as the media properties like Bloomberg News. It also meets customers where they are; if Bloomberg customers are running their own servers in public clouds, the company has worked with AWS and Google to link its own infrastructure with the public cloud servers used by its customers, he said.

Edwards is currently focused on improving Bloomberg's use of machine-learning techniques to improve its services.

The company thinks it has one of the best optical-character recognition systems in the world, which allows it to pull text and objects like tables out of company filings and financial statements and present it in a readable format. It's using machine learning to help predict the price of a bond, which is more complicated than it might sound because bonds trade far less frequently than stocks, and it's also using these techniques to help traders prioritize incoming messages from clients and capitalize on trade opportunities.

"We're a big data company, but we don't have nearly the amount of bytes in storage that Instagram has. But we have an embarrassingly large amount of heterogeneous data sets," Edwards said. "So when we say we understand documents, it's because of the 30 years of putting this together."

Fintech

Wall Street is warming up to crypto

Secure, well-regulated technology infrastructure could draw more large banks to crypto.

Technology infrastructure for crypto has begun to mature.

Illustration: Christopher T. Fong/Protocol

Despite a downturn in crypto markets, more large institutional investors are seeking to invest in crypto.

One factor holding them back is a lack of infrastructure for large institutions compared to what exists in the traditional, regulated capital markets.

Keep Reading Show less
Tomio Geron

Tomio Geron ( @tomiogeron) is a San Francisco-based reporter covering fintech. He was previously a reporter and editor at The Wall Street Journal, covering venture capital and startups. Before that, he worked as a staff writer at Forbes, covering social media and venture capital, and also edited the Midas List of top tech investors. He has also worked at newspapers covering crime, courts, health and other topics. He can be reached at tgeron@protocol.com or tgeron@protonmail.com.

Sponsored Content

Great products are built on strong patents

Experts say robust intellectual property protection is essential to ensure the long-term R&D required to innovate and maintain America's technology leadership.

Every great tech product that you rely on each day, from the smartphone in your pocket to your music streaming service and navigational system in the car, shares one important thing: part of its innovative design is protected by intellectual property (IP) laws.

From 5G to artificial intelligence, IP protection offers a powerful incentive for researchers to create ground-breaking products, and governmental leaders say its protection is an essential part of maintaining US technology leadership. To quote Secretary of Commerce Gina Raimondo: "intellectual property protection is vital for American innovation and entrepreneurship.”

Keep Reading Show less
James Daly
James Daly has a deep knowledge of creating brand voice identity, including understanding various audiences and targeting messaging accordingly. He enjoys commissioning, editing, writing, and business development, particularly in launching new ventures and building passionate audiences. Daly has led teams large and small to multiple awards and quantifiable success through a strategy built on teamwork, passion, fact-checking, intelligence, analytics, and audience growth while meeting budget goals and production deadlines in fast-paced environments. Daly is the Editorial Director of 2030 Media and a contributor at Wired.
Policy

How I decided to go all-in on a federal contract — before assignment

Amanda Renteria knew Code for America could help facilitate access to expanded child tax credits. She also knew there was no guarantee her proof of concept would convince others — but tried anyway.

Code for America CEO Amanda Renteria explained how it's helped people claim the Child Tax Credit.

Photo: Code for America

Click banner image for more How I decided series

After the American Rescue Plan Act passed in March 2021, the U.S. government expanded child tax credits to provide relief for American families during the pandemic. The legislation allowed some families to nearly double their tax benefits per child, which was especially critical for low-income families, who disproportionately bore the financial brunt of the pandemic.

Keep Reading Show less
Hirsh Chitkara

Hirsh Chitkara ( @HirshChitkara) is a reporter at Protocol focused on the intersection of politics, technology and society. Before joining Protocol, he helped write a daily newsletter at Insider that covered all things Big Tech. He's based in New York and can be reached at hchitkara@protocol.com.

Climate

This carbon capture startup wants to clean up the worst polluters

The founder and CEO of point-source carbon capture company Carbon Clean discusses what the startup has learned, the future of carbon capture technology, as well as the role of companies like his in battling the climate crisis.

Carbon Clean CEO Aniruddha Sharma told Protocol that fossil fuels are necessary, at least in the near term, to lift the living standards of those who don’t have access to cars and electricity.

Photo: Carbon Clean

Carbon capture and storage has taken on increasing importance as companies with stubborn emissions look for new ways to meet their net zero goals. For hard-to-abate industries like cement and steel production, it’s one of the few options that exist to help them get there.

Yet it’s proven incredibly challenging to scale the technology, which captures carbon pollution at the source. U.K.-based company Carbon Clean is leading the charge to bring down costs. This year, it raised a $150 million series C round, which the startup said is the largest-ever funding round for a point-source carbon capture company.

Keep Reading Show less
Michelle Ma

Michelle Ma (@himichellema) is a reporter at Protocol covering climate. Previously, she was a news editor of live journalism and special coverage for The Wall Street Journal. Prior to that, she worked as a staff writer at Wirecutter. She can be reached at mma@protocol.com.

Workplace

Why companies cut staff after raising millions

Are tech firms blowing millions in funding just weeks after getting it? Experts say it's more complicated than that.

Bolt, Trade Republic, HomeLight, and Stord all drew attention from funding announcements that happened just weeks or days before layoffs.

Photo: Pulp Photography/Getty Images

Fintech startup Bolt was one of the first tech companies to slash jobs, cutting 250 employees, or a third of its staff, in May. For some workers, the pain of layoffs was a shock not only because they were the first, but also because the cuts came just four months after Bolt had announced a $355 million series E funding round and achieved a peak valuation of $11 billion.

“Bolt employees were blind sided because the CEO was saying just weeks ago how everything is fine,” an anonymous user wrote on the message board Blind. “It has been an extremely rough day for 1/3 of Bolt employees,” another user posted. “Sadly, I was one of them who was let go after getting a pay-raise just a couple of weeks ago.”

Keep Reading Show less
Nat Rubio-Licht

Nat Rubio-Licht is a Los Angeles-based news writer at Protocol. They graduated from Syracuse University with a degree in newspaper and online journalism in May 2020. Prior to joining the team, they worked at the Los Angeles Business Journal as a technology and aerospace reporter.

Latest Stories
Bulletins