Enterprise

How Bloomberg’s data center operations move the market

Bloomberg built its own network to serve its Wall Street customers, and while it works closely with cloud providers to serve certain customers, the secret sauce behind "the Terminal" is all in-house.

Outside the Stock Exchange

The Terminal is basically the popular shorthand for Bloomberg Professional Services, the arm of the company that sources real-time trading data from stock exchanges around the world and feeds it to traders and investors desperate for an edge.

Photo: Wikimedia Commons

There aren't a lot of companies that can credibly claim to be world-class tech infrastructure builders on par with the cloud giants. But Bloomberg, at the heart of the world's financial system, wouldn't have it any other way.

The bulk of the services offered by Bloomberg's financial empire run inside company-managed data centers filled with "highly tuned" Linux servers refined over the years to deliver a huge amount of real-time data to its customers, said Shawn Edwards, Bloomberg's chief technology officer in a recent interview with Protocol. It built its own private network to serve those customers — institutional investors, hedge funds, large banks — long before the cloud providers were operating at their current scale, because "some things don't lend themselves to the public web," he said.

But that doesn't mean Bloomberg is ignoring modern enterprise computing trends. It has worked extensively with AWS to offer services through the cloud giant to customers that want to work on the cloud, and has embraced new ideas such as containers and Kubernetes alongside systems that have been running for over a decade.

"We're big believers in evolution, instead of always having to write something brand new and have to take it over," Edwards said. The result is a sprawling array of enterprise tech that gives over 6,000 Bloomberg engineers the infrastructure to manage 200 billion messages a day containing market-moving information.

A cloud before the cloud

Bloomberg rolled its own technology from the very beginning.

"Bloomberg always had a modern architecture," Edwards said. "Chuck Zegar and Tom Secunda, two of the founders along with Mike Bloomberg, they had built kind of a web model before the web."

First released in 1982 for Merrill Lynch, Bloomberg's centralized computing model in New York powered what we now know as "the Terminal," the source of the lion's share of Bloomberg's revenue and profit. The Terminal is basically the popular shorthand for Bloomberg Professional Services, the arm of the company that sources real-time trading data from stock exchanges around the world and feeds it to traders and investors desperate for an edge.

That original system was written in Fortran, a programming language that predates COBOL, but modified in C++ around the time Edwards joined the company 17 years ago, he said.

Around that time, Bloomberg made several other changes to its underlying infrastructure that would eventually become blueprints for future distribution computing applications. It built a new user interface in JavaScript that ran server-side with "a lightweight toolkit" running on the client side, years before node.js simplified the process of running JavaScript — the most popular programming language running the last eight years, according to Stack Overflow — on both the server side and client side of an application, Edwards said.

It also began to organize its applications around a service-oriented architecture, building its own middleware that was similar to an open-source called gRPC released by Google in 2015 that brought the concept of microservices to a larger audience.

And on the hardware side, Bloomberg made a big bet on OpenStack, which over the years has become an example of how not to run an open standards organization. OpenStack was a response to some of the cloud computing concepts pioneered by AWS, but designed for and by tech vendors catering to companies that thought they still wanted to manage their own data centers.

Over time, it became clear to a lot of those end users and vendors that nothing was going to stop AWS, and support for OpenStack fizzled. But it still provided a solid blueprint for companies like Bloomberg that had already invested a ton of money in data center infrastructure, and Bloomberg continues to run much of its operation on that combination of hardware and software design principles.

Teaching the machines

Today, Bloomberg is still operating much of the same technology but with a few modern flourishes here and there.

OpenStack can't solve all its needs, and Bloomberg does run "purpose-built dedicated hardware for things that require it," Edwards said, such as "real-time data that doesn't belong on [virtual machines]."

It doesn't actually build its own servers like the major cloud companies do, but it is very picky about the hardware it puts into its environment and tweaks the Linux kernel running on those servers around its unique needs. The company has "experimented" with special-purpose chips like FPGAs (field programmable gate arrays) and hardware accelerators, Edwards said, but for the most part relies on off-the-shelf hardware customized by its engineering team.

Bloomberg does use public cloud services for what Edwards called the "dot-com" parts of its business, such as the media properties like Bloomberg News. It also meets customers where they are; if Bloomberg customers are running their own servers in public clouds, the company has worked with AWS and Google to link its own infrastructure with the public cloud servers used by its customers, he said.

Edwards is currently focused on improving Bloomberg's use of machine-learning techniques to improve its services.

The company thinks it has one of the best optical-character recognition systems in the world, which allows it to pull text and objects like tables out of company filings and financial statements and present it in a readable format. It's using machine learning to help predict the price of a bond, which is more complicated than it might sound because bonds trade far less frequently than stocks, and it's also using these techniques to help traders prioritize incoming messages from clients and capitalize on trade opportunities.

"We're a big data company, but we don't have nearly the amount of bytes in storage that Instagram has. But we have an embarrassingly large amount of heterogeneous data sets," Edwards said. "So when we say we understand documents, it's because of the 30 years of putting this together."

Enterprise

SaaS valuations cratered in early 2022. But these startups thrived.

VCs were still bullish on supply chain, recruiting and data startups despite the economic environment that chopped the valuations of newly public companies and late-stage enterprise startups.

While private equity has been investing in enterprise tech for decades, the confluence of several trends in the sector is making it more competitive than ever before.
Image: Getty Images; Protocol

Despite a volatile tech stock market so far this year that has included delayed IPOs, lowered valuations and declining investor sentiment, a few enterprise tech categories managed to keep getting funding. Data platforms, supply chain management tech, workplace software and cybersecurity startups all dominated the funding cycle over the past quarter.

When it comes to enterprise SaaS, the number of mega-deals — VC funding rounds over $100 million — spiked last year, according to data from Pitchbook. Partially driven by the onset of a pandemic that accelerated the need for everything from contact centers to supply chains to move into the cloud, the number of large VC deals tripled between 2020 and 2021. That growth has extended into this year, where the number of mega-deals has already outpaced all of 2020.

Keep Reading Show less
Aisha Counts

Aisha Counts (@aishacounts) is a reporter at Protocol covering enterprise software. Formerly, she was a management consultant for EY. She's based in Los Angeles and can be reached at acounts@protocol.com.

Sponsored Content

Foursquare data story: leveraging location data for site selection

We take a closer look at points of interest and foot traffic patterns to demonstrate how location data can be leveraged to inform better site selecti­on strategies.

Imagine: You’re the leader of a real estate team at a restaurant brand looking to open a new location in Manhattan. You have two options you’re evaluating: one site in SoHo, and another site in the Flatiron neighborhood. Which do you choose?

Keep Reading Show less
Fintech

Plaid is striking back after Stripe entered its core business

Onboarding customers through identity verification and ACH transfers is a hot sector in fintech, and the two fast-growing fintechs are set to battle it out.

Plaid is looking to help banks and fintech companies with anything related to the onboarding of a customer onto a financial product, said Plaid CTO Jean-Denis Greze.

Photo: Plaid

Plaid is moving into identity verification in a crucial expansion beyond its roots connecting banks and fintechs — a move that could put it in more direct competition with Stripe, another company known for its financial software tools.

In conjunction with its Plaid Forum customer conference this week, the company is also announcing two products focused on ACH transfers as it moves into payments.

Keep Reading Show less
Tomio Geron

Tomio Geron ( @tomiogeron) is a San Francisco-based reporter covering fintech. He was previously a reporter and editor at The Wall Street Journal, covering venture capital and startups. Before that, he worked as a staff writer at Forbes, covering social media and venture capital, and also edited the Midas List of top tech investors. He has also worked at newspapers covering crime, courts, health and other topics. He can be reached at tgeron@protocol.com or tgeron@protonmail.com.

Workplace

Getting reproductive benefits at work could be a privacy nightmare

A growing number of tech companies are extending abortion-related travel benefits. Given privacy and legal fears, will employees be too scared to use them?

How employers can implement and discuss reproductive benefits in a way that puts employees at ease.

Photo: Sigrid Gombert via Getty Images

It’s about to be a lot harder to get an abortion in the United States. For many, it’s already hard. The result is that employers, including large companies, are being called upon to fill the abortion care gap. The likelihood of a Roe v. Wade reversal was the push some needed to extend benefits, with Microsoft and Tesla announcing abortion-related travel reimbursements in recent weeks. But the privacy and legal risks facing people in need of abortions loom large. If people have reason to fear texting friends for abortion resources, will they really want to confide in their company?

An employee doesn’t have “much to worry about” when it comes to health privacy, said employee benefits consultant Jessica Du Bois. “The HR director or whoever's in charge of the benefits program is not going to be sharing that information.” Employers have a duty to protect employee health data under HIPAA and a variety of state laws. Companies with self-funded health plans — in other words, most large companies — can see every prescription and service an employee receives. But the data is deidentified.

Keep Reading Show less
Lizzy Lawrence

Lizzy Lawrence ( @LizzyLaw_) is a reporter at Protocol, covering tools and productivity in the workplace. She's a recent graduate of the University of Michigan, where she studied sociology and international studies. She served as editor in chief of The Michigan Daily, her school's independent newspaper. She's based in D.C., and can be reached at llawrence@protocol.com.

Enterprise

VMware CEO Raghu Raghuram: Edge is growing faster than cloud

The now-standalone company is staking its immediate future on the multicloud era of IT and hybrid work, while anticipating increased demand for edge-computing software.

VMware CEO Raghu Raghuram spoke with Protocol about the company's future.

Photo: VMware

Nearly a year into his tenure as CEO, Raghu Raghuram believes VMware is well-positioned for the third phase of its evolution, but acknowledges its product transformation still needs some work.

The company, which pioneered the hypervisor and expanded to virtualized networking and storage with its vSphere operating environment, now is helping customers navigate a distributed, multicloud world and hybrid work with newfound freedom as an independent company after being spun off from Dell Technologies last November.

Keep Reading Show less
Donna Goodison

Donna Goodison (@dgoodison) is Protocol's senior reporter focusing on enterprise infrastructure technology, from the 'Big 3' cloud computing providers to data centers. She previously covered the public cloud at CRN after 15 years as a business reporter for the Boston Herald. Based in Massachusetts, she also has worked as a Boston Globe freelancer, business reporter at the Boston Business Journal and real estate reporter at Banker & Tradesman after toiling at weekly newspapers.

Latest Stories
Bulletins