Enterprise

How Bloomberg’s data center operations move the market

Bloomberg built its own network to serve its Wall Street customers, and while it works closely with cloud providers to serve certain customers, the secret sauce behind "the Terminal" is all in-house.

Outside the Stock Exchange

The Terminal is basically the popular shorthand for Bloomberg Professional Services, the arm of the company that sources real-time trading data from stock exchanges around the world and feeds it to traders and investors desperate for an edge.

Photo: Wikimedia Commons

There aren't a lot of companies that can credibly claim to be world-class tech infrastructure builders on par with the cloud giants. But Bloomberg, at the heart of the world's financial system, wouldn't have it any other way.

The bulk of the services offered by Bloomberg's financial empire run inside company-managed data centers filled with "highly tuned" Linux servers refined over the years to deliver a huge amount of real-time data to its customers, said Shawn Edwards, Bloomberg's chief technology officer in a recent interview with Protocol. It built its own private network to serve those customers — institutional investors, hedge funds, large banks — long before the cloud providers were operating at their current scale, because "some things don't lend themselves to the public web," he said.

But that doesn't mean Bloomberg is ignoring modern enterprise computing trends. It has worked extensively with AWS to offer services through the cloud giant to customers that want to work on the cloud, and has embraced new ideas such as containers and Kubernetes alongside systems that have been running for over a decade.

"We're big believers in evolution, instead of always having to write something brand new and have to take it over," Edwards said. The result is a sprawling array of enterprise tech that gives over 6,000 Bloomberg engineers the infrastructure to manage 200 billion messages a day containing market-moving information.

A cloud before the cloud

Bloomberg rolled its own technology from the very beginning.

"Bloomberg always had a modern architecture," Edwards said. "Chuck Zegar and Tom Secunda, two of the founders along with Mike Bloomberg, they had built kind of a web model before the web."

First released in 1982 for Merrill Lynch, Bloomberg's centralized computing model in New York powered what we now know as "the Terminal," the source of the lion's share of Bloomberg's revenue and profit. The Terminal is basically the popular shorthand for Bloomberg Professional Services, the arm of the company that sources real-time trading data from stock exchanges around the world and feeds it to traders and investors desperate for an edge.

That original system was written in Fortran, a programming language that predates COBOL, but modified in C++ around the time Edwards joined the company 17 years ago, he said.

Around that time, Bloomberg made several other changes to its underlying infrastructure that would eventually become blueprints for future distribution computing applications. It built a new user interface in JavaScript that ran server-side with "a lightweight toolkit" running on the client side, years before node.js simplified the process of running JavaScript — the most popular programming language running the last eight years, according to Stack Overflow — on both the server side and client side of an application, Edwards said.

It also began to organize its applications around a service-oriented architecture, building its own middleware that was similar to an open-source called gRPC released by Google in 2015 that brought the concept of microservices to a larger audience.

And on the hardware side, Bloomberg made a big bet on OpenStack, which over the years has become an example of how not to run an open standards organization. OpenStack was a response to some of the cloud computing concepts pioneered by AWS, but designed for and by tech vendors catering to companies that thought they still wanted to manage their own data centers.

Over time, it became clear to a lot of those end users and vendors that nothing was going to stop AWS, and support for OpenStack fizzled. But it still provided a solid blueprint for companies like Bloomberg that had already invested a ton of money in data center infrastructure, and Bloomberg continues to run much of its operation on that combination of hardware and software design principles.

Teaching the machines

Today, Bloomberg is still operating much of the same technology but with a few modern flourishes here and there.

OpenStack can't solve all its needs, and Bloomberg does run "purpose-built dedicated hardware for things that require it," Edwards said, such as "real-time data that doesn't belong on [virtual machines]."

It doesn't actually build its own servers like the major cloud companies do, but it is very picky about the hardware it puts into its environment and tweaks the Linux kernel running on those servers around its unique needs. The company has "experimented" with special-purpose chips like FPGAs (field programmable gate arrays) and hardware accelerators, Edwards said, but for the most part relies on off-the-shelf hardware customized by its engineering team.

Bloomberg does use public cloud services for what Edwards called the "dot-com" parts of its business, such as the media properties like Bloomberg News. It also meets customers where they are; if Bloomberg customers are running their own servers in public clouds, the company has worked with AWS and Google to link its own infrastructure with the public cloud servers used by its customers, he said.

Edwards is currently focused on improving Bloomberg's use of machine-learning techniques to improve its services.

The company thinks it has one of the best optical-character recognition systems in the world, which allows it to pull text and objects like tables out of company filings and financial statements and present it in a readable format. It's using machine learning to help predict the price of a bond, which is more complicated than it might sound because bonds trade far less frequently than stocks, and it's also using these techniques to help traders prioritize incoming messages from clients and capitalize on trade opportunities.

"We're a big data company, but we don't have nearly the amount of bytes in storage that Instagram has. But we have an embarrassingly large amount of heterogeneous data sets," Edwards said. "So when we say we understand documents, it's because of the 30 years of putting this together."

Fintech

Judge Zia Faruqui is trying to teach you crypto, one ‘SNL’ reference at a time

His decisions on major cryptocurrency cases have quoted "The Big Lebowski," "SNL," and "Dr. Strangelove." That’s because he wants you — yes, you — to read them.

The ways Zia Faruqui (right) has weighed on cases that have come before him can give lawyers clues as to what legal frameworks will pass muster.

Photo: Carolyn Van Houten/The Washington Post via Getty Images

“Cryptocurrency and related software analytics tools are ‘The wave of the future, Dude. One hundred percent electronic.’”

That’s not a quote from "The Big Lebowski" — at least, not directly. It’s a quote from a Washington, D.C., district court memorandum opinion on the role cryptocurrency analytics tools can play in government investigations. The author is Magistrate Judge Zia Faruqui.

Keep Reading Show less
Veronica Irwin

Veronica Irwin (@vronirwin) is a San Francisco-based reporter at Protocol covering fintech. Previously she was at the San Francisco Examiner, covering tech from a hyper-local angle. Before that, her byline was featured in SF Weekly, The Nation, Techworker, Ms. Magazine and The Frisc.

The financial technology transformation is driving competition, creating consumer choice, and shaping the future of finance. Hear from seven fintech leaders who are reshaping the future of finance, and join the inaugural Financial Technology Association Fintech Summit to learn more.

Keep Reading Show less
FTA
The Financial Technology Association (FTA) represents industry leaders shaping the future of finance. We champion the power of technology-centered financial services and advocate for the modernization of financial regulation to support inclusion and responsible innovation.
Enterprise

AWS CEO: The cloud isn’t just about technology

As AWS preps for its annual re:Invent conference, Adam Selipsky talks product strategy, support for hybrid environments, and the value of the cloud in uncertain economic times.

Photo: Noah Berger/Getty Images for Amazon Web Services

AWS is gearing up for re:Invent, its annual cloud computing conference where announcements this year are expected to focus on its end-to-end data strategy and delivering new industry-specific services.

It will be the second re:Invent with CEO Adam Selipsky as leader of the industry’s largest cloud provider after his return last year to AWS from data visualization company Tableau Software.

Keep Reading Show less
Donna Goodison

Donna Goodison (@dgoodison) is Protocol's senior reporter focusing on enterprise infrastructure technology, from the 'Big 3' cloud computing providers to data centers. She previously covered the public cloud at CRN after 15 years as a business reporter for the Boston Herald. Based in Massachusetts, she also has worked as a Boston Globe freelancer, business reporter at the Boston Business Journal and real estate reporter at Banker & Tradesman after toiling at weekly newspapers.

Image: Protocol

We launched Protocol in February 2020 to cover the evolving power center of tech. It is with deep sadness that just under three years later, we are winding down the publication.

As of today, we will not publish any more stories. All of our newsletters, apart from our flagship, Source Code, will no longer be sent. Source Code will be published and sent for the next few weeks, but it will also close down in December.

Keep Reading Show less
Bennett Richardson

Bennett Richardson ( @bennettrich) is the president of Protocol. Prior to joining Protocol in 2019, Bennett was executive director of global strategic partnerships at POLITICO, where he led strategic growth efforts including POLITICO's European expansion in Brussels and POLITICO's creative agency POLITICO Focus during his six years with the company. Prior to POLITICO, Bennett was co-founder and CMO of Hinge, the mobile dating company recently acquired by Match Group. Bennett began his career in digital and social brand marketing working with major brands across tech, energy, and health care at leading marketing and communications agencies including Edelman and GMMB. Bennett is originally from Portland, Maine, and received his bachelor's degree from Colgate University.

Enterprise

Why large enterprises struggle to find suitable platforms for MLops

As companies expand their use of AI beyond running just a few machine learning models, and as larger enterprises go from deploying hundreds of models to thousands and even millions of models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

As companies expand their use of AI beyond running just a few machine learning models, ML practitioners say that they have yet to find what they need from prepackaged MLops systems.

Photo: artpartner-images via Getty Images

On any given day, Lily AI runs hundreds of machine learning models using computer vision and natural language processing that are customized for its retail and ecommerce clients to make website product recommendations, forecast demand, and plan merchandising. But this spring when the company was in the market for a machine learning operations platform to manage its expanding model roster, it wasn’t easy to find a suitable off-the-shelf system that could handle such a large number of models in deployment while also meeting other criteria.

Some MLops platforms are not well-suited for maintaining even more than 10 machine learning models when it comes to keeping track of data, navigating their user interfaces, or reporting capabilities, Matthew Nokleby, machine learning manager for Lily AI’s product intelligence team, told Protocol earlier this year. “The duct tape starts to show,” he said.

Keep Reading Show less
Kate Kaye

Kate Kaye is an award-winning multimedia reporter digging deep and telling print, digital and audio stories. She covers AI and data for Protocol. Her reporting on AI and tech ethics issues has been published in OneZero, Fast Company, MIT Technology Review, CityLab, Ad Age and Digiday and heard on NPR. Kate is the creator of RedTailMedia.org and is the author of "Campaign '08: A Turning Point for Digital Media," a book about how the 2008 presidential campaigns used digital media and data.

Latest Stories
Bulletins