enterprise| enterpriseauthorTom KrazitNoneAre you keeping up with the latest cloud developments? Get Tom Krazit and Joe Williams' newsletter every Monday and Thursday.d3d5b92349
×

Get access to Protocol

I’ve already subscribed

Will be used in accordance with our Privacy Policy

Protocol | Enterprise

How Bloomberg’s data center operations move the market

Bloomberg built its own network to serve its Wall Street customers, and while it works closely with cloud providers to serve certain customers, the secret sauce behind "the Terminal" is all in-house.

Outside the Stock Exchange

The Terminal is basically the popular shorthand for Bloomberg Professional Services, the arm of the company that sources real-time trading data from stock exchanges around the world and feeds it to traders and investors desperate for an edge.

Photo: Wikimedia Commons

There aren't a lot of companies that can credibly claim to be world-class tech infrastructure builders on par with the cloud giants. But Bloomberg, at the heart of the world's financial system, wouldn't have it any other way.

The bulk of the services offered by Bloomberg's financial empire run inside company-managed data centers filled with "highly tuned" Linux servers refined over the years to deliver a huge amount of real-time data to its customers, said Shawn Edwards, Bloomberg's chief technology officer in a recent interview with Protocol. It built its own private network to serve those customers — institutional investors, hedge funds, large banks — long before the cloud providers were operating at their current scale, because "some things don't lend themselves to the public web," he said.

But that doesn't mean Bloomberg is ignoring modern enterprise computing trends. It has worked extensively with AWS to offer services through the cloud giant to customers that want to work on the cloud, and has embraced new ideas such as containers and Kubernetes alongside systems that have been running for over a decade.

"We're big believers in evolution, instead of always having to write something brand new and have to take it over," Edwards said. The result is a sprawling array of enterprise tech that gives over 6,000 Bloomberg engineers the infrastructure to manage 200 billion messages a day containing market-moving information.

A cloud before the cloud

Bloomberg rolled its own technology from the very beginning.

"Bloomberg always had a modern architecture," Edwards said. "Chuck Zegar and Tom Secunda, two of the founders along with Mike Bloomberg, they had built kind of a web model before the web."

First released in 1982 for Merrill Lynch, Bloomberg's centralized computing model in New York powered what we now know as "the Terminal," the source of the lion's share of Bloomberg's revenue and profit. The Terminal is basically the popular shorthand for Bloomberg Professional Services, the arm of the company that sources real-time trading data from stock exchanges around the world and feeds it to traders and investors desperate for an edge.

That original system was written in Fortran, a programming language that predates COBOL, but modified in C++ around the time Edwards joined the company 17 years ago, he said.

Around that time, Bloomberg made several other changes to its underlying infrastructure that would eventually become blueprints for future distribution computing applications. It built a new user interface in JavaScript that ran server-side with "a lightweight toolkit" running on the client side, years before node.js simplified the process of running JavaScript — the most popular programming language running the last eight years, according to Stack Overflow — on both the server side and client side of an application, Edwards said.

It also began to organize its applications around a service-oriented architecture, building its own middleware that was similar to an open-source called gRPC released by Google in 2015 that brought the concept of microservices to a larger audience.

And on the hardware side, Bloomberg made a big bet on OpenStack, which over the years has become an example of how not to run an open standards organization. OpenStack was a response to some of the cloud computing concepts pioneered by AWS, but designed for and by tech vendors catering to companies that thought they still wanted to manage their own data centers.

Over time, it became clear to a lot of those end users and vendors that nothing was going to stop AWS, and support for OpenStack fizzled. But it still provided a solid blueprint for companies like Bloomberg that had already invested a ton of money in data center infrastructure, and Bloomberg continues to run much of its operation on that combination of hardware and software design principles.

Teaching the machines

Today, Bloomberg is still operating much of the same technology but with a few modern flourishes here and there.

OpenStack can't solve all its needs, and Bloomberg does run "purpose-built dedicated hardware for things that require it," Edwards said, such as "real-time data that doesn't belong on [virtual machines]."

It doesn't actually build its own servers like the major cloud companies do, but it is very picky about the hardware it puts into its environment and tweaks the Linux kernel running on those servers around its unique needs. The company has "experimented" with special-purpose chips like FPGAs (field programmable gate arrays) and hardware accelerators, Edwards said, but for the most part relies on off-the-shelf hardware customized by its engineering team.

Bloomberg does use public cloud services for what Edwards called the "dot-com" parts of its business, such as the media properties like Bloomberg News. It also meets customers where they are; if Bloomberg customers are running their own servers in public clouds, the company has worked with AWS and Google to link its own infrastructure with the public cloud servers used by its customers, he said.

Edwards is currently focused on improving Bloomberg's use of machine-learning techniques to improve its services.

The company thinks it has one of the best optical-character recognition systems in the world, which allows it to pull text and objects like tables out of company filings and financial statements and present it in a readable format. It's using machine learning to help predict the price of a bond, which is more complicated than it might sound because bonds trade far less frequently than stocks, and it's also using these techniques to help traders prioritize incoming messages from clients and capitalize on trade opportunities.

"We're a big data company, but we don't have nearly the amount of bytes in storage that Instagram has. But we have an embarrassingly large amount of heterogeneous data sets," Edwards said. "So when we say we understand documents, it's because of the 30 years of putting this together."

Power

Yes, GameStop is a content moderation issue for Reddit

The same tools that can be used to build mass movements can be used by bad actors to manipulate the masses later on. Consider Reddit warned.

WallStreetBets' behavior may not be illegal. But that doesn't mean it's not a problem for Reddit.

Image: Omar Marques/Getty Images

The Redditors who are driving up the cost of GameStop stock just to pwn the hedge funds that bet on its demise may not be breaking the law. But this show of force by the subreddit r/WallStreetBets still represents a new and uncharted front in the evolution of content moderation on social media platforms.

In a statement to Protocol, a Reddit spokesperson said the company's site-wide policies "prohibit posting illegal content or soliciting or facilitating illegal transactions. We will review and cooperate with valid law enforcement investigations or actions as needed."

Keep Reading Show less
Issie Lapowsky
Issie Lapowsky (@issielapowsky) is a senior reporter at Protocol, covering the intersection of technology, politics, and national affairs. Previously, she was a senior writer at Wired, where she covered the 2016 election and the Facebook beat in its aftermath. Prior to that, Issie worked as a staff writer for Inc. magazine, writing about small business and entrepreneurship. She has also worked as an on-air contributor for CBS News and taught a graduate-level course at New York University’s Center for Publishing on how tech giants have affected publishing. Email Issie.
Protocol | Enterprise

SAP unveiled a big sales promo. It's a bid to juice cloud customer numbers.

The move is the culmination of CEO Christian Klein's efforts to turn around the German software giant.

SAP unveiled "RISE with SAP" on Wednesday.

Image: SAP

SAP CEO Christian Klein is trying out a major sales gambit in his attempt to get more customers onboard the software giant's signature cloud platform.

A new offer unveiled on Wednesday called "RISE with SAP" bundles together several products, including the flagship S/4 HANA platform, under one contract with a flat cost, a promotion that the company is hoping will encourage more users to more quickly switch from the on-premise services that dominated the company's product line until the last few years.

Keep Reading Show less
Joe Williams

Joe Williams is a senior reporter at Protocol covering enterprise software, including industry giants like Salesforce, Microsoft, IBM and Oracle. He previously covered emerging technology for Business Insider. Joe can be reached at JWilliams@Protocol.com. To share information confidentially, he can also be contacted on a non-work device via Signal (+1-309-265-6120) or JPW53189@protonmail.com.

About Protocol | Enterprise

‘It’s not OK’: Elastic takes aim at AWS, at the risk of major collateral damage

Elastic's long-running dispute with AWS entered a new chapter last week with big changes to two of its open-source projects. AWS now plans to take those projects under its wing.

"I don't know why this is surprising to people," Elastic CEO Shay Banon said in an interview with Protocol.

Photo: Michael Nagle/Getty Images

Fed up with what he sees as unfair competition from AWS, Elastic CEO Shay Banon felt he had no choice but to restrict the way third parties can use two important open-source projects developed by his company. Yet much of enterprise tech thinks he just threw the baby out with the bathwater.

Last Thursday, Elastic published a blog post — curiously titled "Doubling down on open, Part II" — announcing that Elasticsearch and Kibana, two widely used open-source projects in enterprise tech, would no longer be available under the permissive Apache 2.0 license. Instead, all subsequent releases to those projects will only be available under either a controversial new license known as the SSPL, or the Elastic License, both of which were designed to make it difficult for cloud companies to sell managed versions of the open-source projects they're applied to.

Keep Reading Show less
Tom Krazit

Tom Krazit ( @tomkrazit) is a senior reporter at Protocol, covering cloud computing and enterprise technology out of the Pacific Northwest. He has written and edited stories about the technology industry for almost two decades for publications such as IDG, CNET, paidContent, and GeekWire. He served as executive editor of Gigaom and Structure, and most recently produced a leading cloud computing newsletter called Mostly Cloudy.

Protocol | China

More women are joining China's tech elite, but 'Wolf Culture' isn't going away

It turns out getting rid of misogyny in Chinese tech isn't just a numbers game.

Chinese tech companies that claim to value female empowerment may act differently behind closed doors.

Photo: Qilai Shen/Getty Images

A woman we'll call Fan had heard about the men of Alibaba before she joined its high-profile affiliate about three years ago. Some of them were "greasy," she said, to use a Chinese term often describing middle-aged men with poor boundaries. Fan tells Protocol that lewd conversations were omnipresent at team meetings and private events, and even women would feel compelled to crack off-color jokes in front of the men. Some male supervisors treated younger female colleagues like personal assistants.

Within six months, despite the cachet the lucrative job carried, Fan wanted to quit.

Keep Reading Show less
Shen Lu

Shen Lu is a Reporter with Protocol | China. She has spent six years covering China from inside and outside its borders. Previously, she was a fellow at Asia Society's ChinaFile and a Beijing-based producer for CNN. Her writing has appeared in Foreign Policy, The New York Times and POLITICO, among other publications. Shen Lu is a founding member of Chinese Storytellers, a community serving and elevating Chinese professionals in the global media industry.

Protocol | Enterprise

January's Slack outage started with an AWS networking error

And problems with Slack's infrastructure meant things got far, far worse from there.

This sign was about as interactive as Slack's software on Jan. 4.

Photo: Justin Sullivan/Getty Images

The hours-long outage that kicked off the 2021 working year for Slack customers was the result of a cascading series of problems initially caused by network scaling issues at AWS, Protocol has learned.

According to a root-cause analysis that Slack distributed to customers last week, "around 6:00 a.m. PST we began to experience packet loss between servers caused by a routing problem between network boundaries on the network of our cloud provider." A source familiar with the issue confirmed that AWS Transit Gateway did not scale fast enough to accommodate the spike in demand for Slack's service the morning of Jan. 4, coming off the holiday break.

Keep Reading Show less
Tom Krazit

Tom Krazit ( @tomkrazit) is a senior reporter at Protocol, covering cloud computing and enterprise technology out of the Pacific Northwest. He has written and edited stories about the technology industry for almost two decades for publications such as IDG, CNET, paidContent, and GeekWire. He served as executive editor of Gigaom and Structure, and most recently produced a leading cloud computing newsletter called Mostly Cloudy.

Latest Stories