The Big Story

It's time for real-time

Snowflake's smash-hit IPO was validation that non-hyperscalers can thrive as standalone vendors for data analytics. Confluent's Thursday debut was more subdued, but the 25% first-day rise in its shares still sent a clear signal.

The message? Enterprise data strategies are poised to get more sophisticated. Confluent, which runs on Apache Kafka, touts its ability to let users constantly compile and analyze data from Twitter, mobile apps, website clicks and many other sources — what it calls "data in motion." It's part of a burgeoning field known broadly as real-time analytics, one that is already seeing notable interest from investors.

The rise of real-time analytics could have a major impact on the future of database architecture. Advocates of real-time analytics say it can process data in seconds. That's much faster than the data warehouses that companies have used for decades, and it opens up different real-world uses.

• "There's a ton of excitement around modern cloud data systems, [and] what that unlocks for companies," Confluent CEO Jay Kreps told Protocol. "When you look at most of the other data companies, they're about storage. That gives us unique differentiation … and it's a key aspect of how companies connect all this up."
• Investors are taking note. Amplitude has raised $337 million. Rockset and Imply have raised $62 million and $115 million, respectively. And then there's also more established players like MongoDB and DataStax.

While a few minutes may not seem like a huge deal, it shows a growing divide within the industry. Customers need to decide whether they want to spend money on software for specific use cases versus a single platform that looks to meet every worker's demands.

• Increasingly, it's not CIOs who are making purchasing decisions — at least, for smaller buys. Instead, line of business leaders have more freedom to adopt the tools that address their team's specific needs. Any enterprise-wide purchase, however, still involves top technologists, according to CIOs and executives at software vendors.
• That's in part because of the simplicity of many of the new systems. But it's also a testament to the varying needs of roles within organizations. Analysts that need to pull daily reports on the status of the business, for example, don't need that instant processing that vendors like Confluent offer. Marketing chiefs, however, who want up-to-the-minute updates on campaigns, could see value in adopting real-time analytics.
• As enterprises mature in their data strategies, these are the kinds of decisions that leaders will grapple with. And given the relatively nascent state of tech, it's still an open question what the architecture of the future will look like.
• "It's a delicate balance," Linh Lam, CIO for ICE Mortgage Technology, told Protocol. "Real-time analytics can be extremely powerful, but it can be extremely detrimental if there isn't integrity behind the data that's being presented."

These are not just internal shifts. The rise of real-time analytics could have an impact on providers like Snowflake and Tableau.

• Instead of running everything on Snowflake or Google Cloud's BigQuery, for example, real-time analytic platforms give enterprises more leeway to run specialized queries on engines designed specifically for those workloads.
• That could help organizations avoid getting massive data warehouse bills, according to Amplitude and other providers.
• "As companies use our product, it typically frees up the data scientist to be able to work on other use cases. It's not that they are trying to reduce their spend on Snowflake, but they are optimizing it more," said Amplitude's Justin Bauer, senior vice president of product. "If you try to do everything [in Snowflake], it's going to get very expensive."
• Snowflake has sought to address this complaint, including banking on its closed architecture to help drive cost savings for customers. And if more companies run more operational analyses in real-time systems, that could ultimately lower enterprise reliance on visualization tools like Tableau and PowerBI.
• "They're going to need to respond in some way. Or they are just going to be part of that reporting layer, which may not be as high-growth for them," said Bauer. "There's a reason why I'm at Amplitude."

For now, competition and cooperation go hand in hand. Confluent has partnerships and go-to-market agreements with Snowflake and Google Cloud, among others. It was even voted one of Google Cloud's top partners.

• But it's very early days. And IT dollars, even when considering the broader purchasing landscape, aren't infinite.
• Eventually, enterprises are going to consolidate down to fewer vendors or interweave tech in a cost-advantageous way. That is going to mean some providers lose. Still, many in the industry will say there's enough business to go around.

History has shown that major shifts in technology can be a huge advantage for the upstarts. The data analytics industry seems to be proving that yet again.

— Joe Williams

A MESSAGE FROM MICRON

Recently, Micron announced new memory and storage innovations across its portfolio based on its industry-leading 176-layer NAND and 1α (1-alpha) DRAM technology. But what does "1α" mean, and just how amazing is it?

Learn more

This Week On Protocol

Salesforce's equality head retired: Tony Prophet, who also led recruiting for the software giant, had been on medical leave for the past few months. His departure, which was reportedly announced earlier this month in a Slack message from HR chief Brent Hyder, comes after Salesforce's simmering DEI issues were thrust into the public spotlight when two Black female managers posted resignation letters on social media sites alleging a culture of microaggressions and inequitable treatment at Salesforce.

Five Questions For...

Brooke Wenig, machine learning practice lead, Databricks

Wenig was on our recent list of 10 people defining the new database landscape. Read the whole list here.

What excites you the most about the future of the database industry?

I'm continually amazed by the seemingly endless creativity of the community. The novel applications that we see today are possible thanks to the explosion of open-source tools, online courses and tutorials, and the accessibility of data sets. This has democratized the practice of working with big data and leveled the playing field for everyone.

What's your biggest career mistake or learning lesson?

I once designed the wrong product; I was trying to develop a classifier for a new application, and I thought I had all the information I needed and knew what I was doing. I couldn't have been more wrong. I learned several key lessons about cross-team communication and scoping projects: focus on the business value over technical minutia; iterate frequently with stakeholders and ensure you have direct access to them to avoid playing broken telephone; and ask questions about the data and validate any key assumptions with SMEs.

What's your advice to younger technologists who want to build a career in this field?

Just do it! Don't be intimidated by the hype. It's an ever-changing field and no one is fully caught up with all the latest advancements, so don't feel like you're starting late or behind others. Just jump in and work on a project that interests you. It doesn't matter if you're using the latest and greatest tools. The important thing is that you're solving a problem that you're passionate about and you're learning along the way. I highly recommend checking out kaggle.com to find fun data sets, form a team with friends and enter competitions to motivate you in your data journey.

What's one piece of recommended reading that you think should be a requirement for those in the industry?

I highly recommend reading the Datasheets for Datasets paper by [Timnit] Gebru et al. Every company tries to extract value from their data, but it's more difficult when there is no documentation about the data, such as how the data was collected, which fields can be null, how the data should (and should not) be used, etc. By creating "datasheets" for your data, you'll improve the quality of your data and any downstream applications.

What's the biggest hurdle companies are going to face in becoming a data-driven enterprise?

A lot of people focus on the technical aspects, but aligning people and the processes are just as important. Often data teams in large corporations operate in silos, and thus their data also exists in silos. Breaking down these silos is critical to becoming a data-driven enterprise, as the right people need access to the right data at the right time to make the right decisions.

Around the Enterprise

• Salesforce and Zendesk are voicing concerns to the FTC about Facebook's $1 billion Kustomer deal, which pushes it into CRM, The Information reports.
• AWS is buying encrypted messaging app Wickr for an undisclosed sum. The cloud provider reportedly considered buying Signal first, per Business Insider.
• Microsoft's Windows update is a reminder of the glory days when the company, not Apple or Amazon, was the subject of Washington's antitrust furor, reports Bloomberg.
• Rockwell Automation is buying Plex Systems from investment firm Francisco Partners for $2.2 billion, a move to boost the industrial software provider's cloud-based portfolio.

A MESSAGE FROM MICRON

Recently, Micron announced new memory and storage innovations across its portfolio based on its industry-leading 176-layer NAND and 1α (1-alpha) DRAM technology. But what does "1α" mean, and just how amazing is it?

Learn more