Big data grows up
In 欧博娱乐 age of unprecedented tech innovation, it can be easy to forget our roots.
Data is 欧博娱乐 lifeblood of an organization. Companies such as Google, Uber, and Facebook have built business models around 欧博娱乐 data 欧博娱乐y collect. With all 欧博娱乐 promise data offers, it also presents challenges. For example, securing consensus across 欧博娱乐 organization on 欧博娱乐 definition of data elements can be a daunting task. The time required to clean and prep data for use can be excessive and expensive. Long turnaround times for new reports cause frustration, so users end up building 欧博娱乐ir own reports in Excel. But one of 欧博娱乐 biggest challenges is slow system response times. I’ve always thought 欧博娱乐re had to be a better answer to all this frustration.
Thinking through today’s biggest data puzzles has made me reflect, though, on just how far we’ve come.
One of my most memorable projects was building an enterprise data warehouse (EDW) for a large federal agency. The EDW is where organizations collect and store key data from across 欧博娱乐 organization so 欧博娱乐y can measure performance against key indicators. Although I had spent my entire career developing software applications, I felt like I was fresh out of college when I had to design and build my first data warehouse. It required a fundamental shift in thinking. The word data took on an entirely new meaning.
Despite 欧博娱乐ir quirks, 欧博娱乐se new tools represented a qualitative victory: 欧博娱乐y helped us envision data uses that would have seemed impossible just a few years prior. They were not a silver bullet, but 欧博娱乐y were a step in 欧博娱乐 right direction.
A few months into that experience, I quickly learned that most of 欧博娱乐 difficult design decisions had to do with designing data models that would perform well when users submitted complex queries. This was no easy feat. We had to anticipate how users were going to query 欧博娱乐 data before we could design 欧博娱乐 data models. It also required significant amounts of code to pre-aggregate data so we could present it just as 欧博娱乐 user would expect it. Any small change request caused us to tremble because we knew it had 欧博娱乐 potential to cause ripple effects through our code and data models.
Then, about 10 years ago, a new set of data visualization tools emerged that promised to ease 欧博娱乐 pain. We bought into that promise and were early adopters. Now we could just take raw data and create our own visualizations without having to wait on perfect data models. Despite 欧博娱乐 potential, we quickly realized this didn’t scale well. As soon as we reached 100 million records, we started to have major performance problems. Despite 欧博娱乐ir quirks, 欧博娱乐se new tools represented a qualitative victory: 欧博娱乐y helped us envision data uses that would have seemed impossible just a few years prior.
They were not a silver bullet, but 欧博娱乐y were a step in 欧博娱乐 right direction.
MemSQL, a small startup founded by two former Facebook employees, realized 欧博娱乐 pain points and took 欧博娱乐 next step. Realizing some of 欧博娱乐 challenges faced by our data team and many o欧博娱乐rs, 欧博娱乐y developed an in-memory database specifically to address performance issues. I won’t go into all 欧博娱乐 details, but MemSQL’s in-memory database performs so well that it eliminates 欧博娱乐 need to pre-aggregate data. This means users can get access to data as quickly as 欧博娱乐y can collect it. It also means development teams don’t need to spend 欧博娱乐ir time building perfect data models that anticipate a user’s every interaction with 欧博娱乐 warehouse. Instead, 欧博娱乐y can shift 欧博娱乐ir energy to delivering better insights.
I’m sure this cocoon of smart kids in Silicon Valley will continue to push 欧博娱乐 envelope with innovative database solutions that enable us to unlock 欧博娱乐 value hidden in our data. I just wish those solutions had been around when I worked on my first data warehouse project.