Quarterly Q – April 2018

Hello READER,

Hope everyone had a successful 1^st quarter.  As Q2 kicks off, wanted to share more insight on a newer capability we’re hearing more and more about – Cloud Data Warehousing.  As we all know, infrastructure is starting to move into the cloud at a very fast rate and Data Warehousing is becoming one of those components that companies are honing in on.  Wanted to introduce our friend Frank Bell. Frank runs IT Strategists, a consulting firm helping teams deliver data solutions to their organizations, which include Disney, Ticketmaster, Nissan, Toyota, USAF and Unilever.  He’s very excited to share his experience and insight on this topic with all of us.

Snowflake – Cloud Data Warehousing Revolution

It’s the age of disruption and companies must be agile and data-driven, or it’s quite certain in due time they will be disrupted and replaced.  Of course, this scenario is very scary, but it’s also an exciting time.

As most of us can attest to, data problems and inefficiencies are prevalent within all of our organizations in some capacity.  While Data Warehousing and even Big Data solutions have been around awhile including Teradata, Netezza, Vertica, Impala, Presto, Redshift, and Hadoop, these technologies are very complex to integrate and yet still lead to scaling challenges, slow implementation efforts, remain very costly and none offer sustainable flexibility.

From our experience, most of the organizations we survey, assess, and work with have a ton of data and managing it is getting more and more complex.  Companies are always on the prowl to achieve access to data faster and faster to increase analysis and automation value.  Slow moving technology executives and teams are constantly getting side-stepped from marketing and operations’ teams which are moving data to their own cloud silos.  Data complexity is growing rapidly and companies are experiencing many problems including:

• Data speed.  Data loading for many businesses is still batch driven and often takes hours and sometimes even days.  Modern businesses just cannot wait this long to analyze and drive automation.
• Data concurrency problems.  Business users often cannot access the latest data fast enough or have to wait until loading is done.
• Data sources are more numerous and varied.  (Not just traditional rows/columns but JSON, AVRO, Parquet, etc.)
• Data is almost always in silos and cannot be cross-referenced.
• Data access is too often complex.

In addition, data security is a huge concern as breaches continue to increase across the ecosystem.

One of the tools that we feel is imperative to combat these complexities is Snowflake.  First off, Snowflake is easy to use, very fast and handles concurrency issues & limitations effortlessly.  Some of the other efficiencies that Snowflake brings to the table includes:

• SQL is the most common technical language used.  It’s relatively easy for even business users to pick up versus learning new syntax and languages.
• Being able to easily load, query, and relate JSON, XML, Parquet, and other sources with relational data make analysis much faster.
• It allows the creation of entire clones of production in seconds.  No more waiting hours to duplicate content.  This is amazingly efficient for QA and Data Quality.
• Security is now taken care of for you with security experts.
• Time Travel eliminates the needs for costly and complex backup operations.  You can even query your previous data table(s) down to the millisecond.
• Separating Compute and Storage opens up major innovations not available before.
• Since compute can now be separated, organizations can have isolated workloads for data loading, marketing, operations, data scientists, etc. etc.
• Paying only for what you use. Now you can effectively size your costs for your workload when you need it. No longer do you have to buy hardware to scale for the maximum use cases.
• Going to 1/10th the cost of database administration is amazing for TCO. All that expertise you had with indexing, vacuuming, etc. is no longer needed to pay for. It comes as part of it.

We have seen very positive results with Snowflake implementations in a very short amount of time including:

• 78% cost savings replacing on-prem data warehouse and Hadoop.
• Implementation time goes from months or years to weeks.
• ETLs adjust from days to hours or even minutes.

by Frank Bell
Big Data Principal | IT Strategists | www.ITStrategists.com