Today an article with a headline like “CRAMMING MORE Components Onto Integrated Circuits” would probably not do much for you as far as click baits go. But this was the title of Gordon Moore’s article, published in the Apr. 19, 1965 issue of Electronics magazine – 50 years ago this month–that introduced the world to a singular, shape-changing idea that would later become known as Moore’s Law.
Moore begins his article thus: “The future of integrated electronics is the future of electronics itself. The advantages of integration will bring about a proliferation of electronics, pushing this science into many new areas. Integrated circuits will lead to such wonders as home computers–or at least terminals connected to a central computer–automatic controls for automobiles, and personal portable communications equipment. The electronic wristwatch needs only a display to be feasible today.”
The point Moore was trying to make was that semiconductor chips, invented just a few years before, were improving at a mind boggling pace. He tried to plot these price and performance points on regular graph paper, but the slope was too steep, so he switched to logarithmic graph paper. He got a nice straight and shallow line. It showed that the performance of these chips was doubling every few years (18 months at the time, 24 now). It was undoubtedly impressive, but no one then–including Moore himself–realized this graph would continue to hold for 50 years and set the blistering pace of change for the modern world. We all now live in the world of Moore’s Law, and we likely will for at least another quarter-century.
As a result, data warehouses aren’t just bigger than a generation ago; they’re faster, support new data types, serve a wider range of business-critical functions, and are capable of providing actionable insights to anyone in the enterprise at any time or place. All of which makes the modern data warehouse more important than ever to business agility, innovation, and competitive advantage.
An interesting whitepaper from Oracle highlights and quite succinctly might I add, the top trends and opportunities in data warehousing. The following are the trends that are a direct result of Moore’s Law:
In-memory technologies supercharge data warehouse performance
Making the most of big data means quickly acquiring and analyzing a high volume of data generated in many different formats. All warehouse data used to be stored on magnetic disks. Now that data is being moved into RAM to achieve performance improvements that are orders of magnitude faster than previous methods. Thanks to new database capabilities, the entire data warehouse doesn’t need to be placed in-memory at the same time, enabling even the largest data warehouses to gain in-memory performance benefits. Data warehouse administrators can configure these environments to optimize data among RAM, Flash, and disk-based access methods based on heuristic access patterns. Important or frequently accessed data resides in RAM where processing becomes instantaneous. This lightning-fast database processing can entirely eliminate the need to create analytics indexes.
All that power opens up new analytics opportunities. For example, a restaurant chain might want to generate a daily consumption report to do dynamic spot promotions the following day. A bank might want to monitor unusual purchase patterns in real time to minimize loss from fraudulent credit card transactions.
On-demand sandbox analytics environments meet rising demand for rapid prototyping and information discovery
Business intelligence and analytics are resource-intensive activities that lend themselves to on-demand computing due to their iterative nature and fluctuating workloads. Forward-looking organizations are establishing analytics as a service (AaaS) environments within public and private clouds. These versatile “sandbox” environments can flex with a shifting volume and velocity of data, making them ideal for analyzing energy usage, monitoring shop floor operations, gauging consumer sentiment, and undertaking many other large-scale analytics challenges. Cutting-edge technologies, such as multitenant databases, enable organizations to set up one cloud environment with dozens or even hundreds of “pluggable” databases for people to use. Some leading companies are even monetizing their AaaS environments by setting up subscriber networks for other companies in their supply chains. On-demand provisioning makes it easy for business users both inside and outside the enterprise to utilize these environments as they respond to high-velocity demands of big data analysis.The “datafication” of the enterprise spawns more-capable data warehouses
Historically, data warehouses were populated with structured business data from enterprise applications. Today, however, data is pouring in from human-generated, cloud-generated, and machine-generated sources—more than 90 percent of which is unstructured. This data can be collected not only from computers, but also from billions of mobile phones, tens of billions of social media posts, and an ever-expanding array of networked sensors from cars, utility meters, shipping containers, shop floor equipment, point-of-sale terminals, and many other sources. With the help of new big data technologies, data warehouses are expanding in variety and scope. This improves the quality and speed of business decision-making as people learn how to acquire, organize, and analyze this massive influx of information.
For example, manufacturing companies commonly embed sensors in their machinery to monitor usage patterns, predict maintenance problems, and enhance production quality. Studying the data streaming from these sensors allows them to improve their products and devise more-accurate service cycles. CIOs are responding to an increased use of data in the enterprise by deploying newer, faster, and more-capable information management systems.
My Take
Storage and performance are no longer a limiting factor. A company with a half decent BI budget could create a DW monstrosity if it wished to. However, there is a tipping point and it is coming. We’ve grown so accustomed to living in the world of Moore’s Law that we forget we’re dealing with one of the most explosive forces in history. We’ve become so adept at predicting, incorporating and assimilating each new, upward tick of the curve that we assume we have this monster under control. We don’t.
References:
