Wave Analytics is great

Salesforce just announced Analytics Wave, the analytics tool that is not only for the common user, but also for the common smartphone user, which makes up pretty much all of us. There have been many inputs from the market of analytics tools that claim to be a tool “for the rest of us”, so I fear reporting this fact again. But in reality, there are none that I know of that (a) I can use, (b) I can find my own information on and (c) that I can use on my iPhone.

This has pretty much revolutionized the industry for analytics and is something that will take the industry by storm. Why? Because as soon as you have answered one question, most users will have a follow up question to ask. If I now know, which region is selling the most in my area, I will suddenly want to know more. Which products are selling better? Where to I personally make the most money? Are the margins the same across all of my products?

With Analytics Wave, the user can interactively model the data any way she wants it and at any time.

Here is the article that I learned this from: http://techcrunch.com/2015/02/19/salesforce-analytics-tool-waves-hello-to-mobile/

Human Decisions: the real reason for Big Data

It is true and well stated by Joe Hellerstein, CEO of Trifacta. We find ourselves often in love with technology and the development of something and lose site of its real intent. Same is happening with Big Data and the statement that some are making that technology will eventually eliminate the need for a Data Scientist.

Let's remember that humans make qualitative decisions out of the quantitative data that is provided to them, ever faster and more complete by Big Data technology (review article written on this blog). Within the data research process that ends with enough data to make a decision, there is a constant analytical process that asks how can I get to what I want to know using the data I have. The data scientist will always be in control of that process, no matter how much we automate it.

This was well described in Joe's article.

Integrity in Business - a thing of the past? Or do we all aspire to be a "Wolf of Wall Street"?

It's hard not to want what the Wolf has. He has it all. He has money, friends, drugs, wild parties where prostitutes are used like the salsa bowl. And, as the movie painted for us, he never really had to feel bad - not even a hangover the next day! He never felt bad about the investments he misappropriated, of the lies he told to those who were trying to invest their money. Most of his clients lost everything, but the Wolf didn't lose a thing.

And when we hoped our laws would put such scum where he belonged, he ratted out his closest associates and only got three years in a celebrity jail cell. The last scene shows that he got out and is a celebrated author, speaker and probably got a % of the $12 of my movie ticket. What a hero!

My values are different. I am a consultant who does not lie to his clients. I don't lie to my employer. I don't lie while I am selling something. I don't lie to my wife of 26 years and have always chosen to come back to my family. Martin Scorsese would have me believe I screwed up and have missed out in life. This movie is heralded by my society as a great movie that will win many awards.

Am I wrong?
Do we have to lie to clients to get ahead as a consultant?
In order to sell something do I have to lie?

If I do, then I quit and accept my success as the little that those get who believe integrity has value...and I will be happy with it.

The Focal Point of Data Integration: Did Salesforce Get It Right?

We've all been brought up believing that the focus of data integration is a back office function and that this process should be something that is completed by large ERP and CRM applications deep in the bowels of the IT department. All ERP vendors promote their own add-ons that can facilitate the data integration process.

The problem with data integration is not the transfer of data, which has to go through a migration process and follows a rigorous MDM (master data management) exercise to determine a unified enterprise-wide data schema. No, the problem lies in determining the final functionality and higher level business goals that need to be reached after the integration has been completed.

Traditional integration projects worry about ensuring that the back office systems are integrated and are based on sometimes justifiable needs to share data between systems. But the focal point of determining what data needs to be integrated from which back office systems has to happen at the customer level, in the front office to ensure that the business process flows correctly.

This is where Salesforce.com has it right. A company should determine what their sales people, customers or partners need on a smart phone or tablet to ensure that the business process flows faster and more efficiently. Based on that need, an integration project needs to be completed. The focal point of all data integration needs to be in the front office as SFDC proposes, not in the back office where we usually find it.

Last Night: I was invited to Advanced Screening of The Hobbit IMAX in 3D!

This was an amazing play on the 3D effects that only come to life when viewed in full IMAX technology. I have never seen anything like it and it gets a full 2 thumbs up!  It was simply amazing. This is as much a roller coaster ride as I have ever experience even in real life. The 3D makes this movie amazing, so that the experience is more than you physically fathom and I walked out drained of energy.

Go and see this, but don't waste your time with 2D or anything less than IMAX!

2013 Dreamforce: Lots to learn from results

I was unable to attend Dreamforce, which is the major event put on by Salesforce.com, but there were a lot of comments that could boil down some of the lessons for many of us who weren't there.

On this blog, I have spoken much about the Cloud transition and how the datacenter of the future will be in the cloud. Many companies offer real cloud solutions for companies ready to go, the leader in this industry is AWS. But for the mainstream customer with a data center the transition to the cloud is a major disruption and until things break, they are unwilling to move.

The other offering is the traditional SaaS model that companies can transition to, where a major business process transition that involves the use of a new SW package in the cloud can make the same transition. Here, there is often no choice. If the functionality you are searching for is in the cloud, then you will move to the cloud. 

In the case of Salesforce.com, many companies are finding the best of both worlds. Yes, following an improved sales strategy with a CRM model will lead you to their core CRM package and therefore to the cloud, since SFDC (=Salesforce.com) does not offer any other model. But what they have been able to create is a cloud based platform that can take the core of your customer data and expand it to include all other data your company needs or will need from here. SFDC can run your company's cloud transition completely from soup to nuts. Yes, AWS offers the cheapest computing and storage option and many different SW packages are becoming available in a SaaS model. But no where will you find the complete data landscape package that will allow you a transition on your time, piecemeal or complete, as you can with SFDC.

Here are some choice links that I learned a lot from:

Many new ideas
Idea is the customer
Take aways from Marissa Meyers

PaaS the Fastest Growing Cloud Segment

This is something I have written about before, but it seems that the market has finally caught on: PaaS will be the dominant segment throughout 2014. For more information on exactly what the market is and to understand what PaaS means, read this.

Why PaaS? Because the easiest business case for cloud adoption is the area of Software development and operations, called DevOps. The business case is simple and ask anyone who has ever run a data center why. Whenever a new version or release of a piece of code was ready for functional/regression/production test, there needed to be a new environment created for that test. The datacenter guys had to find servers, implement servers, install servers and create a complete environment to complete the test. Once the test was done, the hardware needed to stay untouched, so that a new test (yes, they did find bugs) could be implemented quicker.

The problem here is not only unused hardware and wasted hardware set up skills, but the worst was the time it takes to get a new release out the door.

Cloud and especially PaaS promises to change that forever. Users (SW development guys) can set up an environment on their own as a cloud account, test, and then only use the hardware environment when they need it. As soon as they don't need it, the resources no longer cost money. DevOps is the easiest to understand use of Cloud computing.

Is Risk Really Holding Back Big Data Projects?

In his WSJ article titled "The Risks of Big Data for Companies", Dr. Jordan explains a number of risks that companies should be aware of before jumping into this new and unchartered market.

He outlines that too much data could be dangerous for companies who might invoke inner company politics or make unjustified decisions too early with the results they make. I would imagine there are no risks in ignoring information that the data gives us and just keep moving along as a horse with blinders.

WHAT? If people seriously took the advice of this column in the WSJ, we would never advance in business. More importantly the #1 risk that is keeping up every CEO is not whether a plant in Canada has better output than the same in the US, thusly creating an "inner company rivalry". No, the #1 threat is the competition. The competition can close down your market and make your company's revenue tumble.

Listening to this sort of advice is not only dangerous, but plainly stupid. No company that I have helped with Big Data projects has decided to throw out human intuition and experience for decisions made based on the data alone. Instead all functioning companies want to see whether the data can complement decision making. That is what a pilot will show and the ensuing business case will prove it, or not. Pilot programs should be done with Big Data in small areas to show what we could know, without fear that we will uncover some terrible new insights that could cause inner company conflict.

If you are starting a new Big Data project, please do NOT read this article. Instead follow your own intuition and call me in as an adviser.

Big Data Projects: Who is the Organization's Big Data Leader

Deloitte Touche Tohmatsu Ltd.

The big question for any consulting organization trying sell Big Data projects is to find the Big Data leader in that organization. There is always one person in every organization who has spurred the company you are selling into to use the massive amounts of data that they have access to to create solid, decision driving data.

Who is that person? Is she the CIO? or is he the CMO? or could she be a Director-level person with a special charter? Tom Davenport, the Data Analytics Guru from Harvard discusses this well within his Wall Street Journal blog. You can read it here. Tom's opinion is pushing for a decision within a corporate organization for where the Big Data Leader should be. I agree with his overall complaint from the Deloitte poll shown below that the leader should be someone who works enterprise-wide, so that an overall Big Data strategy can evolve. In a perfect world that would be great.

But I think reality is different in the companies I have insight to when it comes to their Big Data adoption. In reality, a Big Data project comes out of a need to know more about your customers than your competitors know. It has to start with a dream somewhere in the organization that says, if we had this piece of information, we could sell more, grow market share or save the cost of creating products no one wants. This need could develop in an IT organization, but only if their goals are clearly aligned with the business goals, which sadly is not often the case. Most of the times, I have seen this need arise was in the business unit level, where goals are clearly tied to the success of selling products.

If 64% plan a Big Data project, why are so few IT consulting companies prepared?

-->

According to a Gartner survey, 64% of the companies surveyed plan to invest in some sort of Big Data project in 2013. Many of those surveyed will run some sort of pilot project, not investing in hardware as much as investing time with a singular goal: where will Big Data payoff? how much needs to be invested to get which results?

Despite this strong opportunity, few IT consulting organizations have strengthened their ranks with specialist who can guide their clients through the uncertainty of this new market. One indicator is the lack of career positions posted for Big Data sales specialists and consultants who can create a solid business assessment that is strong enough to justify further funding. There are some who are not only specialized in this area, but also host cloud environments, where their clients can start a real pilot without any data center investments. It seems that these few will do well in the upcoming boom as others scramble for Big Data market share.

Using Video within a Big Data Query for IVS

What is the real value to digital video in a Big Data query? Should video even be a consideration for an other source of data?

Most Big Data applications are out for one simple thing: Getting decision driving information from multiple, disparate, structured and unstructured data sources. If all we had to do is query our RDBMs to get to good decisions, then Big Data as a market would not exist. But it isn't and things like weblogs and twitter feeds - all examples for large sets of unstructured data, are extremely useful for data scientists to use to find decision driving data.

Well, what about video? Can digital video be useful for a query? Early indications are that it is and the more you think about it, the more you find good, solid use cases in the industry. IVS (intelligent video surveillance) is clearly an early adopter to use video. The more that intelligent, digital computer vision algorithms can be combined with queries of other databases to form a real-time, interactive query process, the more intelligible and actionable information we can glean from the query. Yes, there are computer vision algorithms that can search through a digital video file and detect anomalies. But despite the current speed, can we combine that search with a recognition database, like a facial recognition and give back intelligible information. What about height, speed, color, make and model or even an analysis of behavior?

What if an IVS system could automate a query that someone has entered a safe perimeter, that he is recognized as a known person on the terrorist watch list and is spending time cutting through a fence? What security level would this anomaly trip to get action at the highest level. Anyone who claims that any anomaly should trigger highest level does not understand the IVS market. After three or four false alarms, most companies turn off their detection and revert to good, old-fashioned human visual detection.

Big Data has this promise, but video is not an integral part of the query capabilities yet. There is nothing on the market today that allows companies to bring a digital video feed in and query it using SQL like any other data. But that could soon change (stay tuned).

Can we automatically index UGC?

Here is one of the largest dilemas under the current Big Data sun: will be ever be able to automatically index user generated video content (UGC)? Seems like a an impossibility, but would have great benefits if possible.

Why would we want to create automatic indices and meta data for the uploaded content? Well for one, it would make the mass of user created content that is uploaded to sites like YouTube searchable. Making this mass of video content searchable means we can find the content we are interested in, saving hours of sifting through content that is only saved by the good graces of someone who titled the video correctly or added descriptive meta data to their uploads.

Video is a content rich source of data, since we humans perceive much more data from video than could ever be tagged. But an auto-tagging system could learn over time to get better using facial and behavioral recognition patterns to improve accuracy. This would be an immediate and significant improvement over the voluntary methods we currently use.

The basis for starting automated indexing would be to bring the digital video stream to a standard format where it could be combined with other data sources and included in a standard SQL search. That is now possible, as you can see by my previous posts.

Why is transfer of lots-of-data an issue for Big Data?

This may just be a rhetorical question, but there are a few companies developing technologies that allow a massive amount of data to be transported to a DW environment where - using the latest Big Data tools - this data can be analyzed and further used within larger queries. 

One such company is Attunity (www.attunity.com) and their latest product blog post states clearly there solution allows data to boldly go where and when you need it.

What I don't understand is this: Wasn't that the original use for MapReduce that data did not have to reside in a singular DW so that I could query it? As I understand it, this was Google's motivation to build this technology since they clearly understood that all data of the web would not be located on one spot.

According to a paper from TerraData there is a growing issue in the amount of data produced by oil wells and getting that data transferred across the country or countries to a data center is expensive. I guess my simplistic way of design is to ask: Why not put together a couple of servers at the drilling site, build a Hadoop cluster remotely onsite and remotely run queries on it where it is? Isn't it harder to try to store, transport and restore the massive amount of data?

Just a question. Please comment.

Breakthrough: Hadoop-based Video Analytics

Pivotal, a company formed from EMC and VMware has created a breakthrough for video analysis. Dr. Victor Fang from their Big Data Science Lab has a demo where he chews through and analyses 5 GB of MPEG-2 video using Pivotal HD and HAWQ in near real-time. His output are exception images, place of movement, speed and many other parameters of the detection.

Why is this big? It will revolutionize security industry's Intelligent Video System market because it will learn to recognize exceptions and feed these to security personnel. Now, the system will intelligently provide information to the monitoring team, rather than hoping to detect with parameter movement algorithms, which have yet to prove themselves.

Watch this video:
http://www.gopivotal.com/resources#http://bitcast-a.v1.sjc1.bitgravity.com/greenplum/pivotalvideo/Unstructured_Data_Video_Analytics_on_Hadoop_Session_1.m4v

What are good Big Data opportunities that Startups should pursue

The question has come up in a great Quora forum, which I highly recommend and should get a number of members in this group excited about either joining or founding a startup. Most companies I speak with do not have the capability to analyze data like they could using the Big Data tools, the time difference between payoff and investment is far too long to warrant this type of investment.

So, a huge startup opportunity is in providing the information these companies would like to have - and based on some of these ideas, large corporations are willing to spend a pretty penny on these.

The four areas of that Startups could make money using Big Data that I find most promising are:

1.) Recruiting: this industry is still suffering from poor conclusions from unstructured data
2.) Online Video Processing: much could be machine recognized to enable indexing
3.) Knowledge Management: still a fleeting target for many companies
4.) Store Location Information: the right mix of parameters could be gold for some

Any others? Comments?

Here the link to the Quora forum: http://www.quora.com/What-Big-Data-opportunities-will-be-most-interesting-and-profitable-for-startups

"We don't really buy hardware anymore,"

This quote from a Louisiana based healthcare staffing company. Five years ago, they would buy about 5,000 servers a year and now this year only bought one from HP. (see details in article below). This is a significant change that most IT workers out there had better listen to intently.

Who does this trend not effect? It does not effect the server makers as a total, since computing power is still going to be needed in the world. But it effects the data center workers who have just learned how to get everything working with their server farm. This will be a museum piece as more and more companies will put their workloads out to the cloud.

It is time for the IT department to start a plan to transition out of their blissful life of being the most powerful organization in the company to being an enabler of information flow with a budget of 50%.

CIOs who do not have a 5 year plan to decrease their budget by 50% and increase the IT functionality by 300% need to be replaced with those thinking outside of the box, who can efficiently meld the trends of this transition with the business needs of their company. This along with Big Data are going to be the greatest transitions we have seen in IT since the implementation of RDBMs. Like so many trends, many will be caught in the middle, unaware that things would change so quickly.

If there is anything we should learn from the trends of the past, everything will change much quicker than we thought it would. You can bet your career on it!

FT states it well: Big changes over the next five years

Please read this article:

Tech executives facing up to hard realities of the cloud

The FT columnist Richard Waters is finally describing the pink elephant in the board room of so many tech giants around the world. I have stated it publicly and will do so again here:

In the next five years we will see IT increasing its functionality by 300% while decreasing its budget by 50%.

The proof is in the pudding or better in the cloud. Countless executives from large IT users such as DHL have said it again and again that they want to buy computing services in the same way they currently buy power and water. DHL has stated again and again that they want to look at the cost of shipping a package as 100% and know exactly that the IT burden on that cost is 8%. I just made up that number.

But as you can imagine in today's marketplace this is not possible. Servers and mainframes bought years ago are going on refresh and enterprise applications are bloating the need for more storage. Yes, after the financial year is over, DHL can go through and make its calculations, but they are far too high and uncontrollable in a cycle such as this. They want what they get from the power company: if they ship a lot this month, they will pay a lot, but if they ship little, their bill should go down in its reflection.

Amazon gets it - maybe not through their ingenious foresight, but since they were in the right place to rent server capacity. Big server manufacturers, including the one I work for, do not.

Big Data Easy Start: Amazon Elastic MapReduce (Amazon EMR)

Amazon Web Services has a product/service that could make it easy to break into the world of Big Data in a real way. With Elastic MapReduce, any company can use their Hadoop framework and start building data-intensive tasks for applications without having to build the underlying infrastructure or tuning the Hadoop clusters. Not that tuning is impossible and products from Hortonworks make it all work easier, but you would need to build a lot of structure before you ever really see the results. This seems like a clearly easy way to pay your $90 and get full access to infrastructure and start web indexing, analyzing log files, mining data for results, traditional data warehousing, artificial intelligence through machine learning, financial analysis, biometrics research or even scientific simulation.

For many companies the start into Big Data is expensive, cumbersome and without any clearly foreseeable results. So many sit on the outside of Big Data and wait for the competition to make the first move. By then, unfortunately, catch up will be very expensive and hard to break into.



Should these projects prove themselves and result in real data that aids in decision making, then think about big vendor's tools and infrastructure. This is a great way to start.

Big Data growth in tools or in Information Mediaries?

Where is the growth going to be for Big Data? Where are the money makers going to be? As we look at the current market of companies putting out Big Data tools, all promising to make Hadoop easier to use and only garnering 10% of the up an coming bubbled market.

Personally, I know many people in IT departments and they are overworked getting the business projects they are working on complete and on time and do not have a whole lot of time or resources devoted to getting some new MapReduce tool to work in their data warehouse. I am certainly exaggerating and I know there are many working on Big Data in their IT department, but I think those of us who are chasing the money might be missing a very large development step in between.

Instead of Big Data tool maker selling its wares to a big company with a lot of data, what about the many companies who are putting the outputs of these efforts to work and selling the actual results of the data analysis. One such company is masFlight, who uses the platform they have worked hard to install as a service at an airline's site, to immediately analyze said data for the decision results an airline needs to have. masFlight is an excellent combination of smart data gurus who are taking their industry knowledge to create an industry solution. One step further is RetailNext, who does not even sell a solution platform, but instead the raw data output, primed for retail decision making. RetailNext claims to be the ultimate source for brick and mortar retailers in providing them with the in-store data they need.

So, to answer the Big Data money question ("Where is the money?"), we need to look not only at those companies making the tools like the Hortonworks and Clouderas, but also the information mediaries using these tools to provide the industry with the decision making data they need to get that competitive edge.

Will European Countries start with Data Protection schemes?

http://www.bbc.co.uk/news/world-europe-23178284

As we look at more and more governments admitting to data surveillance within the reach they have in their country, you will start to see more and more countries come up with protectionism laws in the hope that data will remain secure. As everything that any government does, expect this to be slow and very ineffective. By the time new legislation is constructed, designed and finally put into law, many moon cycles will have passed and we will have completely different concerns that are heavy in the press by then.

What I do love about this article is the clear and immediate condemnation across the world of the NSA only to be slowly followed by local European governments' admissions to either copying the same tactics as in France or heavy cooperation as in Germany. Opposition parties are quick to draw swords, but are not sure how they would react if in power during this post 9/11 era, where we are all enjoying a modicum of peace without global terrorist attacks.