FT states it well: Big changes over the next five years

Please read this article:

Tech executives facing up to hard realities of the cloud

The FT columnist Richard Waters is finally describing the pink elephant in the board room of so many tech giants around the world. I have stated it publicly and will do so again here:

In the next five years we will see IT increasing its functionality by 300% while decreasing its budget by 50%.

The proof is in the pudding or better in the cloud. Countless executives from large IT users such as DHL have said it again and again that they want to buy computing services in the same way they currently buy power and water. DHL has stated again and again that they want to look at the cost of shipping a package as 100% and know exactly that the IT burden on that cost is 8%. I just made up that number.

But as you can imagine in today's marketplace this is not possible. Servers and mainframes bought years ago are going on refresh and enterprise applications are bloating the need for more storage. Yes, after the financial year is over, DHL can go through and make its calculations, but they are far too high and uncontrollable in a cycle such as this. They want what they get from the power company: if they ship a lot this month, they will pay a lot, but if they ship little, their bill should go down in its reflection.

Amazon gets it - maybe not through their ingenious foresight, but since they were in the right place to rent server capacity. Big server manufacturers, including the one I work for, do not.

Big Data Easy Start: Amazon Elastic MapReduce (Amazon EMR)

Amazon Web Services has a product/service that could make it easy to break into the world of Big Data in a real way. With Elastic MapReduce, any company can use their Hadoop framework and start building data-intensive tasks for applications without having to build the underlying infrastructure or tuning the Hadoop clusters. Not that tuning is impossible and products from Hortonworks make it all work easier, but you would need to build a lot of structure before you ever really see the results. This seems like a clearly easy way to pay your $90 and get full access to infrastructure and start web indexing, analyzing log files, mining data for results, traditional data warehousing, artificial intelligence through machine learning, financial analysis, biometrics research or even scientific simulation.

For many companies the start into Big Data is expensive, cumbersome and without any clearly foreseeable results. So many sit on the outside of Big Data and wait for the competition to make the first move. By then, unfortunately, catch up will be very expensive and hard to break into.



Should these projects prove themselves and result in real data that aids in decision making, then think about big vendor's tools and infrastructure. This is a great way to start.

Big Data growth in tools or in Information Mediaries?

Where is the growth going to be for Big Data? Where are the money makers going to be? As we look at the current market of companies putting out Big Data tools, all promising to make Hadoop easier to use and only garnering 10% of the up an coming bubbled market.

Personally, I know many people in IT departments and they are overworked getting the business projects they are working on complete and on time and do not have a whole lot of time or resources devoted to getting some new MapReduce tool to work in their data warehouse. I am certainly exaggerating and I know there are many working on Big Data in their IT department, but I think those of us who are chasing the money might be missing a very large development step in between.

Instead of Big Data tool maker selling its wares to a big company with a lot of data, what about the many companies who are putting the outputs of these efforts to work and selling the actual results of the data analysis. One such company is masFlight, who uses the platform they have worked hard to install as a service at an airline's site, to immediately analyze said data for the decision results an airline needs to have. masFlight is an excellent combination of smart data gurus who are taking their industry knowledge to create an industry solution. One step further is RetailNext, who does not even sell a solution platform, but instead the raw data output, primed for retail decision making. RetailNext claims to be the ultimate source for brick and mortar retailers in providing them with the in-store data they need.

So, to answer the Big Data money question ("Where is the money?"), we need to look not only at those companies making the tools like the Hortonworks and Clouderas, but also the information mediaries using these tools to provide the industry with the decision making data they need to get that competitive edge.

Will European Countries start with Data Protection schemes?

http://www.bbc.co.uk/news/world-europe-23178284

As we look at more and more governments admitting to data surveillance within the reach they have in their country, you will start to see more and more countries come up with protectionism laws in the hope that data will remain secure. As everything that any government does, expect this to be slow and very ineffective. By the time new legislation is constructed, designed and finally put into law, many moon cycles will have passed and we will have completely different concerns that are heavy in the press by then.

What I do love about this article is the clear and immediate condemnation across the world of the NSA only to be slowly followed by local European governments' admissions to either copying the same tactics as in France or heavy cooperation as in Germany. Opposition parties are quick to draw swords, but are not sure how they would react if in power during this post 9/11 era, where we are all enjoying a modicum of peace without global terrorist attacks.

Truth exposed by Big Data: Humans are fundamentally of a greater order than algorithms

This is a bit of review of something that was explained in a presentation by a founding member of Palantir, Stephen Cohen at a Wired Magazine seminar last year. For the purpose of this discussion, algorithms are defined as a plan so well defined that there is no ambiguity to its execution. 

That might need to sit for a bit, but Cohen explains it through history: Before the industrial revolution, every product was produced individually using an ad hoc method. Then came the industrial revolution where we learned to mass produce and therefore created algorithms, which consistently reproduced steps in a production process.

This is the fundament of an algorithm, which is the foundation of all computing process based on the confines of Boolean algebra (if then else).  The algorithm is ruthless in its ability to repeat performance based on a set scale of explicit and upfront inputs. It never wavers and can do this repetitive task many times over.

Big data is the phenomenon of these algorithms to not only do their task, but put out information along the way. So, the big data concept is a phenomenon of algorithms self-propagating an information flow.  Due to this recurring nature, the shear amount of data is exploding not only in size, but also in type.

So, as exciting as the algorithms are, what can’t they do?

Well as amazing as algorithms are, in that they make decisions without context for quality is exactly their limitation, which eventually bounds their potential. So an algorithms cannot treat or produce qualitative data like hunger, fear, happiness, etc. And it is exactly this qualitative data that we humans need to make decisions.

In order for algorithms to be able to make decisions, I have to strip the data of its qualitative nature and make it shallow and open for interpretations. I have to scale my hunger on a scale from one to ten, which cannot be done without ambiguities.

Algorithms also fail to capture subtle contexts, which would make their efficiency go away. Algorithms are efficient because everything fed to them has to be explicit and upfront. A complex human situation can be understood by humans, but very hard to communicate. Thus they are hard to break down to purely quantitative data and fed into an algorithm.

The final result of all this good stuff: Computers will never replace humans.

All of this information is copied 100% from a speech given by Stephen Cohen, founding member of Palantir. No original parts have been added and I take no credit for authorship.

Perfect Marriage: CSC and Infochimps

More and more Big Data startups will be looking for an exit to their investments and this is a perfect example of things working out. How many times have we seen companies in a bubble all jump on a hot topic and get floods of VC dollars pumped in to them. Like in other hypes, there are literally 100s of startups all claiming to work to get 10% market share. Mathematically, many people will lose their shorts.

What I really like about CSC is their openness to the cloud and their leadership to take their clients into areas that need cloud. They have been doing this for years now, but with Infochimps, which was really not a tool, but a professional services model, there is a perfect compliment of capabilities. This is augment CSC to offer even greater services to their clients that will result in unique business value. Infochimps' technology will see client engagements they had only dreamed up previously.

Compliments to both management teams for having the foresight to complete these marriage!

More info:
http://gigaom.com/2013/08/07/csc-buys-infochimps-and-its-big-data-platform/
http://blog.infochimps.com/2013/08/06/infochimps-a-csc-company-big-data-made-better/

Germany invests 20 times more in industrially-relevant R&D than the U.S.

Germany invests 20 times more, as a share of GDP, in industrially-relevant research and development than the U.S. 

 Article and research is here

 I know, I hate to bring up the obvious and politically motivated question, but just how much is our government doing to make the US more competitive in this world economy.

Merkel and Obama G20 Summit

 Before everyone jumps up and comments about how our education system needs to change, let me save you some key strokes: I completely agree. The major change that is needed in the US is our education system no question. But education system changes will take a long time to take hold, whereas the immediate changes of our economy could be steered by our current administration.

To come to Obama's defense, everything the government invests in will not pay off, see the Solyndra. But what I wanted to bring attention to is the mere size of investments. When we compare ourselves to Germany, we are one 20th of their investment with respect to GDP. That means the immediate effect on the economy will not be felt nearly as much on the US economy and our workforce as it will in Germany.

I don't want to sound pro-German, since they do many things wrong as well, but the results are clear. When Germany has an economic downturn, there are far fewer unemployed workers in the market place then when the US goes through its wild swings like we did in 2008. Banking control? Please, that is why the good Germany banks have subs in the US, since they realize that fewer controls can make them all personally rich.

There is nothing wrong emulating what works in Germany. Let's study the way they fund innovation and find industries that need a boost to grow strong and keep jobs in the US. Not to hard to ask, is it?

Could computing could lose because of NSA? Oh please!

Cloud computing industry could lose on NSA security concerns

http://www.itif.org/publications/how-much-will-prism-cost-us-cloud-computing-industry

Somehow I find it fascinating that people can still spread the rumor that putting data in the cloud somehow makes that data insecure. Strangely enough, the only people propagating these false rumors are those who do not understand technology and still have not grasped what a network is. I try to educate these Luddites that the only way to keep data 100% secure is to lock it up in your house. This data could be on a server, but the server must not be on any network wired or Wifi. Most people think that if the data is in their four walls it is certainly secure, but fail to see that as soon as they hook up a simple connection to the Internet it is accessible by thousands.

But people fail to see that and do not understand that a good cloud provider would have security tools that will dwarf whatever they have put on their firewall and if they really want data to be in secure hands, choose a serious cloud provider.

This goes against our human nature thinking, which has been blown by invention of the transistor about 40 years ago.