What’s in a Name; 7 Blockchain Benefits for the Finance Industry

What’s in a Name; 7 Blockchain Benefits for the Finance Industry

A few days ago, The Merkle ran a story that R3CEV, the largest blockchain consortium of banks and technology firms, admitted that the technology they are developing does not use a blockchain and as such they admitted defeat. A day before that article, R3CEV released a story about when a blockchain is not a blockchain to explain that what the R3 partnership is developing is actually not a blockchain, but an open source distributed ledger technology (Corda). As R3CEV explains in their article, it is “heavily inspired by and captures the benefits of blockchain systems, but with design choices that make it able to meet the needs of regulated financial institutions”.

The distributed ledger platform that has been developed by R3CEV in collaboration with 70 global institutions from all corners of the financial services industry has a few unique settings that, according to R3CEV, makes it not a blockchain. These changes were required to satisfy regulatory, privacy and scalability concerns. As such, the platform restricts access to data within agreements to predetermined actors and the financial agreements used are smart contracts that are actually legally enforceable as they are rooted firmly in law.

Whether it is a blockchain or not, or simply ...


Read More on Datafloq
Garbage In is Garbage Out; How Big Data Scientists Can Benefit from Human Judgment

Garbage In is Garbage Out; How Big Data Scientists Can Benefit from Human Judgment

This article is Sponsored by Search Strategy Solutions, experts in offering your data scientists high-quality, reliable human judgments and support.

The quality of your data determines the quality of your insights from that data. Of course, the quality of your data models and algorithms have an impact on your results as well, but in general it is garbage in, garbage out. Therefore, (Total) Data Quality Management (DQM) or Master Data Management (MDM) have been around for a very long time and it should be a vital aspect of your data governance policies.

Data governance can offer many benefits for organizations, including reduced head count, higher quality of data, better data analysis and time savings. As such, those companies that can maintain a balance of value creation and risk exposure in relation to data can create competitive advantage.

Human judgments and Data Quality

Garbage in, garbage out. Especially with the hype around artificial intelligence and machine learning, that has become more important than ever. Any organization that takes itself serious and employs data scientists to develop artificial intelligence and machine learning solutions should take the quality of data very serious. Data that is used to develop, test and train algorithms should be of high quality, ...


Read More on Datafloq
Algorithms are Black Boxes, That is Why We Need Explainable AI

Algorithms are Black Boxes, That is Why We Need Explainable AI

Artificial Intelligence offers a lot of advantages for organisations by creating better and more efficient organisations, improving customer services with conversational AI and reducing a wide variety of risks in different industries. Although we are only at the beginning of the AI revolution that is upon us, we can already see that artificial intelligence will have a profound effect on our lives. As a result, AI governance is also becoming increasingly important, if we want to reap the benefits of artificial intelligence.

Data governance and ethics have always been important and a few years ago, I developed ethical guidelines for organisations to follow, if they want to get started with big data. Such ethical guidelines are becoming more important, especially now since algorithms are taking over more and more decisions. Automated decision-making is great until it has a negative outcome for you and you can’t change that decision or, at least, understand the rationale behind that decision. In addition, algorithms offer tremendous opportunities, but they have two major flaws:


Algorithms are extremely literal; they pursue their (ultimate) goal literally and do exactly what is told while ignoring any other, important, consideration;
Algorithms are black boxes; whatever happens inside an algorithm is only known ...


Read More on Datafloq
The Future of Artificial Intelligence; Why We Need Explainable AI

The Future of Artificial Intelligence; Why We Need Explainable AI

Artificial Intelligence offers a lot of advantages for organisations by creating better and more efficient organisations, improving customer services with conversational AI and reducing a wide variety of risks in different industries. Although we are only at the beginning of the AI revolution that is upon us, we can already see that artificial intelligence will have a profound effect on our lives. As a result, AI governance is also becoming increasingly important, if we want to reap the benefits of artificial intelligence.

Data governance and ethics have always been important and a few years ago, I developed ethical guidelines for organisations to follow, if they want to get started with big data. Such ethical guidelines are becoming more important, especially now since algorithms are taking over more and more decisions. Automated decision-making is great until it has a negative outcome for you and you can’t change that decision or, at least, understand the rationale behind that decision. In addition, algorithms offer tremendous opportunities, but they have two major flaws:


Algorithms are extremely literal; they pursue their (ultimate) goal literally and do exactly what is told while ignoring any other, important, consideration;
Algorithms are black boxes; whatever happens inside an algorithm is only known ...


Read More on Datafloq
How Blockchain Could Help End Poverty in All Its Forms

How Blockchain Could Help End Poverty in All Its Forms

Technological advancements have reduced global poverty significantly in the past 100 years. Although many people have been able to leave poverty due to this, however, still more than 1.3 billion people live in extreme poverty. Extreme poverty is defined as having less than $1.25 to spend every day. There are a wide variety of causes for poverty, which differ per country, but in general causes for poverty include: lack of education, environmental problems, lack of access to banking facilities, lack of legal ownership of property, lack of rule of law, overpopulation, epidemic diseases or changing trends in a country’s economy.

Overcoming poverty is vital if we want to create a world that is peaceful and fair for everyone. Besides, in 2015 the United Nations adopted the Sustainable Development Goals, which includes challenging global leaders to help end poverty in all its forms, everywhere, by 2030. In this article, I will argue why and how Blockchain could help achieving this goal. Breaking the cycle of poverty begins with investing in children and providing them with quality education, knowledge and skills to enable them to realise their full potential. Next to education comes access to affordable proper health care, access to clean water ...


Read More on Datafloq
Top 10 Insights with Datafloq Articles in 2016

Top 10 Insights with Datafloq Articles in 2016

2016 has come to an end, so it is time to look back at all the great articles that were published on Datafloq. From artificial intelligence to blockchain and the Internet of Things, our most popular articles covered leading and emerging technology trends. Take a minute and read-up on the articles help you better understand these trends.

For 2017, we will continue to publish great articles to help you understand and get started with these technologies. At Datafloq we are looking to grow our network of expert contributors. If you have expertise in Big Data, AI, Robotics, IoT, VR, Cloud, or Technology in general and are interested in contributing please click here.

For now, enjoy reading these articles and we wish you a fantastic 2017 with new emerging technology and trends!

12 Algorithms Every Data Scientist Should Know

Algorithms have become part of our daily lives and can be found in any aspect of business. What are the 12 algorithms every data scientist should know?

The Top 7 Big Data Trends for 2017

Which Big Data trends will create the Year of Intelligence? Here are the top 7 Big Data trends for 2017 that will affect organisations and governments.

13 Predictions on Artificial Intelligence

Artificial Intelligence already had a ...


Read More on Datafloq
How Blockchain Can Help Combat Climate Change

How Blockchain Can Help Combat Climate Change

Blockchain technology will enable us to decentralise the web and decentralise many, if not all, the services that are offered via the web. A decentralised internet was how the internet was originally envisioned, but somehow in the past 25 years, it ended up in the hands of a few very powerful companies. As Sir Tim Berners-Lee said during the Decentralised Web Summit in 2016:

“The web was designed to be decentralised so that everybody could participate by having their own domain and having their own webserver and this hasn’t worked out. Instead, we’ve got the situation where individual personal data has been locked up in these silos.”

Fortunately, Blockchain will allow us to bring back the power to the users and to create a decentralised society. Already, Blockchain challenges many industries, of which the financial industry will see the largest impact of Blockchain in the coming years. The question, however, is how Blockchain will have an impact on other global problems. In a new series of Blockchain posts, I will dive into different problems we face and see if and how the Blockchain could help solve these issues. I will discuss problems such as climate change, poverty, refugee crisis, voting fraud, censorship ...


Read More on Datafloq
The Top 7 Big Data Trends for 2017

The Top 7 Big Data Trends for 2017

It is the end of the year again and a lot has happened in 2016. Google’s AlphGo algorithm beat Lee Se-dol in the game of Go, Blockchain really took off and governments around the globe are investing heavily in smart cities. As every year, I will provide you with the big data trends for the upcoming year, just as I did for 2014, 2015 and 2016. 2017 promises to be a big year for Big Data. The Big Data hype is finally over and, therefore, we can finally get started with Big Data. That is why I would like to call 2017 the Year of Intelligence. So, which big data trends will affect your organisation in 2017? Let’s have a look at the seven top big data trends for 2017.

1. Blockchain-enabled Smart Contracts: Blockchain 2.0



Zapp2Photo/Shutterstock

In 2016, Blockchain took off with a lot of media attention on the distributed technology that will drastically change organisations and societies. Many organisations are exploring Blockchain solutions. The R3 Partnership, which involves over 70 of the largest banks in the world, seeks to invest almost $60 million in the development of their blockchain platform. Although four prominent banks left the consortium, it shows that banks ...


Read More on Datafloq
What is the Blockchain – part 5 – ICOs and DAOs

What is the Blockchain – part 5 – ICOs and DAOs

The Blockchain has the potential to completely change our societies. However, it is still a nascent technology, one which can be difficult to grasp. Therefore, I am writing a series of blog posts on what is the blockchain. This is part 5 and the final blog post of this series. The first blog post was a generic introduction to the blockchain. The second post provided insights into consensus mechanisms and different types of blockchain. The third post offered information on five challenges of the blockchain that need to be solved and the fourth post talked about smart contracts on the blockchain. In this final post, I will dive into two concepts that have enormous potential to radically change our world; ICOs and DAOs.

ICOs – Every Company Its Own Central Bank

Startups have always been looking for funds to invest in their venture to build the next Facebook or Google. However, money is expensive and any startup that raises money has to give a share of the company to the investors. The earlier an investor joins, the higher the risk, the more expensive it becomes. That has been the paradigm for the past decades. Not anymore. Since the rise of the Blockchain, ...


Read More on Datafloq
How to Transform Your Business into an Analytics Company?

How to Transform Your Business into an Analytics Company?

This thought leadership article is brought to you by SayOne Technologies – providing tailored data analytics applications for customers around the world.

Many organizations are affected by the rapid changes in our society due to new technologies. The rate of change that organizations experience has increased to such an extent that half of S&P 500 companies are expected to be replaced by newcomers in the next 10 years. These newcomers use emerging technologies, such as Artificial Intelligence, Robotics, 3D printing or the Internet of Things. In addition, most of the new organizations take a different form than traditional organizations; many new organizations, or startups, are decentralized platform organizations that have been around for less than a decade but have experienced exponential growth.

Examples of these newcomers are Uber, which is the world’s largest taxi company that does not own any taxis; AirBnB, the world’s largest accommodation provider that does not own any hotels; WhatsApp, the world’s largest telecom company that does not own any telecom infrastructure or Alibaba, the second largest retailer that does not own any inventory. Each of these innovators disrupts an entire industry and they have two common denominators; they collect data in everything they do and they use ...


Read More on Datafloq
What is the Blockchain – part 4 – Transactions and Smart Contracts

What is the Blockchain – part 4 – Transactions and Smart Contracts

Blockchain is rapidly gaining attention from organisations in every industry. However, it is a difficult to understand technology that, if not executed correctly could result in serious harm for your organisation. Therefore, in this series of posts on the blockchain, I explain what the Blockchain is and how it affects your organisation. The first part was a generic introduction on the Blockchain, while the second part focused on different types of blockchains and dApps. The third blog provided insights in several startups that are working hard on developing the required technology as well as several Blockchain challenges that need to be overcome before we will see wide-scale adoption of the Blockchain. In this fourth post, I will dive deeper in the different type of transactions that can be recorded on the blockchain as well as one particular type of transactions; smart contracts.

Different Transactions

A key characteristic of the blockchain is that it removes the need for trusted intermediaries; centralised organisations that take a fee for verifying transactions. Removing the middlemen, completely changes the game for organisations that want to do business with each other. Last week, for the first time, a transaction took place between two organisations across the globe which ...


Read More on Datafloq
What is the Blockchain – part 3 – Blockchain Startups and Five Challenges to Overcome

What is the Blockchain – part 3 – Blockchain Startups and Five Challenges to Overcome

In this series of posts, I am providing insights in a technology that will change our world. Blockchain has been said to be as important invention as the Internet and Johann Palychata, a research analyst from BNP Baripas, called Blockchain an invention like the steam or combustion engine.

In part 1 of this series I gave an introduction to Blockchain and in part 2 I provided insights in different types of Blockchain and consensus algorithms. This third part will discuss some of the major challenges we will need to overcome to make Blockchain truly change our world for the better. But first, let’s look at some startups who are trying to change the world through Blockchain

Blockchain Startups

Ethereum

Ethereum has the ambition to reinvent the internet and they are well on track to achieve that. Ethereum has been around for a few years now and it is a decentralised platform to develop Decentralised Applications (DApps) that run through smart contracts. These smart contracts are small software programs that execute a task, a sort of If This Then That statement, but then a lot more complex. They run on a custom-built blockchain and as such, there is not a chance for fraud, censorship or ...


Read More on Datafloq
What is the Blockchain – part 2 – and Why It Will Change Our World

What is the Blockchain – part 2 – and Why It Will Change Our World

For the tech-savvy people among us, the Blockchain might be nothing new and it may be clear that it will have a big impact on the world. However, for many people, the Blockchain is still a mystery, a puzzle or an unknown unknown. Therefore, in a series of posts, I share with you what the Blockchain is, how it works and how it will completely change the world as we know it, if we get it right.

In my first post about the Blockchain, I explained the basics of the Blockchain and in this post I will go a bit deeper and talk about the different types of Blockchains, some examples of dApps and talk about the most important part of the Blockchain; the consensus algorithms to validate the data.

Different Types of Blockchains

The most well-known Blockchain is the Bitcoin Blockchain. The Bitcoin Blockchain was envisioned by Satoshi Nakamato in 2008 and this is a so-called Permissionless Blockchain, or public Blockchain. This means that anyone interested to join the Blockchain, can do so by simply hooking-up his/her computer to the decentralised Blockchain network, download the Blockchain and contribute to the processing of transactions. It is not required to have a previous relationship with ...


Read More on Datafloq
Kick-Starting the 4th Industrial Revolution, One Blockchain at a Time

Kick-Starting the 4th Industrial Revolution, One Blockchain at a Time

We live in a future of accelerated change and today’s world is changing faster than we have ever seen before. New technology is changing the way we live, work and collaborate. No longer is it sufficient to for organisations to sit back and stick to the status quo. Today’s new technologies require an active attitude by organizations, if they want to remain in business in the next decade.

I am talking about the 4th Industrial Revolution that is rapidly approaching and it will bring change at unlike we have ever seen before. In fact, it will change what it means to be humans. It also offers us a tremendous opportunity to create a world that is good for all, where technology is used for the good, privacy of consumers is respected and data is used to improve the lives of all humans. The 4th Industrial Revolution is all about Algorithms, Machine-Learning and Artificial Intelligence. It is about robotics, 3D printing and VR/AR, Nano technology, and many more emerging technologies. It is disruption on all levels, resulting in system-wide innovations that can change an industry in years instead of decades. The combination of such revolutionary emerging technologies will bring us realities that until recently would ...


Read More on Datafloq
What is the Blockchain and Why is it So Important?

What is the Blockchain and Why is it So Important?

Blockchain is growing in importance. Increasingly organisations have to explore what this revolutionary technology will mean for their business. Marc Andreessen from the well-known VC firm Andreessen Horowitz calls it as big an invention as the internet. Last year, in my Big Data Trends prediction for 2016, I already foresaw that 2016 would become the year of the Blockchain and now also Gartner has included in their Hype Cycle for Emerging Technologies.

Many organisations are already exploring the possibilities of the Blockchain, although primarily still in the Financial Services industry. The R3 Partnership is a consortium of 45 of the biggest financial institutions, investigating what the Blockchain means for them. Next to the R3 consortium, four of the biggest global banks, led by Swiss bank UBS, have developed a “Utility Settlement Coin” (USC), which is the digital counterpart of each of the major currencies backed by central banks. Their objective is to develop a settlement system that processes transactions in (near) real-time instead of days. A third example is Australia Post, who have released plans for developing a blockchain-based e-voting system for the state of Victoria.

The possibilities of the Blockchain are enormous and it seems that almost any industry that deals ...


Read More on Datafloq
Why the 2016 Hype Cycle for Emerging Technologies is all about Data

Why the 2016 Hype Cycle for Emerging Technologies is all about Data

Every year, Gartner published the Gartner Hype Cycle for Emerging Technologies and today revealed the 2016 edition. Last year, Gartner tempered the expectations of Big Data, by predicting it would take another 5-10 before it would reach the plateau of productivity. This year, Gartner finally added “emerging” technologies such as the Blockchain and Machine Learning. Let’s have a look at the 2016 Hype Cycle for Emerging Technologies and see what it means.



Gartner identified three key technology trends that organisations need to track in order to gain competitive advantage:

1. The Perceptual Smart Machine Age

Gartner added ‘General Purpose Machine Intelligence’ as an ‘Innovation Trigger’ to this year’s hype cycle and expects it to take more than 10 years to reach the plateau of productivity. Of course, machine intelligence is nothing new, but that is (very) specific machine intelligence, i.e. a machine is extremely good in doing one, simple, task. General purpose machine intelligence is something different and requires extreme amounts of computing power and near-endless amounts of data, which is why it will probably take a lot longer. It is also called Artificial General Intelligence and it is an emerging field dealing with the development of ‘thinking machines’ with intelligence that is ...


Read More on Datafloq
Why Blending Data Analytics and Gut-Feeling Benefits your Business

Why Blending Data Analytics and Gut-Feeling Benefits your Business

Understanding the impact of Big Data is not self-evident for many companies. Big Data offers almost endless possibilities and as such organizations are overwhelmed. Big Data requires different technologies, new IT systems, new processes and a different way of working. In addition, Big Data requires a different culture and changing your company culture is always hard, especially when new technology is involved.

In order to be successful with big data, you need a culture that incorporates data-driven decision-making. That does not mean, however, that organizations should only focus on big data analytics and that they should ignore gut-feeling. Gut-feeling, or intuitive synthesis, is an important aspect of decision-making and successful companies are capable of combining the two in what has become known as Design Thinking.

Design Thinking; A Creative and Data-Driven Process

In the past decade, design thinking, also known as a human-centered approach to innovation, has become a popular practice at organizations from around the world to generate innovative and competitive strategies. Although the history of design thinking can be traced back to the 1960s, the adaptation of design thinking for business purposes followed in 1991 by the founder of IDEO, David Kelley. Specialized design thinking firms such as IDEO help organizations create ...


Read More on Datafloq
Why Humanizing Algorithms Could Prevent them from Going Awry

Why Humanizing Algorithms Could Prevent them from Going Awry

Algorithms are taking over the world. Not yet completely and not yet definitely, but they are well on their way to automate a lot of tasks and jobs. This algorithmization offers many benefits for organizations and consumers; boring tasks can be outsourced to an algorithm that is exceptionally well at a very dull task, much better than humans could ever become. More complicated tasks can benefit from insights derived from analyzing multiple data sources and these insights can help humans in the task at hand. Soon, however, also these tasks could be taken over by algorithms.

We know many examples of the first, ranging from robots that build your smartphone to algorithms that find that particular website within milliseconds. More and more we also see great examples of the latter, from an algorithm that has a seat at the board of directors of Hong Kong venture capital firm Deep Knowledge Ventures to algorithms that can instantly translate spoken language into a different language.

Algorithms are therefore rapidly changing how we do business. Businesses consist of value propositions, customer segments, consumer relationships, channels, revenue streams, cost structures, limited resources, partnerships and activities. Algorithms enable each of these elements to be automated and using ...


Read More on Datafloq
Why Humanizing Algorithms Is a Good Idea

Why Humanizing Algorithms Is a Good Idea

Algorithms are taking over the world. Not yet completely and not yet definitely, but they are well on their way to automate a lot of tasks and jobs. This algorithmization offers many benefits for organizations and consumers; boring tasks can be outsourced to an algorithm that is exceptionally well at a very dull task, much better than humans could ever become. More complicated tasks can benefit from insights derived from analyzing multiple data sources and these insights can help humans in the task at hand. Soon, however, also these tasks could be taken over by algorithms.

We know many examples of the first, ranging from robots that build your smartphone to algorithms that find that particular website within milliseconds. More and more we also see great examples of the latter. There is an algorithm that has a seat at the board of directors of Hong Kong venture capital firm Deep Knowledge Ventures. In addition, there ar algorithms that can instantly translate spoken language into a different language.

Algorithms are therefore rapidly changing how we do business. Businesses consist of value propositions, customer segments, consumer relationships, channels, revenue streams, cost structures, limited resources, partnerships and activities. Algorithms enable each of these elements to be automated ...


Read More on Datafloq
How Unlimited Computing Power, Swarms of Sensors and Algorithms Will Rock our World

How Unlimited Computing Power, Swarms of Sensors and Algorithms Will Rock our World

We have entered a world where accelerated change is the only constant. The speed at which technologies are currently developing is unlike any other since the existence of mankind. When we look at the past two hundred years, we have seen multiple inventions that changed society as we knew it. First we had the invention of the book press, which made books available for the (general) public. Then we had the invention of the steam machine, which significantly altered any industry on earth and pushed mankind to the next level. And in the 20th century we saw the invention of the Internet and the computer. This latest ‘information revolution’ is of a different scale than the industrial revolution made possible by the steam machine. 

Although when the Internet was invented, it did not yet look like something major was going one. But isn't that always the case with ground breaking inventions? It takes some time before you know what has happened. Now, almost fifty years later, we can finally get a glimpse about the profoundness of this invention as we slowly, but surely, enter the information era. 

In the past decades, we have seen the creation of a digital infrastructure that is ...


Read More on Datafloq
How Algorithms Could Propel Us to Earth 2.0

How Algorithms Could Propel Us to Earth 2.0

We have entered a world where accelerated change is the only constant. The speed at which technologies are currently developing is unlike any other since the existence of mankind. When we look at the past two hundred years, we have seen multiple inventions that changed society as we knew it. First we had the invention of the book press, which made books available for the (general) public. Then we had the invention of the steam machine, which significantly altered any industry on earth and pushed mankind to the next level. And in the 20th century we saw the invention of the Internet and the computer. This latest ‘information revolution’ is of a different scale than the industrial revolution made possible by the steam machine. 

Although when the Internet was invented, it did not yet look like something major was going one. But isn't that always the case with ground breaking inventions? It takes some time before you know what has happened. Now, almost fifty years later, we can finally get a glimpse about the profoundness of this invention as we slowly, but surely, enter the information era. 

In the past decades, we have seen the creation of a digital infrastructure that is ...


Read More on Datafloq
How to Win your Customers for Life with Predictive Analytics

How to Win your Customers for Life with Predictive Analytics

Winning your customer for life is a challenging task for organizations. How can you connect with your customer and how can you ensure that they stay with your organization for a long time? Questions that many organizations face.  Fortunately, with the advance of big data and analytics, it has become a little bit easier for organizations. Last week, I spoke at the Retail, eCommerce, Payments and Cards conference in Dubai, one of the biggest in the Middle East, and I would like to share some of my keynote insights with you through this article.

These are challenging times for organizations. Organizations have to face disruptive innovations from many different angles and accelerated change in technological advances require organizations to constantly change and adapt. On the other hand, we have moved from descriptive and diagnostic analytics to the more advanced predictive analytics and we are moving towards prescriptive analytics. The more we use data to predict what will happen and what action should be taken, the more difficult it becomes, but also the more value that can be created.



Source: Gartner

New organizations that disrupt multiple industries understand this very well. They use data in every possible way. At every possible touchpoint with customers ...


Read More on Datafloq
Datafloq Launches Recruitment Platform around Data Professionals

Datafloq Launches Recruitment Platform around Data Professionals

A unique recruitment platform focused around data professionals to hire the best big data talent or find that dream job.

According to research from Gartner, more that 75 percent of organizations are investing or planning to invest in big data, meaning an increased demand for data professionals around the globe. Fortunately, Datafloq is a large community of data lovers, engineers, data scientists, managers and chief data officers. Therefore, I am proud to announce that Datafloq launched a unique recruitment platform around data professionals to hire the best big data talent or find that dream job: datafloq.com/work

Data Employers Finding Talent

Organizations looking to attract data professionals will have extensive possibilities to find the best big data talent. They will be able to post, and promote, job posts using a unique skills and technology system, resulting in more qualified professionals to choose from. Users will be able to easily find big data jobs. They will be able to create unique data employer branding pages to showcase the unique company culture that aligns with the employment branding goals using video’s, images, employee quotes and much more, allowing professionals to find the best data employer.

Organizations can also find talent via our professional database search engine, which ...


Read More on Datafloq
12 Qualities Your Next Chief Data Officer Should Have

12 Qualities Your Next Chief Data Officer Should Have

The Chief Data Officer is on the rise! In a 2015 Forrester research of 3005 data analytics decision-makers, 45% said that their company had appointed a Chief Data Officer (CDO). In addition, a survey among 254 CIO’s worldwide revealed that 90% of the current CIO’s believe that data is changing their business and 92% believe that a CDO could best manage this. Another survey revealed that 76% of CIO’s would like to see the CDO as a board level position by 2020.

It may be clear, the Chief Data Officer is here to stay and it is/will be an important role for organizations. It is also a difficult role, because the CDO has to have the right balance of Big Data skills and business skills in order to be able to deal with all aspects related to big data. Earlier, I developed a Chief Data Officer profile, to help organizations get an idea of what the CDO role should look like and what tasks the Chief Data Officer would have. Since many organizations are on the lookout for a suitable Chief Data Officer, let’s now have a look at what the key qualities and characteristics are that your next CDO should ...


Read More on Datafloq
12 Algorithms Every Data Scientist Should Know

12 Algorithms Every Data Scientist Should Know

Algorithms have become part of our daily lives and they can be found in almost any aspect of business. Gartner call this the algorithmic business and it is changing the way we (should) run and manage our organizations. There are all kinds of algorithms and for each aspect of your business there are different algorithms, which nowadays you can even buy at an algorithm marketplace. Algoritmia provides developers with over 800 algorithms in the fields of audio and visual processing, machine learning and computer vision, saving developers precious time and money.

However, the algorithms available on the Algoritmia marketplace might not be suitable for your particular need. After all, for different circumstances you require different algorithms and the same algorithm in a different environment can produce different results. In fact, there are many different variables that determine which algorithm to be used and how the algorithm will perform. These variables include the type and volume of the data, the industry the algorithm will be applied to, the application it will be used for etc.

Therefore, sometimes buying an off-the-shelve algorithm and then tweaking it might not be the best option. Data scientists should still educate themselves in the most important algorithms; how ...


Read More on Datafloq
3 Powerful Applications of Using Analytics-as-a-Service

3 Powerful Applications of Using Analytics-as-a-Service

This article is sponsored by CloudMoyo - Partner of choice for solutions at the intersection of Cloud & Analytics

Analytics-as-a-Service is the combination of analytics software and cloud technology. Instead of hosting any analytics software on premises using your own servers, you use a ready-to-go solution that is easy to deploy and most of the time has a pay-as-you-go payment system. It is part of a larger ‘as-a-Service’ solutions such as ‘Software-as-a-Service’ or ‘Platform-as-a-Service’. Thanks to the advancements made by well-known hosting providers such as AWS and Microsoft Azure, Analytics-as-a-Service has really taken off in the past years and is here to stay.

There are a lot of advantages for organizations if they use an Analytics-as-a-Service solution. Of course, the elimination of manual IT tasks will benefit many organizations, removing the need to hire expensive DevOps and Engineers. But the most benefits for organizations are in the central use and access to all internal, and external, data. This enables business analysts and end-users to have easy access to all the data and to explore the data at hand interactively, and potentially collaboratively.

Getting Started with Analytics-as-a-Service

In order to benefit from an Analytics-as-a-Service, organizations should make all of their internal data available in the ...


Read More on Datafloq
How Secure is Your Payment Data? Not Very Secure It Seems!

How Secure is Your Payment Data? Not Very Secure It Seems!

Mobile payments are on the rise. Samsung and Apple have each introduced their own payment systems already some time ago and many new startups are entering the market as well, with of course Square being on top of the list. Consumers are also slowly getting used to mobile payments, be it with their smartphone or with their smart watch. In addition, large brands are also entering the mobile payment market. The best example is the Starbucks Card Mobile App, where Starbucks created an iPhone App that enables users to pay for their coffee. Although it is a closed loop system, it is a very convenient service for customers.

Next to these mobile payments systems, there are also various new payment opportunities. Across the world, new systems are being developed from the New Payments Platform (NPP) in Australia to the startup TransferWise, which claims to reduce the costs involved in transferring money abroad. All these new payment systems rely on big data and they work because they combine various data sources in smart ways.

All these new payment systems might be very convenient, the question remains whether they are also secure. After all, your mobile payment data consists of a lot of private ...


Read More on Datafloq
Algorithms Are Changing Business: Here’s How to Leverage Them

Algorithms Are Changing Business: Here’s How to Leverage Them

When Google’s algorithm AlphaGo beat South Korean Go Grandmaster Lee Se-dol by 4-1 last week, it was a significant event in the world of algorithms and artificial intelligence. This is because it represented a new form of artificial intelligence: intuitive artificial intelligence, something which is remarkably more challenging than standard artificial intelligence.

The disruption happening thanks to algorithms is happening all around us. The largest taxi company in the world, Uber, owns no taxis, but uses smart algorithms to connect drivers and passengers. The largest telephone company in the world, WhatsApp, has no telecom infrastructure, but sends over 35 billion message per day. Finally, the world’s second most valuable retailer, Alibaba, owns no inventory but uses algorithms to help others sell products.

Companies like Uber, WhatsApp and Alibaba clearly show that smart algorithms can disrupt an entire industry. But we are just at the start of this disruption and the coming decade will likely see all industries being disrupted thanks to algorithms. Gartner calls this trend the “Algorithmic Business” and it will fundamentally change how we do business.

From Data to Algorithms

With the advance of technology, companies and consumers are generating more and more data. Some organisations, such as Walmart, create and store ...


Read More on Datafloq
How to Close the Big Data Talent Gap at Your Organization

How to Close the Big Data Talent Gap at Your Organization

This article is sponsored by Acamar - a staffing agency specializing in emerging Information Management (IM) competencies.

Big data offers many benefits for organizations in all industries, but unfortunately a lot of companies don’t reap these benefits yet. The reason is not that they don’t want to start with big data, nor that they don’t understand what big data is. The challenge many companies face is attracting the right big data talent.

Big data talent is scarce and what is scarce is expensive. Finding the right data professionals for a big data project remains difficult for a lot of organizations. This does not come as a surprise if we look at the numbers around global big data talent.

A Growing Global Big Data Talent Shortage

Back in 2011, McKinsey already estimated that in 2018 there would be a shortage of 290,000 data scientists in the United States alone. Globally, demand for data scientists is projected to exceed supply by more than 50 percent by 2018. In the UK, the expected shortage is 56,000 data scientists by 2020. Currently there are over 500,000 big data jobs listed globally.

As it may be clear, demand is skyrocketing and this will only increase in the future. On the ...


Read More on Datafloq
How The Internet of Things Is About to Explode

How The Internet of Things Is About to Explode

The Internet of Things is heading our way and it is heading here fast! In the past years we have seen that the Internet of Things (IoT) is really taking of and we can safely say that we are about to reach a tipping point. Any industry is currently working on connected devices. Ranging from connected cutlery (HAPIfork), smart locks for your door (August) or for your bike (SkyLock), connected toothbrush (Oral-B) to connected jewels (Wistiki).

The supply of connected devices is tremendous and the ones just mentioned are only those for consumers. Also the industries such as the mining industry are developing connected smart glasses (Vandrico), the transportation industry of course (what about self-driving gigantic mining trucks) and of course the rise of the robots. 

All these connected devices will have one or more sensors incorporated and this plethora of sensors will drive massive amounts of data. Together all these devices will create an Internet of Things ecosystem that will enable organizations, consumers and devices to be connected with each other. Of course, there are still several challenges that we need to solve before we will have a global, one, Internet of Things, but that is just a matter of time (and ...


Read More on Datafloq
5 Cyber Risks Affecting the Internet of Things and How to Manage These Risks

5 Cyber Risks Affecting the Internet of Things and How to Manage These Risks

The Internet of Things will offer organizations tremendous value and will provide consumers with fantastic benefits. However, the Internet of Things also comes with a wide variety of cyber risks that could harm organizations and consumers who work with the IoT.

In order to protect your organization from these cyber risks you should have the right data governance procedures in place to protect the data that is generated by IoT devices. However, which data governance measures you should take depends on the different cyber risks that could affect your IoT business. And there are quite a few cyber risks to take into account:

5 Cyber Risks Affecting the Internet of Things

Denial of Service Attacks

A distributed denial-of-service (DDoS) attack are, unfortunately, very common attacks, where multiple systems flood the bandwidth or resources of a system, such as a web server. The result is that the web server goes down due to the unexpected massive amount of traffic a web server suddenly has to deal with.

Since the Internet of Things involves a network of sensors, devices and wearables connected with each other through the Internet, a DDoS attack could seriously harm your organization. There are several ways a DDoS attack could affect an IoT ...


Read More on Datafloq
Why Trust and Data Breaches Don’t Get Along

Why Trust and Data Breaches Don’t Get Along

If you want to retain the trust of your customers, you better make sure that you don't have any data breaches. As this infographic, developed by Gemalto, shows, data breaches have a significant impact on consumer trust, loyalty and the perception of how seriously companies take the security of customers' personal and financial data.

According to the survey "nearly two-thirds (64%) of consumers surveyed worldwide say they are unlikely to shop or do business again with a company that had experienced a breach where financial information was stole​n, and almost half (49%) had the same opinion when it came to data breaches where personal information was stolen."

A loss of trust can be very expensive for any organization, especially if it is due to a data breach. Therefore, organizations should do what it takes to prevent a data breach from happening. By taking the right technical security measures as well as cultural / process measures a lot of data breaches can be prevented as quite often a data breach occurs due to human errors.

Unfortuantely, data breaches will not go away very soon and it is likely that if you are a large organization, it is just a matter of time before you ...


Read More on Datafloq
How Big Data Can Upgrade the Customer Experience

How Big Data Can Upgrade the Customer Experience

Booking a hotel has become really easy in the past years. There are multiple booking engines that make it very simple to book the right hotel for your stay. Great customer experience is vital for these booking websites, as the competitor is quite often just one click away.

Therefore, Hotels.com is using Big Data to create the best customer experience to ensure that visitors book via Hotels.com and not via another booking website. In this interesting video, Thierry Bedos, CTO at Hotels.com, explains how they have embedded Big Data within their organization to upgrade the booking experience for their customers.

Customers' expectations have risen in the past few years. Customers much better know what they want and they expect to receive that when they book a hotel in terms of different features and the speed at which the website operates and delivers those features. Customers want a relevant and meaningful experience that is consistent through any channel and they want instant response or otherwise they go to the competitor. This of course is valid for any online organization and therefore any online organization should take a Big Data approach.

As Thierry Bedos explains in this video, Hotels.com have gone through a number of steps ...


Read More on Datafloq
Why Humans Are the Weakest Link with Data Breaches

Why Humans Are the Weakest Link with Data Breaches

The recent TalkTalk data breach is going to be an expensive data breach. It seams that the British Internet Service Provider will have to spend 35 million pounds on recovery measurements. Although that is a lot of money, it is better than going bankrupt as happened to the Dutch security firm Diginotar after being hacked in 2011.

Being hacked is a threat for any organization that deals with a lot of data, which means it is a threat for basically any organization. Therefore, as an organization you should do what it takes to prevent a possible data breach from happening.

Apart from the required IT security measures, it is also important to focus on the processes within an organization. As the below infographic from TSG shows, employees are the weakest link in terms of a possible data breach happening.

There are a lot of possibilities, ranging from finding a USB-stick and being curious to what’s on the device to leaving a laptop on the train and many more different options. Check out the below infographic to gain a better understanding.


...


Read More on Datafloq
5 Ways How Data Analytics is Changing Law Firms

5 Ways How Data Analytics is Changing Law Firms

Like any industry, also the legal industry is affected dramatically by big data analytics. Law firms that do not want to make the move to a data-driven legal practice are going to lose out from the innovative legal firms that embrace big data. Doing manual document review, using printed documents and storing documents in old-fashioned physical cabinets is no longer enough to win in court. Big Data is rapidly changing the way law firms do business.

This video was taken at the Big Law Business summit and it shows how some of the top law firms in the US are applying big data to improve their business. Law firms can no longer be reactive, they have to become pro-active if they want to stay in business. In short, there are 5 ways how data analytics is rapidly changing law firms:


Certain legal cases require certain expertise, something which might be in-house or something that need to come from outside. Big Data enables law firms to better inform whom to hire for outside counseling or, and that’s especially relevant for large law firms, which lawyer would best fit a certain case.
Information is power and having access to massive amounts of documents can only ...


Read More on Datafloq
7 Important Big Data Trends for 2016

7 Important Big Data Trends for 2016

It is the end of the year again and that means it is time for the Big Data trends for next year. I did that for 2014, I did it for 2015 and now it is time for 2016. What is awaiting us in 2016? Which Big Data trends will have an impact on the global Big Data domain? How will Big Data affect organizations in 2016? Let’s have a look at seven of the most important Big Data trends for the year 2016.

1. The Rise of the Algorithms



Big Data is out, Algorithms are in. Data has become a commodity and every organization is capable of collecting and storing vast amounts of data. Analysing all that data is also not so spectacular anymore. Every organization can hire or train Big Data Analysts to understand the patterns within the data.

In 2016 it will be all about what actions you will derive from the data you have access to. Bring in the algorithms. Algorithms define action and they are very specific pieces of software that are very good at a very specific action, much better than humans can do. Think for example of quickly determining the right advertisement based on your profile when you visit ...


Read More on Datafloq
The Future of Big Data: How Data Lakes Open New Possibilities for Your Organization

The Future of Big Data: How Data Lakes Open New Possibilities for Your Organization

This thought leadership article is brought to you by Zaloni – providing enterprise data management solutions for Hadoop Data Lakes

In the past years I have seen a flood of information on data lakes and it looks like it is becoming as much of a buzzword as Big Data. I described data lakes as a growing trend in 2015 – stating that it was time for organizations to experiment with data lakes. But are we seeing much progress?

Sure, organizations like Facebook, Google and Yahoo have advanced considerably and their developers experience numerous benefits using data lakes. But what does it actually mean - a data lake - and what are the benefits of it? And are ‘offline’ organizations such as retailers and financial services companies also moving towards a data lake model? What are the advantages and challenges of a data lake and how can you derive value from it? I think it’s time for a deep dive into data lakes.

The Data Lake Definition

But first, what exactly is a data lake. Data lakes are defined as a repository containing vast amounts of raw data, in native formats, while allowing different users to access and analyse that data as required. A data ...


Read More on Datafloq
7 Reasons why the Algorithmic Business will Change Society

7 Reasons why the Algorithmic Business will Change Society

The world around us is changing so rapidly, that even the hype big data is already outdated. Big data is nothing new any more and by now we all know that more data is coming our way, rapidly.

But in big data itself is no value at all.  We can all generate massive amounts of data after all. By itself, big data is not transformative. Not even big data analytics is driving the change, after all any company can hire analysts to provide insights into your data. Where the real value lies is in algorithms. Algorithms define action and dynamic algorithms are at the core of future businesses. Welcome to the Algorithmic Business. 

The Algorithmic Business

The algorithmic business is a company build around smart algorithms. Algorithms that define company processes, that deliver customer services, that take action when necessary and as such define the way the world works. During the latest Gartner ITXpo, Gartner predicted the rise of the algorithmic business to forever change how we do business.

In order to understand all the data coming our way, algorithms will be required. In the digital era ahead of us, algorithms will be able to operate independently. These algorithms will be able to understand ...


Read More on Datafloq
How Engineering Siemens Creates Value for Their Customers Using Big Data Analytics

How Engineering Siemens Creates Value for Their Customers Using Big Data Analytics

Siemens is a 168-years old engineering company that has prepared itself for the future. While most consumers might know Siemens from washing machines and electronic equipment, that image is largely out-dated. Today, Siemens manufactures trains, power plant equipment, healthcare equipment and offers a wide range of smart software solutions.

In the past decade, they have really moved forward and combined their engineering capability with great new analytical capabilities to really help their customers perform better. Let's look at three examples of how they are changing the game for the energy industry, racing industry and smart cities. 

From Price Per Turbine to Price per Gigawatt 

The energy industry faces difficult challenges, especially with the transition from fossil fuel to clean energy. Energy companies are in need to better understand what is going on within their power plant in order to reduce cost and work more efficiently. 

The amount of data that is created during power generation is growing rapidly. Siemens for example already has 8 global diagnostic centres that monitor over 9000 units. 

Their gas turbines create 30 gigabyte of data per day, their smart energy grid platform creates 25 gigabytes of data a day and Siemens' wind turbines create 200 gigabytes of data every day. 

With ...


Read More on Datafloq
Why a Data Loss Could Mean the End of Your Organization

Why a Data Loss Could Mean the End of Your Organization

Prevention is always better than to repair things. Only 6% of the companies that don’t have an disaster recovery plan when they experience the loss of crucial company data will survive it. Although no one can predict when disaster will strike and what type of disaster will strike, we can and should always be prepared for it.

There can be a wide variety of causes in data loss, including battery failure, human errors, cyber attacks or weather related issues. According to Ponemon Institute’s Cost of Downtime study, every minute of downtime for large organization averages $ 7.900 in lost revenue, per minute. In addition, they estimated that the average recovery time for such an unplanned outage was 119 minutes, meaning an average loss of $ 901,500!

Of course, this amount will be much higher, the bigger the organization and the more the business depends on their digital activities. Not ensuring a well thought-through data recovery plan could therefore significantly endanger the survival of the organization.

So, which steps should be incorporated into a data recovery plan? Well, according to Singlehop who created the below infographic, there are three different steps to take:


Perform a Business Impact Analysis;
Perform a Risk Assessment;
Manage Your Risks.


Next to these ...


Read More on Datafloq
5 Ways Big Data Will Improve Civil Infrastructure

5 Ways Big Data Will Improve Civil Infrastructure

On August 1, 2007 as people were driving home from work on the I-35W Mississippi River Bridge near Minneapolis, it all of a sudden collapsed. Thirteen people died and 145 people were injured. The bridge was constructed in 1967 and was expected to carry 66.000 vehicles per day. However, in 2004 an estimated 141.000 vehicles crossed the bridge on a daily basis. That’s a big difference and turned out to be too much.

The quality of the civil infrastructure within a country, determines its economic possibilities as well as a society’s wealth and quality of life. Therefore, if the infrastructure such as roads, rail tracks and bridges are deteriorating, it could have a big impact on a society. The Federal Highway Administration reported that the costs resulting from the loss of a critical bridge or tunnel could exceed $10 billion.

As the Civil Infrastructure System Task Group of the National Science Foundation once stated: “A civilization that stops investing in its infrastructure takes the first step toward decline”. Infrastructure is important, but also expensive. Fortunately, Big Data can help to bring down costs while improving safety. Let’s have a look:

Monitoring Dikes to Prevent Flooding

In The Netherlands, the IJkdijk has been a decade-long ...


Read More on Datafloq
5 Ways Big Data Will Improve Civil Infrastructure

5 Ways Big Data Will Improve Civil Infrastructure

On August 1, 2007 as people were driving home from work on the I-35W Mississippi River Bridge near Minneapolis, it all of a sudden collapsed. Thirteen people died and 145 people were injured. The bridge was constructed in 1967 and was expected to carry 66.000 vehicles per day. However, in 2004 an estimated 141.000 vehicles crossed the bridge on a daily basis. That’s a big difference and turned out to be too much.

The quality of the civil infrastructure within a country, determines its economic possibilities as well as a society’s wealth and quality of life. Therefore, if the infrastructure such as roads, rail tracks and bridges are deteriorating, it could have a big impact on a society. The Federal Highway Administration reported that the costs resulting from the loss of a critical bridge or tunnel could exceed $10 billion.

As the Civil Infrastructure System Task Group of the National Science Foundation once stated: “A civilization that stops investing in its infrastructure takes the first step toward decline”. Infrastructure is important, but also expensive. Fortunately, Big Data can help to bring down costs while improving safety. Let’s have a look:

Monitoring Dikes to Prevent Flooding

In The Netherlands, the IJkdijk has been a decade-long ...


Read More on Datafloq
How to Keep the Internet of Things Safe

How to Keep the Internet of Things Safe

The Internet of Things offers a lot of potential and when fully operational will have changed our lives dramatically. Once we live in a world were billion devices are connected to the Internet, our live will become a lot easier. There are already ample examples of this such as domotica in smart homes, the quantified self movement that monitors our daily behavior, smart cities were everything is monitored and even smart roads or bridges. 

But connecting all those devices to the Internet is only one aspect of the Internet of Things. Another, very important, aspect is the security aspect of the Internet of Things. How do we ensure that the data is safe and cannot be hacked? How do we ensure that connected cars cannot be hacked, as happened to Jeep, when hackers remotely killed a Jeep that was on the highway?

The security aspect of the Internet of Things is a major aspect if we want to successfully build the Internet of Things. An IoT that can will be trusted by consumers who will rely on it. Of course there is a technological aspect in securing the Internet of Things, but there is also a pracitcal aspect in securing the Internet of Things, ...


Read More on Datafloq
How to Keep the Internet of Things Safe

How to Keep the Internet of Things Safe

The Internet of Things offers a lot of potential and when fully operational will have changed our lives dramatically. Once we live in a world were billion devices are connected to the Internet, our live will become a lot easier. There are already ample examples of this such as domotica in smart homes, the quantified self movement that monitors our daily behavior, smart cities were everything is monitored and even smart roads or bridges. 

But connecting all those devices to the Internet is only one aspect of the Internet of Things. Another, very important, aspect is the security aspect of the Internet of Things. How do we ensure that the data is safe and cannot be hacked? How do we ensure that connected cars cannot be hacked, as happened to Jeep, when hackers remotely killed a Jeep that was on the highway?

The security aspect of the Internet of Things is a major aspect if we want to successfully build the Internet of Things. An IoT that can will be trusted by consumers who will rely on it. Of course there is a technological aspect in securing the Internet of Things, but there is also a pracitcal aspect in securing the Internet of Things, ...


Read More on Datafloq
7 Best Practices How Big Data Can Help the Air Force

7 Best Practices How Big Data Can Help the Air Force

In 1952, when the NSA was founded, they quickly employed thousands of cryptologists to deal with the massive amounts of data that was flowing into the organization. Since then, governments from around the world have been analyzing large streams of data. Therefore, it does not come as a surprise, that the armed forces have also been using data, and more lately big data, to gather intelligence and optimize the organization.

There are a lot of opportunities for the armed forces when they start using Big Data analytics. These advantages can be found within organizational efficiencies, regarding the equipment and personnel as well as on the battlefield.

The objective for the army is to rely on the OODA-model (observe, orient, decide and act) faster and more efficient than the enemy and Big Data can help in achieving that objective. Let’s therefore take a look at some examples how armed forces from around the world apply Big Data:

Social Media Analytics to Find the Enemy

Since the appearance of ISIS, we have seen surprised by their ability to use social media to recruit as well as to broadcast their victories. But this eagerness to share their results with the world has been turned against them in the ...


Read More on Datafloq
Datafloq Launches Big Data Company Search Engine, Enabling Organizations to Easily Find a Big Data Technology

Datafloq Launches Big Data Company Search Engine, Enabling Organizations to Easily Find a Big Data Technology

A unique index that incorporates over 4.700 Big Data companies will make it easy to find the right Big Data technology

The Hague - Big Data is changing the world rapidly. As a result, the global Big Data market is growing at an unprecedented rate. It is a volatile market, where new Big Data startups and Big Data companies are entering the market daily. Acquisitions and investments are very common, resulting in a Big Data market that is hard to follow. For organizations to develop a Big Data strategy, it has become difficult to find the right solution for their particular problem.

Not any more. Today, Datafloq, the number one Big Data platform that empowers organizations to create value with data, has launched a Big Data company index; a search engine that enables organizations to find the right Big Data technology for their specific needs.
 

Growing Market

The Big Data company index includes over 4.700 companies from around the world, ranging from large corporates to very small and innovative startups. These Big Data companies were founded by more than 5800 founders and have received over $ 24.5 billion in funding from 2800 investors from around the world. 
 

Freemium 'Netflix-Model'

Datafloq has created the first overview of the global ...


Read More on Datafloq
Big Data-as-a-Service Solutions Will Revolutionize Big Data

Big Data-as-a-Service Solutions Will Revolutionize Big Data

Big Data services offered in the cloud is nothing new. In the past years we have seen many Big Data vendors that have created Big Data solutions that can be accessed via the web to crunch and analyse your data. Recently however, we have seen the rise of a new type of offering: Big Data-as-a-Service solutions.

These solutions differ from Software-as-a-Service solutions or Infrastructure-as-a-Service solutions, as they are more or less a combination of those two. This results in a complete package for companies to start working with Big Data. But what actually is Big Data as a Service and how could your organization benefit from it?

Big Data-as-a-Service basically brings together data analytics services for analysing large data sets over the web, while also hosting all that data on scalable cloud hosting services such as Amazon Web Services. It is therefore a complete Big Data solution, accessible over the web, without the requirements for in-house solution or a lot of Big Data expertise. 

Basically it can be seen as a combination of Software-as-a-Service and Infrastructure-as-a-Service. This has a lot of potential for organizations, and especially smaller and medium sized enterprises can benefit from it, as I already wrote in my 2014 Big ...


Read More on Datafloq
How HPC and Big Data Can Bring Smart Innovation To Your Company

How HPC and Big Data Can Bring Smart Innovation To Your Company

With the amount of data that organizations have to deal, with expected to grow into the exabytes in the coming year(s), we will need better technology. Bring in High Performance Computing, or HPC in short, and a completely new world opens for you.

HPC is dramatically changing the playing field.  In the past years it has grown from an innovative feature used only by the most advanced scientific research centres, to a tool that enables organizations across industries to gain advantages of Big Data. Presidents from the USA to Russia have stressed the importance of HPC. Today more and more organizations are also valuing the qualities and possibilities of High Performance Computing. But what is HPC and what can you do with it? Let’s take a deep dive into High Performance Computing.

 

What is HPC and Why Do We Need It?

High Performance Computing refers to technology that is capable of storing, processing and analysing massive amounts of data in merely milliseconds. A HPC infrastructure basically is a large amount of clustered storage and computing servers, which are interconnected using extremely fast and efficient networking.

In itself it is not that interesting, but when you use HPC to analyse massive amounts of data, it ...


Read More on Datafloq
5 Easy Steps to Embed Big Data in Your Business

5 Easy Steps to Embed Big Data in Your Business

For many organizations it is still difficult to understand what Big Data is and how it should be incorporated in their business. This is quite understandable, as Big Data offers such radical and disruptive new possibilities as well as requires a dramatic cultural change for many organizations. The past weeks I have been thinking a lot about this and how this process could be simplified. Because when more organizations understand how to embed Big Data in their business, the more it will drive innovation and economic growth.

We know by now that Big Data can have a big impact on any part of your organization. The Big Data Use Case Framework that I developed some time ago gives a great overview of these different. But when I talk to organizations, I still get a lot of questions how they should start with Big Data and what they should do to become really data-driven.

Well, as it turns out, there is a rather simple five-step approach that could help any organization to datafy their business and processes. These steps are:


Determine the Different Processes to be Improved
Determine the Stakeholders Involved
Turn Connections and Processes into Binary Code
Start Mixing and Analyzing the Different Data Sources
Continuously Improve ...


Read More on Datafloq
How to Improve Your Customer Service with Big Data

How to Improve Your Customer Service with Big Data

Any organization with a call centre or with customer-facing employees receive complaints and compliments on a daily basis. They talk to customers via the phone, email, messaging App or face-to-face and while doing that, they create massive amounts of data. Unfortunately, very few organizations actually use that data to improve their customer service. While in fact, that data contains valuable information about your products or services.

Thanks to Big Data, organizations are not better capable of understanding their customers’ (latent) needs. It can enable them to better respond to customers’ requests, simply because they better know their customers. But then again, organizations should start collecting, and using, all that data.

Unfortunately however, many organizations just log customer complains/compliments/requests in a standard CRM program. They are not opening up the data to the rest of the organization, let alone combine it with other data sources for more insights. That is a pity, because the potential for organizations to improve their customer service with Big Data is enormous.

The below infographic, developed by ClickSoftware, offers an overview of the various opportunities for organizations when they start applying Big Data best practices. Above all, Big Data enables sales and marketing teams to have a better understanding ...


Read More on Datafloq
5 Reasons Apache Spark is the Swiss Army Knife of Big Data Analytics

5 Reasons Apache Spark is the Swiss Army Knife of Big Data Analytics

We are living in exponential times. Especially if we are talking about data. The world is moving fast and more data is generated every day. With all that data coming your way, you need the right tools to deal with the growing amounts of data.

If you want to get any insights from all that data, you need tools that can process massive amounts of data quickly and efficiently. Fortunately the Big Data open source landscape is also growing rapidly and more and more tools come to the market to help you with this. One of these open source tools that is making fame at this moment is Apache Spark.

This week I was invited to join the IBM |Spark analyst session and the Apache™ Spark Community Event in San Francisco, where the latest news was shared on Apache Spark.

IBM announced their continuous contribution to the Apache Spark community. At the core of this commitment, IBM wants to offer Spark as a Service on the IBM Cloud as well as integrate Spark into all of its analytics platforms. They have also donated the IBM SystemML machine learning to the Spark open source ecosystem, allowing the Spark community to benefit from the powerful ...


Read More on Datafloq
Big Data at Walmart is All About Big Numbers; 40 Petabytes a Day!

Big Data at Walmart is All About Big Numbers; 40 Petabytes a Day!

Some time ago, I already wrote about how Walmart has made Big Data part of their DNA. Since I wrote that post about two years ago, Walmart has continued to expand their Big Data practices. Their objective is “to know what every product in the world is, to know who every person in the world is and to have the ability to connect them together in transaction.”

Rather ambitious goals that Neil Ashe, Walmart CEO of Global E-commerce, stated some time ago. However, seeing their big data strategy, how they have progressed in the past two years and how they made big data part of their DNA it is not an unforeseeable future.

Currently Walmart processes over 40 Petabytes of data, per day. They have the second largest in-memory platform in the world. Billions of rows of data are mined every day in order to get valuable information about their 11.000 stores located in 27 countries, 11 ecommerce website or 250 million weekly customer needs.

They have build applications that rely on over 200 internal and external datasets, which are integrated into one big comprehensive Big Data platform. This means they can layer in things like weather, product sales status, pricing, inventory and ...


Read More on Datafloq
The Ultimate Battle: R vs Python; Which is Better for Data Analysis?

The Ultimate Battle: R vs Python; Which is Better for Data Analysis?

Data analysis is of course an important part of Big Data. Analyzing your data streams and combining different data sources for new insights can be done with a variety of tools and programming languages. Which programming language or tool is the right tool for the job at hand? That is an important questions and especially for newbie data analysts it is difficult to determine whether to use Python or R for your data analysis project.

Fortunately, the guys from Datacamp have created a very detailed infographic on this hot topic. The below infographic combines multiple resources to gain a better understanding which programming language should be used for what project as well as explains more about the strengths and weaknesses of both languages. It shows in great detail the differences between the two programming languages from a data science perspective, so that as a data scientist, you can make the right choice.

Both Python and R can be seen as the most popular programming languages for data analysis and both have a large community that contribute to it. Both Python and R are easy to install and to get started with. Obviously, each language has is advantages and disadvantages for different projects ...


Read More on Datafloq
Big Data Hadoop Alternatives: What They Offer and Who Uses Them

Big Data Hadoop Alternatives: What They Offer and Who Uses Them

Many people, particularly those new to the concept of Big Data, think of Big Data and Hadoop as almost one and the same. But there are frameworks other than Hadoop that are gaining popularity. The costs of implementing Hadoop can be quite high, and so organizations are exploring their other options.



Alternatives to Hadoop for big and unstructured data are emerging.

The two top Hadoop vendors, Hortonworks and Cloudera, aren't exactly suffering from an increase in competition at this point, but more organizations are discovering that Big Data comprises more than the Hadoop ecosystem. Following are some of these Big Data alternatives to Hadoop.

Apache Spark

Apache Spark promises faster speeds than Hadoop MapReduce along with good application programming interfaces. This open source framework runs in-memory on a cluster and is not tied to the Hadoop MapReduce two-stage paradigm, so repeated access to the same data is faster, plus it can read data directly from the Hadoop Distributed File System (HDFS).

It requires a lot of memory, however, because it loads a process into memory and keeps it there unless told otherwise. For iterative computations that pass over the same data multiple times, Spark excels. But with one-pass extract-transform-load (ETL) jobs, MapReduce is still tops. When all ...


Read More on Datafloq
4 Breakthrough Uses for Big Data and Why You’ll Benefit From Them

4 Breakthrough Uses for Big Data and Why You’ll Benefit From Them

Big data analytics used to be reserved for large enterprises with deep enough pockets to fund both the IT infrastructure and the data scientists necessary to derive insight from massive data collections. Sure, smaller companies might collect or gain access to big data, but getting actionable information from it was an enormous challenge.



Gaining insights from big data analytics is no longer exclusive to big enterprises.

Artificial intelligence and deep machine learning are two advances that are now allowing the general business community - including small and medium-sized enterprises - to get their arms around big data analytics. Smaller enterprises can now perform data analysis so they can visualize and make sense of data. The possibilities are endless and fascinating, and range far beyond intelligent marketing, which has long been a major focus of big data analytics. Following are 4 breakthrough uses for big data, and what these advances tell us about the future of data analysis.

1. Improved Disaster Response

The earthquake that struck Nepal on April 25, 2015 was communicated around the world immediately, prompting charities, relief services, and rescue crews to action. Big data became part of the disaster response too, as crowdsourced efforts to connect people with loved ones and ...


Read More on Datafloq
3 Ways How the Internet of Things Will Revolutionize Marketing

3 Ways How the Internet of Things Will Revolutionize Marketing

An industry that is expected to add $ 19 trillion to the Global GDP will definitely have an impact on all aspects of organizations. As such the Internet of Things will revolutionize marketing. Within a few years, thanks to the Internet of Things, a marketer’s job will look totally different than today. When products will be connected to the Internet, it means that we can start communicating with those products and that opens a whole new world of exciting possibilities for marketers:

Communicating With Your Products

Think of the possibilities that are created when brands will track your engagement with their products through the Internet. All of a sudden they have information about how you use, when you use them, who uses them etc. Information like how long you sleep and how often you toss and turn or how fast you empty your bottle of beer on a hot summer day can be very interesting for marketers. This data will completely change the way brands interact with their customers and that requires a new approach by your marketing departments.

Creating New Innovative Partnerships

When devices create so much data, it becomes interesting for organizations to partner with other companies to create exciting new campaigns. ...


Read More on Datafloq
How the Industrial Internet Will Create an Economy of Services

How the Industrial Internet Will Create an Economy of Services

The consumer Internet connects 5 billion devices. The Industrial Internet, however, will connect 50 billion devices. That will have a dramatic effect on how we do business and how we should organize our factories and companies.

The connected devices will range from energy plants to health care systems to transportation systems. These 50 billion devices will also drive a major flood of new, innovative, startups around the globe that will create new services using the data from these connected devices.

All companies will have to rethink their business models and will have to datafy their business; moving away from products and moving towards services. Services that are connected to the Internet and that will offer the owner as well as the manufacturer a vast amount of data that can be used to improve the business and the utilization rates of the connected devices. These services will change everything.

For example, they will impact the utilizations rates such as that of cars. As Michael Steinbauer, Siemens, tells in this video: “today the utilization rate of cars is maybe 15%. Half of the time it is either in search for a parking spot or waiting in a traffic jams”. Automated systems and self-driving cars will ...


Read More on Datafloq
5 Steps to Datafy Your Business and Be Successful

5 Steps to Datafy Your Business and Be Successful

In the fast-changing world, it has become common to create new words by combining two words or linking it to the –fication suffix. Examples of these include gamification and the latest is datafication. Datafication is the process of making a business data-driven. It involves collecting (new) data from various sources, storing them in a centralized location, combining them with each other and finding new insights that could lead to new opportunities as described in the Big Data Use Case framework. Datafication is relatively new phenomena and is characterised by the interaction between digital and physical objects.

The implications of the datafication of organizations and our lives are enormous. They probably are far more profound than we might expect today and therefore it is important to understand how you can datafy your business. This will enable you to be prepared for a data-driven society and be able to deal with your competitors.

Datafication and the Internet of Things

A great example of datafication is the quantified-self and the wearables trend, of which the most-talked about is probably the Apple Watch that is about to become available. The Apple Watch will enable users to generate massive amounts of data about their personal lives, analyse what’s ...


Read More on Datafloq
Why The Internet of Everything Will Change Innovation As We Know It

Why The Internet of Everything Will Change Innovation As We Know It

We live in a world where accelerated change is the only constant. Organizations and consumers are generating massive amounts of data, which will only exponentially grow in the years to come. More and more organizations see the value of a Big Data strategy and are changing their company towards a data-driven, information-centric organization. As a result, we will see an explosive growth of connected devices that will generate and unfathomable amount of data. This data can and will be stored, analysed and used in decision-making within the most innovative organizations.

We can therefore say that we are at the brink of a completely new, connected world; a connected world where billions or even trillions of devices are connected to the Internet and to each other, in real-time. This connected world will change how organizations should be managed. It will change how organization should approach innovation. And it will change how organizations should connect with their customers. The Internet of Everything will change innovation as we know it.

Cisco defines the Internet of Everything as “bringing together people, process, data, and things to make networked connections more relevant and valuable than ever before—turning information into actions that create new capabilities, richer experiences, and ...


Read More on Datafloq
Five Ways How Startups Can Leverage Big Data

Five Ways How Startups Can Leverage Big Data

Big Data is not only for large organizations and you don’t need to have terabytes or petabytes of data to develop a Big Data strategy. Instead, any organization, small or big, startup or public company, can leverage the exponential data growth the world is currently experiencing.

According to the below infographic, developed by Bluenose, many startups are experiencing a data growth. In fact, nearly 75% of startups say that their collection of data has increased significantly in recent times. Unfortunately, 35% of them haven't even considered leveraging a Big Data solution, which is a missed opportunity for them.

Big Data offers multiple advantages for startups. Analyzing and managing large data sets allow startups to predict and reduce churn, and retarget new and existing customers accordingly. Many Big Data solutions are scalable and live in the cloud, and they're also designed to be user friendly and don’t have to be expensive. So why wait as a startup in developing a Big Data strategy?

The opportunities are almost endless for startups and the five main ways a startup can leverage Big Data are:


Personalize the customer experience and identify key customers in order to personalize your messages, your products and your services;
Predict and reduce churn by ...


Read More on Datafloq
How Greyhound Buses Are Embracing the Internet of Things

How Greyhound Buses Are Embracing the Internet of Things

Greyhound Lines, Inc, is a large inter city bus common carrier that serves over 3.800 destinations across the United States, Canada and Mexico. It is the largest inter city bus transportation company in the United States and it is rapidly driving towards the Internet of Things. Currently they are undergoing a dramatic business transformation, embracing the latest technologies to improve efficiency and safety of the company.

Two years ago they started a five-year transformation program to completely change the organization and prepare it for the digital era. They are currently in year 2 and this covers the whole spectrum of IT, which includes implementing the busses with a wide range of sensors and new mobile applications. Sensors will among other measure engine performance, break performance and how the driver performance.

Traditionally, all those sensors required special devices to get the data, which in addition belonged to the manufacturer of the engine. With API’s, Greyhound Lines obtains a lot more information on how the bus driver is driving and regarding the safety of the passengers. This completely changes the game for the company.

The main aspect that they focus on is safety, as it is core number one value at Greyhound Lines. This means ...


Read More on Datafloq
How Big Data is Driving the Next Wave of Mass Customization

How Big Data is Driving the Next Wave of Mass Customization

Today I was in Johannesburg, South Africa, to give a keynote speech at the Retail Africa event. The below is an excerpt of my presentation.

We live in a world with information overload. Every day so much new data is created that customers feel overwhelmed and they don’t know what to do with all that information. In fact, it is estimated that this year we will create almost 8 Zettabytes of data. That is the equivalent of 18 million libraries of congress of data or 2 trillion DVD’s.

So, while many marketers have so much data at their fingertips, they just don’t know how to use it. This is a pity as research has indicated that 73% of customers prefer retailers who use personal information to create a relevant experience. Thanks to all that amount of data, retailers can now better target customer and offer the right product for the right price via the right channel.

The Way Forward is Mixed Data

In the past years Big Data was all about obtaining as much data as possible and the perception was that you require massive datasets to gain insights from those data sources.

Organizations however start to see that the most important aspect of ...


Read More on Datafloq
Real-Time Analytics and the Internet of Things are a Perfect Match

Real-Time Analytics and the Internet of Things are a Perfect Match

The Internet of Things is here and is here to stay. With billions of devices already connected and another trillion coming our way in the next decade, organizations should prepare themselves for a data flood. A data flood that can generate real-time insights in how the organization is performing. Using a plethora of data sources, such as smart meters, in-store sensors, medical devices, automotive sensors and wearables, organization can start to optimize their business.

According a recent customer survey by Vitria, that is exactly what a lot of the organizations surveyed are already doing. The executives across the enterprise, consumer and industrial sectors see massive opportunity with real-time Big Data analytics to drive business outcomes for their Internet of Things initiatives. In fact, 48% of the companies surveyed are already actively working on a real-time analytics project for the Internet of Things. Internet of Things analytics is seen as a core investment strategy and predictive maintenance is the leading business need for these organizations.

Real-time analytics for Internet of Things projects is rapidly becoming mainstream and according to the survey; it is becoming a strategic area for investments. Analysing the data streams in real-time can be seen as a requirement for organizations ...


Read More on Datafloq
Why All Governments Should Hire a Chief Data Scientist Just Like the US Did

Why All Governments Should Hire a Chief Data Scientist Just Like the US Did

Last week, the White House announced that DJ Patil will be the first ever Chief Data Scientist and Deputy Chief Technology Officer for Data Policy. His job will be to act as an evangelist for new applications of Big Data across all areas of government, with a special focus on healthcare. The US government has come a long way regarding Big Data and President Obama is the most data-driven President ever in office. During last week’s Strata Conference in San Jose, DJ Patil shared with the audience what Obama has done as well as what the US government plans for the coming years with regard to Big Data.

First of all, Obama has created dashboards at Federal level to monitor how major IT technology investments progress. This enables the US government to better control expensive IT projects, something that many governments struggle to achieve.

In addition, he created Data.gov. This platform hosts over 135.000 data sets that can be used by data scientists to explore and experiment with. Especially for new data scientists this is a great place, along with other public data sets from other countries around the world, to start learning data science, which in turn will help you the ...


Read More on Datafloq
How To Become a Data Scientist And Get Hired

How To Become a Data Scientist And Get Hired

The Big Data Scientist is said to be the sexiest job of the 21st century and it is likely to be very well paid. Salaries are in the high $ 100k per year and there is a strong demand for the best Data Scientists. A lot of the 20th century jobs on the other hand will disappear thanks to robotics, artificial intelligence and machine learning. How should you ensure that you can also obtain one of those sexy jobs, how can you become a data scientist and get hired?

Well, it all starts with obtaining the right skills, surprisingly. The challenge however with becoming a data scientist is that you need a long list of skills to get hired. Some time ago, I already published a typical job description of a Big Data Scientist and other infographics show that it is a very long road to become a Big Data Scientist. But with an expected shortage worldwide of 56.000 Big Data Scientists by 2020 in the UK and 140.000 – 190.000 in the USA, it is definitely worth pursuing.

Be Able to Choose the Right Technology

There are a multiple different tools and techniques that a Data Scientist should master. Of course I ...


Read More on Datafloq
Why Is Data Quality Holding Up Data-Driven Business Decisions?

Why Is Data Quality Holding Up Data-Driven Business Decisions?

Only a few years ago, the data in the world used to be primarily structured data that was relatively easy to analyse. However, in the age of information overload, most of the data has become unstructured or semi-structured data. In fact, currently 90% of all data is unstructured data such as documents, videos, images, voice records etc. Combining these different data sources and putting it into context in order to get, real-time, insights is a big challenge for business leaders.

According to a survey by ClearStory Data among 500 business leaders at the Strata Conference in October 2014, data variety is the biggest challenge they need to overcome in order to become successful with Big Data. Almost 50% of the business leaders surveyed indicates that they need to blend 8-15 different data sources. Combining and mixing those data sources takes time and for over 2/3 of the business leaders surveyed this takes too long for them to make it useful. Combining multiple data sources with different varieties provides great insights for organizations, but doing so correctly is not very easy. It requires the right tools and employees to manage it correctly.

As a result, for many of the organizations surveyed, the challenge ...


Read More on Datafloq
Five Signals Your Organization Is Ready For Big Data

Five Signals Your Organization Is Ready For Big Data

To start or not to start with Big Data? If you are a competitive organization that also wants to remain in business the coming five to ten years, the answer should be clear. But when is your organization ready to start with Big Data? When is everything in place to successfully develop and implement a Big Data strategy? When are you set to embark on the Big Data challenge? During my work as Big Data strategist I came across five signals that indicate that your organization is ready for Big Data:

1) A Shared Understanding of Big Data Exists Within Your Organization

If there is no shared understanding of what Big Data can do for your organization, it is of no use to start. Big Data means something different for every one, every organization and every industry. There are so many possibilities with Big Data and if everyone in your organization has a different idea about what it means, it will be difficult to get everyone aligned and to successfully implement a Big Data strategy or to at least start with a Proof of Concept.

2) Business Intelligence Is Part of Decision-Making Processes

Starting with Big Data will become very difficult if you have ...


Read More on Datafloq
9 Generic Big Data Use Cases to Apply in Your Organization

9 Generic Big Data Use Cases to Apply in Your Organization

Big Data means something different for every organization and every industry. What Big Data can do for your organization depends on the type of company, the amount of data that you have, the industry that you are in and a whole lot of other variables. Whenever I advise organization on their Big Data strategy, this is the main problem; there are so many different possibilities and often it is a struggle to find the right use case to develop into a Proof of Concept. That’s why I have developed the Big Data Use Case framework, to help organizations understand the different possibilities of Big Data and what it can do for their business. The framework divides 9 generic Big Data use cases into three different pillars:


Your Customers;
Your Product;
Your Organization.


For each pillar there are three Big Data use cases that can be defined, which are relevant for all organizations across all industries. The framework looks as follows and let’s discuss the Big Data use cases one by one:

360 Degrees Customer View

Developing a complete view of your customer is important for every organization, as it helps to understand what your customer wants, what the needs and preferences are and how the customer ...


Read More on Datafloq
10 Predictions for the Big Data Analytics Space

10 Predictions for the Big Data Analytics Space

The end and beginning of a new year are always filled with trends and predictions for the new year and I have one more interesting infographic that I would like to share with you. This infographic has been developed by Aureus Analytics and covers 10 different Big Data Analytics trends. Let’s briefly discuss a few of them

Extreme Real Time Big Data Analytics

With the growth in data streams from multiple sources comes also the requirement for analysing that data in milliseconds. Especially relevant for online companies with dynamic websites that want to determine the profile of a visitor the moment he or she enters the website in order to offer a relevant and personal website. It is also used in the Online Ad business, where the advertisement needs to be matched with the profile of the visitor in milliseconds.

Growth of Data lakes

The growth of data lakes is also a trend I also foresee and discussed in my trend briefing for 2015. Data lakes contain any sort of data, structured and unstructured, and enable the user to easily combine data and obtain a single source of truth. Data lakes are also a lot cheaper to maintain, instead of different data sources stored ...


Read More on Datafloq
Why Big Decisions Require Big Data Analytics

Why Big Decisions Require Big Data Analytics

Great insights are achieved when you combine different data sources. This Mixed Data approach enables organizations to have a better understanding how their organization is performing and what the different areas of attention should be. Such


Read More on Datafloq
7 Factors to Consider When Choosing Between Public or Private Cloud

7 Factors to Consider When Choosing Between Public or Private Cloud

By now it should not come as a surprise that Big Data and the Cloud are two trends slowly converging into one major trend. Many organizations are thinking of a cloud solution for their IT and that is not a surprise. In fact, already in 2012 KPMG found that 81% of organizations were either evaluating cloud services, planned a cloud implementation or had already implemented a cloud solution. The benefits for a cloud solution are quite clear: a cloud offers organizations scalability, flexibility, agility, elasticity as well as a well-distributed workload across your IT.

The question is not so much if your organization needs a cloud solution; the question is more what sort of cloud solution aligns most closely with your business strategies. There are three different possibilities when talking about the cloud: a private cloud, a public cloud or a hybrid cloud using both public and private solutions. Deciding which solution is the best for your organization is not that easy; therefore here are seven factors, in alphabetical order, to consider when choosing between the public, private or a hybrid cloud:

Available Budget

Of course budget is an important factor to consider. Installing for example a large Hadoop cluster on premises requires substantial hardware, employees to operate and manage it as well as maintenance costs. A public cloud often works with a pay-as-you-go system, which ensures that you only have to pay for the equipment used, sometimes even charged on an hourly, or even shorter, basis.

Big Data Maturity of the Organization

The level of Big Data maturity within your organization also affects the choice between a public or private cloud. If you are just starting with a small Proof of Concept, it is no use to install a complete Hadoop cluster on premises. Quickly using a public cloud can be a lot cheaper and more efficient and effective. However, when you are already very mature, you may come to the conclusion that an in-house operated Big Data cloud can be easier and gives you more advantage than a public cloud.

Commitment to Third Parties

A public cloud often does not involve long-term contracts, especially with the now-a-days popular pay-as-you-go model. After your subscription ends, there is often no obligation to stay with that party. A private cloud however requires hardware from a certain provider. This is often expensive, could come with a maintenance contract and comes with redemption of several years. On the other hand, with a public cloud you have less control over how your data is managed and how the IT hardware is monitored. This is completely done by the third party and requires a strong trust in that third party.

Compliance and Security

What type of data do you deal with? If you deal with highly sensitive and private data, it might be that storing it in the public cloud is not allowed from a compliance perspective. When a strict control over your data is required, a public cloud with servers in other countries might not be the best solution. A private cloud solution enables organizations to actively restrict access externally and internally. However, most of todays cloud providers offer maximum-security options, of course they come at a price, to ensure that your data is stored securely. Depending on company policies one or the other is more suitable.

Performance Requirements

How fast do you need to be able to access the data? A private cloud, which is within a company’s firewall, means a lot faster access to your data. Public clouds on the other hand are dependent on the transfer rate of your Internet Service Provider. Especially with huge volumes of data, this could be a problem as super-fast fibre optic Internet is not yet widely available. In addition, a private cloud is often a lot more customizable in terms of hardware performance, storage performance and network performance as all the equipment is owned.

Scalability Requirements

A public cloud can scale exponentially within minutes. Hosting providers have the available servers ready 24/7 and generally it requires only a few simple steps to scale up. It can even be done automatically, so that it scales up and down as per the requirements. This flexibility in available storage can be very beneficial if you are dealing with volatile data streams that need to be processed and stored. When you have a private cloud on premises, scaling up often requires additional hardware which can both be time-consuming and expensive.

Type of Organization

The type of industry that you operate in or the type of organization you are (governmental or commercial) will determine what is allowed and useful for storing your data. If you deal with a lot of governmental data, it might be required that you store, at least some of, the data on-premises and not in a public cloud.

In addition, if your core competency is technology, it might be wise to use an on-premises cloud, but if IT is a far from your bed show, it might be wise to outsource that part of your company to a hosting provider.

The Choice is Yours

Depending on your specific situation, the size and type of your company, the available budget, your requirements and the type of data you need to store and analyse one of the three options might be the best solution. Of course, the best option might very well change over time, therefore it is important to constantly use the feedback from your employees, customers, organization and IT systems to determine what’s best for you.

What Are The Data Science Trends for 2015?

What Are The Data Science Trends for 2015?

The beginning of the year, as well as the end of the year, is traditionally the moment that a lot of companies make prediction of the year to come. So did I, when I wrote down my five Big Data trends for 2015, focusing on the major trends in the Big Data space that I foresee. Whenever a new infographic appears on Big Data trends, it is always good to share it with our Datafloq readers. This infographic was developed by Crowdflower, a crowdsourcing company and focuses on where data science will be heading in 2015.

They have come up with 8 trends in the data science space, some more spectacular than others. The rise of the Chief Data Scientist is a logical one, when more and more large organizations are developing and implementing Big Data strategies. The Chief Data Scientist, or Chief Data Officer, will be responsible for the overall big data strategy within an organization. The Chief Data Scientist will also be responsible of ultimately over seeing the data scientists within an organization. Crowdflower predicts that every team will get their own in-house data scientist teams, instead of a separate department of data scientists. This is of course only attainable by large organizations, which are able to hire multiple data scientists and analysts.

The prediction on open data is definitely a valid one. More governments (local, regional and national) are opening their data sets for organizations to use. Public data market places are becoming more available and more organizations see the benefits of using these data sets.

Another interesting trend they foresee is Data of Everything, which means the data coming from the Internet of Things. All the data that is created by devices connecting to the Internet will be form a flood of data that by 2020 will result in 40% of all data to be sensor data. Organizations need to be ready for this massive amount of data that is heading our way.

Crowdflower has gathered an interesting list of 8 data science trends for 2015, which can be viewed in below infographic. What do you think of these trends? Please let us know in the comments.

Data Science trends

Three Big Data New Year’s Resolutions Organizations Should Make

Three Big Data New Year’s Resolutions Organizations Should Make

The New Year has started and 2015 promises to be a big year for Big Data and everything that is linked with Big Data. The Internet of Things is expected to take of dramatically this year, Big Data Security Analytics will become common language in the Boardroom and the available data in the world will grow with another 8 Zettabyte this year. 2015 will therefore be an exciting year when talking Big Data and organizations should be ready for the data revolution we are already facing. Therefore, here are three Big Data New Year's resolutions organizations should make in order to stay ahead of the pack and be ready for a data-driven economy:

Stop Throwing Away Your Data

If you really want to achieve the best insights from your data and are seriously thinking about developing and implementing a winning Big Data Strategy, you should stop throwing away data and start collecting and storing as much data as possible. No more deleting of data, as it might very well be that you will need that data in the future. The excuse that at this moment that data is useless or you cannot analyse it, is not valid. Things will change in the future, allowing you to analyse it and achieve valuable insights from it.

Even if you are generating high volumes of transaction-level data that require quite some storage, don’t throw it away. Storage has become so cheap these days, that storage cannot be an issue here. Deep history data can help you a lot when you want to optimize your processes. Anomalies that could indicate improvement points might not show when you have data of a few days, but might appear over a longer time such as months or years. Not having that data will therefore be a serious missed opportunity for your organization.

The above is not only valid for manufactures, but also for ecommerce companies. Log data can be very valuable to understand how to improve your web shop, so start collecting all kinds of data such as search query data, shopping basket data, visitor profile data etc. You should be able to retrieve exactly how someone moved around your website, what the detailed characteristics were of the visitor and what he or she did online. On the long run, this data can be extremely valuable and can be the difference between winning and losing.

Keep Your Data Secure

The cloud is another big trend within the Big Data market. More companies are moving to the cloud, whether it is the public cloud or private cloud. Moving your data to the cloud enables you to scale easily and have access to it from anywhere. But is also becomes more vulnerable if the data is not protected and encrypted correctly.

But that’s not all. Also those companies that do not make use of the cloud are vulnerable for data breaches. Organizations should therefore do whatever it takes to keep their data secure and private. Data security has to be taken very serious by all organizations, seeing the different data breaches of 2014. Start using the latest tools to protect your data as well as monitor your IT network in real time. Implement the right online and offline processes to know what’s going on within your IT systems and to be certain that the data is kept secure and cannot be hacked. Big Data Security Analytics should be part of your Big Data strategy and should be taken seriously.

Start with Your Proof of Concept

Many of the large corporations have started with Big Data in the recent years. The have implemented winning Big Data strategies and are working hard to create data-driven, information-centric cultures within their organization. If you are a smaller company and/or have not yet started with Big Data, 2015 is the year to make your first move. If you keep postponing a Big Data strategy, you will face a difficult time with your competitors who do move forward with Big Data.

The best way to start with Big Data is to develop a Proof of Concept as this will help you understand what Big Data means for your organization and how you can make the most of it. Big Data means something different for any organization and industry and therefore, a Proof of Concept will offer you valuable lessons and knowledge how to propel your company with a Big Data strategy.

2015 is the year that Big Data will become mainstream. The technology is ready for it, universities have created Big Data programs to deliver the talent, storage has become cheap and the consumer is ready for it. These three Big Data New Year's resolutions will help you prepare for the Big Data era and be able to face your competitors who also work on a Big Data strategy. Don’t wait any longer and prepare your organization for the data revolution.

Your Wearable Christmas Present Is All About Big Data

Your Wearable Christmas Present Is All About Big Data

Christmas is the time to give presents and according to Google this year’s top present will be wearables such as the Fitbit or Jawbone. Google analysed the trending gift searches for 2014 and the research revealed that wearable technology is gaining serious attention this month. Also Staples expects that wearables will be hot this year. According to them, mainstream adopters will look for health trackers such as the Jawbone or the Fitbit, while early tech adopters will likely move to smartwatches. Another survey by Currys PC World states that wearables are featured in 1-in-3 Christmas present wish lists. Clearly wearable technology appeals to the consumer market, so a lot of people will probably find some sort of wearable gift underneath the Christmas tree.

Wearables are of course fun to get and fun to use. According to Rohit Yadav, there are four ways that wearables will improve our lives:

  1. They will keep us fit and healthy; when we receive information on how we behave during the day, we become aware of what’s going on. If we see that we only walk 500 steps a day, this might be a trigger to walk more often and walk further. By making us aware of our, positive or negative, behaviour, wearables have the power to change our lifestyles.
  2. They will save lives: we see more and more wearables that are primarily focused on monitoring how our body performs such as our heart rate, blood pressure or your blood oxygen levels. When this information is collected over time, a doctor will get a much better picture of your overall health. In addition, when something does goes wrong, caregivers can be alerted in real-time.
  3. They will keep us save: average smartphone users check their phone 221 day, according to a study by Tecmark. Taking into account that on average we sleep 8 hours, this means that we check our phone every four minutes when we are awake, also when we are on the road. New wearables such as smartwatches or smartlenses from Google, could reduce this amount considerably, giving us more time to pay attention where we are going when we are underway.
  4. They make things fun: knowing how fast you have run and how many calories you have burned can be a fun way to start the competition with your friends and can make running a lot more fun for many people.

The biggest market in wearables is still health and fitness, but also wearables for the office are becoming popular. And of course there are multiple options for wearables in the communication industry, glamor industry or security industry. Below image offers a great overview of the world of wearable technology:

World of Wearables

 

But what does these wearable devices mean from a Big Data perspective? Of course, these wearables devices are collecting all kinds of data, which is not just for your personal information and to keep you healthy, but will also be used by the producer to improve the product over time. In addition to that, the producers of these products, but also apps that can be used to track yourself, can extract all kinds of trends out of the data that says something about the users.

As the below video shows, trends such as sleeping patterns or walking patterns based on geography are quite easily to extract out of the data. The same video predicts that 485 million wearables will be sold annually by 2018, so a lot more data is coming our way that will contribute to global insights in human behaviour. That massive data set is of course a gold mine for healthcare providers, insurers and researchers.

Wearables are therefore all about Big Data, contributing massively to the data flood that is coming our way in the coming years. Of course, the data can be used by consumers to better understand their lives, but it is also very interesting to aggregate the, anonymous, data and obtain a helicopter overview of human behaviour around the world.

How Big Data Security Analytics Will Protect Your Company

How Big Data Security Analytics Will Protect Your Company

We live in an era of increased cyber threats. With dozens of companies being hacked every year, and cyber crime rising to new levels with the Sony Hack, it is time for organizations to start protecting themselves. Big Data Security Analytics can help companies understand what is happening within their company and can help them take action when it is needed most. But what is Big Data Security Analytics and how can it help protect your company?

Unfortunately, protecting your organizations from (would be) hackers is difficult. In fact, according to ESG research, 62% of organizations believe that security management has become more difficult over the past two years. There are several reasons why security management has become more difficult including more complex IT systems that are required, less talent that is available to develop the security measures and the threat landscape has become a lot worse with more sophisticated and successful hacks carried out in the past years.

One of the reasons for this finding is that organizations have not yet adopted the new Big Data Security Analytics, but are still relying on processes and technologies from yesterday. Very few organizations correctly protect their business, let alone store their sensitive documents or passwords correctly, as was the case with the Sony hack.

According to the same ESG research, organizations face quite a lot of challenges when it comes to incident detection and security analytics. Almost 40% of the companies surveyed said that there is a lack of adequate staff in security operations and incident response. In addition, 35% said that there were to many false positives, due to the lack of intelligent analytics resulting in too much noise. Finally, almost 30% said that monitoring depends on too many manual processes and uses tools that are not integrated with each other, resulting in an incomplete overall picture.

It is time that organizations face the facts and start protecting themselves for these threats, as there are quite a few risks for organizations that get hacked. Apart from data and property that gets stolen, it causes major reputational damage that could significantly harm your company when your customers decided that they don’t want to deal with a company that loses their private information.

The Rise of the Chief Data Security Officer

Organizations should therefore introduce the Chief Data Security Officer that is responsible for Big Data Security Analytics. The CDSO should be an important role within the board and they should look at combatting persistent threats and mitigating exposure of the company’s IT systems to large cyber attacks. They should focus on reducing the possibility of fraud on business processes, preventing hacktivism on their networks as well as identifying insider threats.

The Chief Data Security Officer should create an environment that is capable of dealing with large quantities of data. Big Data Security Analytics involves Terabytes of data including log information from monitoring your network, database information, identify information and all kinds of other system data that needs to be analysed in real-time to know what is going on. Within a true Big Data Security Analytics environment, an organization should be able to combine security intelligence with business transactional data as well as unstructured company data such as emails to obtain a complete picture of what is going on. This will allow you to find all kinds of unique patterns and anomalies that actually might be, for example, a very slow moving attack that in the end could do a lot of harm.

A New Approach to Digital Security

The introduction of the Chief Data Security Officer is just the beginning. The world of digital security is changing rapidly and organizations should therefore evolve as well. Cyber criminals are constantly changing their tactics, finding new ways to attack companies, so if a company refuses to stay up-to-date, they are almost asking to be hacked. This new reality requires a new approach to security.

Protection your company should therefore be focuses on prevention, detection and response. On the one hand, you should make it as difficult as possible for criminals to hack your systems. Encrypt your documents, and especially your passwords, and use firewalls to protect your systems from outside intruders. On the other hand, focus on monitoring and detection to know what is going on within your network and company. Combine many different, real-time, automatic tools to discover patterns and anomalies that could expose an intruder, identify offenses as well as security incidents that require your attention. Remove any manual activities and make use of automated intelligent processes that analyse deep internal and external security intelligence. Once a security threat is detected, you should focus on response in order to minimize the possible damage.

Big Data Security Analytics is a difficult field, which involves large amounts of data sets, huge volumes of data, smart algorithms and extensive encryptions. The brightest minds and/or smart software tools should be used and it should be on top of the agenda for every company. For many organizations it will be an expensive investment they have to make, but not doing it could turn out to be a lot more expensive.

The Future of Big Data? Three Use Cases of Prescriptive Analytics

The Future of Big Data? Three Use Cases of Prescriptive Analytics

In 2014, Gartner placed prescriptive analytics at the beginning of the Peak of Inflated Expectations in their Hype Cycle of Emerging Technologies. According to Gartner, it will take another 5-10 years before prescriptive analytics will be common in boardrooms around the world. But what is prescriptive analytics, how can we use it and how can it help organizations in their decision-making process?

Prescriptive analytics can be seen as the future of Big Data. If we see descriptive analytics as the foundation of Business Intelligence and we see predictive analytics as the basis of Big Data, than we can state that prescriptive analytics will be the future of Big Data. Earlier, I already explained the difference between these three types of analytics, but let’s have a small recap: descriptive analytics means looking at historic data, ranging from 1 minute ago to years ago. It can be compared as looking in the rear mirror while driving. Predictive analytics means using all that data to make a prediction about where to go; it is the navigation that tells you how to drive and when you will arrive. Prescriptive analytics is the self-driving car, that knows exactly what the best route is based on infinite data points and calculations. Not surprisingly, Google’s self-driving car makes extensive use of prescriptive analytics.

Prescriptive analytics uses the latest technologies such as machine learning and artificial intelligence to understand what the impact is of future decisions and uses those scenarios to determine the best outcome. With prescriptive analytics it becomes possible to understand and grasp future opportunities or mitigate future risks as predictions are continuously updated with new data that comes in. Prescriptive analytics basically offers organizations a crystal ball. Prescriptive analytics will become really powerful when it has developed into a stage where decision makers can predict the future and make prescriptions to improve that predicted future, without the needs for Big Data scientists.

Although prescriptive analytics is really still in its infancy, we see more and more use cases being developed. Also several Big Data startups focus especially on prescriptive analytics. The most well know is Ayata. They use patented software to predict what is going to happen, when it is going to happen and why it is going to happen. They focus primarily on the oil and gas industry, but there are more use cases of prescriptive analytics. Prescriptive analytics is used in scenarios where there are too many variables, options, constraints and data sets. Without technology it is too complex for humans to efficiently evaluate those scenarios. Also when experimenting in real-life is too risky or expensive, prescriptive analytics can come to rescue. Let’s have a look at three of the possible use cases:

Travel and Transportation Optimization

A key characteristic of prescriptive analytics is the need for many large data sets. The travel industry is therefore an industry that sees a lot of potential in the latest addition of analytics. Online travel websites, such as airline ticketing services, hotel websites or car rental websites, have turned to prescriptive analytics to sift through multiple complex iterations of travel factors, purchase and customer variables such as demographics and sociographics, demand levels and other related data sources to optimize their pricing and sales.

Other applications in the travel industry are segmenting (potential) customers based on multiple data sets to understand how to spend the marketing dollars. More accurate targeting of course results in higher conversion rates and to be able to make detailed segments, a lot of different variables are required. The InterContinental Hotel Group for examples uses 650 variables to determine the best price/product combination for the right customer.

Another use case of prescriptive analytics is route optimization for the logistics industry. UPS is the best example of this and be analysing and combining hundreds of data source can push 10.000s route optimizations per minute to all of their trucks. This saves the company millions of dollars on fuel a year.

Oil Production Through Fracking

In the past years, fracking has taken an enormous flight, especially in the United States. In 2013 alone, $ 31 billion was spent on suboptimal frack stages across 26,100 U.S. wells. In order to know where to frack, make the process safer and to optimize the fracking process, massive data sets (up to petabytes of data) are required. Data sets such as sounds (of fracking and drilling), images (seismic, well logs), videos (cameras monitoring the fracking and sensors measuring all kinds of variables), text documents (notes by drillers) and other kinds of data have to be analysed in real-time to recommend the most optimal fracking location and process in order to have the best result.

Improving the Healthcare Industry

Also the healthcare industry deals with massive amounts of different data sets that need to be analysed. When healthcare providers combine data sets such as patient records, medicine information, economic data, demographical and sociographical data, health trends, hospital data etc. they will be able to offer better healthcare for less money, they will be able to improve future capital investments for new facilities or hospital equipment and improve the efficiency of hospitals.

Combining so many different data sets can also be used to offer doctors recommendations in the best possible treatment for a patient. Thanks to combining and analysing multiple data sets, the Aurora Health Care Centre was able to improve healthcare and reduce re-admission rates by 10%, thereby saving $ 6 million annually.

Also pharmaceutical organizations can benefit from prescriptive analytics by improving their drug development and reduce time-to-market for new medicines. Drugs simulations can improve medicines faster and it becomes easier to find the right patient for clinical trials based on multiple variables.

Prescriptive analytics is the future of Big Data, but it is still a long way away before it will be common language. The potential is enormous, but it also requires massive amounts of data to be able to make correct decisions. Only a handful of organizations and industries have that amount of data and data sets to make something useful out of it with prescriptive analytics. However, in 5-10 years will be as normal as Business Intelligence today. 

The Sony Hack; Are Organizations Ready for Digital Combat?

The Sony Hack; Are Organizations Ready for Digital Combat?

We have seen numerous hacks in the past years, but the Sony hack of December 2014 has topped them all. The hackers brought a multinational corporation down on their knees when Sony decided to cancel the release of their movie The Interview. Apart from the massive amounts of leaked internal documents that show how the relationships in Hollywood are, also all kinds of financial documents are now part of the public domain. Of course, some of the leaked information was quite interesting, but most of it an outrageous invasion of privacy, possibly damaging individual reputations. Apart from all that, the hack will probably cost Sony hundreds of millions of dollars in damage reputation and lost revenues.

The tremendous hack of Sony, and the result that the hackers achieved, stopping the release of the movie The Interview, can set an example for other digital criminals to attack organizations. With the enormous amount of data that organizations create, the vast array of IT systems that they use and the numerous amounts of employees that have access to all kinds of data, we will probably see a lot more hacks in the future.

According to the FBI, hackers are replacing terrorists at the most severe threat for the USA and experts are warning that a major digital attack on for transportation or the energy grid is just a matter of time.

That’s why I foresee a bright future for Big Data Security Analytics, as I described in my Big Data trends forecast for 2015.  As described by Alvaro Cardenas, the objective of Big Data Security Analytics is “to obtain actionable intelligence in real time". These actionable insights can then be used to prevent or stop a digital attack at your organization. Perhaps the Sony hack will cause a spur in the development of Big Data Security Analytics, as organizations are going to do what it takes to prevent this from happening to them as well.

The below infographic, developed by Whoishostingthis, shows some of the major attacks that have been done in the past 2 years (the infographic is from before the Sony attack) and what governments around the world are doing to prevent such attacks. The United States for example, is working hard to develop Intrusion Prevention Systems to reduce the amount of malicious traffic that comes in as well as build a cyber workforce that is capable of combatting with these hackers. Also the European Union is working hard on improving the region’s cyber defences. There is still a long way to go, but with better regulations, harsher legal consequences and of course better Big Data Security Analytics, a more secure world is waiting for us.

Major Cyber Attacks: Are We Prepared for Digital Combat?

Three Ways Pattern Analytics Will Grow Your Business

Three Ways Pattern Analytics Will Grow Your Business

Pattern Analytics can be defined as a discipline of Big Data that enables business leaders to understand how different variables of the business interact and are linked with each other. Variables can be of any kind and within any data source, structured as well as unstructured. Such patterns can indicate opportunities for innovation or threats of disruption for your business and therefore require action.

Finding patterns within the data and sifting it out is difficult. Machine learning can contribute in helping us humans find patterns that are relevant, but too difficult for us to see. This enables organizations to find patterns they act on. Business leaders can learn from these patterns and use them in their decision-making process. Business leaders therefore should rely less on their gut feeling and years of experience, and more on the data.

Pattern Analytics does not require predefined models; the algorithms will do the work for you and find whatever is relevant in a combination of large sets of data. The key with pattern analytics is automatically revealing intelligence that is hidden in the data and these insights will help you grow your business. There are of course multiple applications of pattern discovery, but I would like to focus on three important ones: pattern discovery with structured sensor data, pattern analytics to keep your company secure and optimizing your sales with pattern discovery.

Sensor Data and Pattern Discovery

Operational data from sensors generated by machinery can be used for pattern analysis. In fact, pattern analytics is especially relevant for sensor data and time series data. Time series data are data points consisting of successive measurements made over a time interval. Such data can expose a lot of patterns about the product related to for example energy usage, maintenance requirements and/or possible safety issues. When analysing these structured data sets with pattern analytics, you can find anomalies, commonalities and trends that reveal insights that otherwise would remain unnoticed.

These insights can be used to make predictions related to maintenance (based on sensor data you can predict when maintenance is required for your machines), usage (what are for example the patterns of people using the building so that you can optimize the energy usage based on the expected amount of people in a building) or safety (finding those patterns that could predict a possible hack in the information systems). Especially the latter one is relevant, as any company has to deal with protecting their information systems so let’s take a close look at that one.

Patterns Analytics to Keep Your Company Secure

Companies can leverage the large quantities of structured and unstructured data within their organization to analyse and detect in real-time any possible security threat. Pattern discovery is especially relevant for this as it can reveal what is happening within your IT systems. What are standard patterns that can be ignored and when does something occur that is relevant from a security perspective? Pattern analytics can reveal those standard patterns and compare them with new, unexpected, patterns. The problem is, however, that in order to detect those patterns, you need to collect massive amounts of data on how systems behave, as well as employees behave while using those systems. You have to look at a vast array of data sources, which will keep on growing, to be able to find meaningful patterns and anomalies. Whenever attack patterns have been detected, an organization can take action to stop it or otherwise prevent harm being done.

Driving Your Sales With Pattern Discovery

When you have sufficient data sources and the right algorithms, it is possible to automatically discover patters that affect your sales. Pattern analytics on customer and sales data can therefore indicate market trends, customer interests, latent needs and reveal future sales trends. In addition, pattern analytics can show the top selling items based on geography and interest and news happening around the world. This information can be used by decision-makers to better price products, ensure sufficient stock and as a result drive more sales.

There are many examples of pattern discovery in the retail industry and a well-known example, as discussed before, is that of Walmart’s Strawberry Pop-Tarts. When combining multiple data sources, Walmart found a significant pattern that indicated that whenever there was a hurricane warning, customers would buy considerably more Kellog’s Pop-Tarts. As a result, they introduced the policy to put Pop-Tarts near the entrance during hurricane season.

Pattern analytics and discovery alone is useless. It always requires a second step to do something with the patterns discovered. Whether it is planning or postponing maintenance, stopping a cyber attack or changing your retail assortment and prices. Pattern analytics offers valuable insights for your business, but it does require an action to be taken. Only then can pattern analytics help grow your business.

Image: QDBVE

Tesco and Big Data Analytics, a Recipe for Success?

Tesco and Big Data Analytics, a Recipe for Success?

Tesco is one of the largest retailers in the world and is originally from the United Kingdom and founded in 1919. Currently they have shops in 12 countries under different brands. In the latest fiscal year (‘13/’14) the had $ 110 billion in group sales and $ 3.6 billion in group profit before tax. Over 500.000 employees work in 7599 stores around the world, including franchises. Their largest market is the United Kingdom, with almost 3.500 stores and over 310.000 employees. Next to supermarkets, they also operate among others petrol stations, a bank and mobile phone, home phone and broadband businesses. Of course they also have a loyalty program and together these business sections create massive amounts of data. They started using that data already in the 1990s and have ever since expanded.

Introduction of the Tesco Clubcard

In 1995 they introduced their Clubcard, but instead of just using if for offering discounts, they understood that such a loyalty card could generate valuable insights into the shopping behaviour of their customers. Today, they receive detailed data on two-thirds of all shopping baskets thanks to the Clubcard. All this data enables Tesco to send very specific targeted emails to their customers. Already in 1999 they had 149.000 iterations of their newsletter and that has since then only expanded.  Currently they have over 38 million Clubcard members, of which 16 million are active users, and that offers Tesco valuable information and insights.

Data from the shopping carts for example, offers insights in which products could be best placed close to each other or which products should be closer to the checkouts or the entrance. Thanks to these detailed customer insights, as of course members are asked to provide an array of details when using the Clubcard, Tesco’s knowledge of their customers has become very intimate. This enables them to offer a wide variety of personal “lifestyle” magazines, based on information gathered from individual shopping cards, that matches content and coupons to various life stages and orientations. They have developed a dozen or so core lifestyle classifications, but that results in millions of variations in content and coupons. These personalized magazines of course drive conversion substantially.

All of their data analyses are done by dunnhumby, a wholly owned subsidiary that offers customer insights and applications for personalizing the customer experience. Dunnhumby not only works for Tesco, but for several retailers and brands around the world and in analyses data from over 350 million people in 28 different countries.

However, personalization is not the only Big Data applications of Tesco. They are applying predictive data analytics to forecast how many products will be sold where. By combining weather data and sales data they know what to expect and in the past years that has resulted in over $ 9 million less food wastage in the summer, $ 47 million less wastage due to optimized store operations and $ 78 million less stock in warehouses.

Reduction of Inventory and Food Waste

Tesco receives detailed local weather forecasts three times a day and links this data with 18 million product items as well as important local store information such as surroundings and type of customers. The insights of these analysis are directly stored with suppliers via the web-based system TescoConnect, to ensure that the right amounts of products are supplied.  Combined with simulations to understand and predict how stock moved and will move through the company, they can further optimize the amount of stock at hand.

Refrigerator Data to Reduce the Energy Bill

But that’s not all. Tesco also analyses refrigerator data to reduce their energy bill with almost $ 25 million a year. A joint research project between IBM’s research laboratories and Tesco revealed that Tesco could drastically decrease its energy bill by optimizing performance of their in-store refrigerators. Analyses of gigabytes of refrigerator data showed that the refrigerators where colder than necessary, spilling precious and expensive energy. All refrigerators in Ireland received sensors that measured the temperature every three seconds. Per store this added up to roughly 70 million data points over the course of a year. Thanks to the insights derived from this data, Tesco is now expanding the project across the UK to be able to save millions a year.

Downfall of Tesco’s Market Value

So Tesco is very innovative with their different Big Data initiatives, but is it all a happy story? Not really, because Big Data has not been able to prevent that Tesco’s market value has more than halved to an 11-year low. Due to the problems and because of an accounting scandal where Tesco overstated profit by hundreds of millions of dollars, Tesco chairman Sir Richard Broadbent had to resign.

According to an article by The Telegraph, the problem might by the actually be the Clubcard itself, which used to offer so much insights for Tesco: “…judging by correspondence from Telegraph readers and disillusioned shoppers, one of the reasons that consumers are turning to Aldi and Lidl is that they feel they are simple and free of gimmicks. Shoppers are questioning whether loyalty cards, such as Clubcard, are more helpful to the supermarket than they are to the shopper.”

For years, Tesco was considered a pioneer in the use of Big Data Analytics and an example for many retailers around the world. But things have changed. According to an article by the Harvard Business Review, the author, Michael Schrage, noted: “Tesco’s decline presents a clear and unambiguous warning that even rich and data-rich loyalty programs and analytics capabilities can’t stave off the competitive advantage of slightly lower prices and a simpler shopping experience.”

So, although Tesco uses Big Data to gain detailed customer insights, optimize their inventory and reduce their energy bill, it is not sufficient to prevent a sharp downfall as happened this autumn. Of course, data analytics will help in understanding what’s really going on, Tesco should not forget the human aspect of doing business. Tesco will therefore have to reinvent itself and once again offer a product that matches the needs and demands of its customers.

Three Ways How Santa Claus Uses Big Data this Holiday Season

Three Ways How Santa Claus Uses Big Data this Holiday Season

The holiday season operation is a big deal for Santa Claus. Getting the wide range of presents to kids all over the world is a complex operation that needs to be carried out within just a few days. Of course, nothing can go wrong, as that would significantly depress kids and their parents. Fortunately he can turn to Big Data to drive his different operations, which range from personalization of presents, inventory management and of course elf resources. As we don’t have access to North Pole Inc. lets have a look at how major retailers use Big Data this holiday season:

Image: North Pole Inc. via Gartner

Santa Claus will have a lot of shopping today this year. Forrester expects that online sales alone will reach $89 billion this holiday season. Retailers will have to do what it takes to be ready for this massive online demand and ensure that Santa Claus can find what he is looking for. Big Data analytics of course will help online retailers, as well as offline retailers, in recommending the right products for the right customers at the right moment.

Omnichannel Personalization

Omnichannel personalization is one of the most powerful tools online retailers can turn to. Creating a data-centric, single view of the shopper allows etailers to deliver the most relevant experiences across web, mobile and in store. Win.com, which is the #1 online wine retailers is a great example of how omichannel personalization drives revenue.

Offering thousands of wines to millions of customers across different channels requires deep knowledge about your customers. By combining data sources such as a shopper’s current browsing behaviour, geographical location and their online shopping behaviour they are able to show products to their customers they would not have considered otherwise, resulting in among others a 15% increase in order value.

Inventory Management

Big Data enables retailers to stock their warehouses intelligently. This will ensure that stores across the country are always stocked with the right products and the right amount of these products. Based on a variety of data sources such as buying patterns, weather conditions, market conditions or trends on social media, retailers can predict which products will be required where. Sensors, such as RFID tags, are also already widely used by retailers, online and offline.

Of course, not only inventory management can be improved, but also delivery management can be improved significantly. As can be learned from the case of UPS, weather data, traffic data, truck location data and other kinds of data sets can optimize delivery management in real time.

A Venezuelan supermarket has turned to Big Data to get insights in their inventory management worth $ 20 million. They are now analysing their 6 Terabyte of product and customer data to be able to review in real-time their inventory levels, store sales and costs of goods. This enables them to understand which products are selling better, which products are most profitable and which promotions are successful.

Walmart goes one step further. They are using their social genome to know who their customers are, what they say online and they adjust their offering based on these, as well as multiple other data sources such as the weather. A well-known case is the case of Strawberry Pop-Tarts. Walmart, recording every purchase by every customer for future analysis, noticed that when there was a hurricane warning the Walmart stores in the affected area would increase the sales of Kellog’s Pop-Tarts. Store managers were told to order extra and place the Pop-Tarts near the entrance during hurricane season.

Retailer Resource Management

Large retail chains employ many employees and Big Data can be used to improve their resource management. Retailers can use Big Data to monitor employees’ performance in real-time and take action if required. The data can also reveal top performers as well as workers who are unhappy in their job. Big Data can therefore improve employee engagement across a wide range of variables.

But that’s not all. Big Data can also be used to combat employee theft. Retailers can lose a lot of money due to employee theft and most of the time this is difficult to stop. However, Ontario-based Compass Group Canada, an owner and operator of more than 2,000 food service locations, like Tim Hortons, Subway and Starbucks is applying Big Data in a new way to combat theft. Of course they have camera’s in each of the service locations, but monitoring the millions of hours of video is undoable. Therefore they have developed a self-service analytics system. This system collects data from point of sale systems, enterprise resource planning system, employee time and attendance software and inventory management systems and combines them to find disparities. Once a discrepancy is found, the corresponding video system is analyzed to see what has happened. Thanks to this system, the company can better trace and therefore stop theft.

Retailers can benefit from Big Data in multiple ways and these three use cases are only the beginning. In the coming years we will see more examples of retailers using Big Data to improve their quality, improve contract management or improve their marketing and communications.

How Big Data Shapes Urban Waste Management Services in Manchester

How Big Data Shapes Urban Waste Management Services in Manchester

Big Data can be used for strategic policy making in almost any field and the Greater Manchester Waste Disposal Authority (GMWDA), England’s largest Waste Disposal Authority, has turned to Big Data to better plan their services. In order to do that, they are collaborating with the University of Manchester who uses the data generated by the GMWDA. Together they help create environmentally sustainable solutions for Manchester and the 1.1 million tonnes of waste that is produced each year.

Waste is an important aspect of societies and getting rid of it links societies with all the economic and consumption practices that we are so familiar with as well as how we can improve our environment. Big Data can greatly contribute to that and help governments better understand what’s going on and how they can incentivy citizens to improve their lives.

How the GMWDA applies Big Data, as can be seen in this video, is only one way to improve urban waste management. In many cities around the world, Big Data practices are used to reduce waste and improve waste management. In the city of Songdo for example, a true smart city in the making, citizens have to use a chip card to dispose their garbage. This enables the government to measure how much waste is disposed of when and where. In addition, sensors are placed inside the containers that measure all kinds of data. When combined with usage trends data or historical data, cities can forecast when the ideal moment is to empty the containers as well as optimize waste collection routes.

Researchers in Ethiopia are even combining geographic and socioeconomic data to better understand how household waste is spatially distributed to better manage waste practices for the whole city. Researchers from the University of Stockholm are using Big Data to identify how waste collection routes in the city can be optimized. Using a wide variety of data such as roughly half a million entries of waste fractions, locations and weights they were able to develop waste generation maps of Stockholm, revealing quite a few inefficiencies.

Big Data has only recently started to be used by local governments, but urban waste management is only one application of Big Data that we will see a lot more in the smart cities of the future. There are a wide range op applications ranging from public safety, traffic management or water management that can be optimized using Big Data. The smart city of the future will be a lot more effective and efficient thanks to Big Data analytics.

Five Big Data Trends for 2015

Five Big Data Trends for 2015

The future is approaching rapidly and we are about to enter the second half of the second decennium of the 21st century. Therefore it is time to look ahead and see which Big Data trends we can expect to take of in 2015. Where is the market heading and which areas should we keep a close look at. Earlier I reviewed the Big Data trends that I described for 2014 and after having done thorough research again I would like to share my thoughts for the Big Data trends for 2015:

A Connected Future: The Internet of Things Taking Off

I have mentioned it before, but 2015 will be the year of the Internet of Things. Next year we are going to hear a lot about new connected products that will be launched. Gartner recently named the Internet of Things at the peak of inflated expectations in their Hype Cycle of Emerging Technologies. The Internet of Things is therefore at the peak of the hype and in 2015 we will see this become concrete. We are reaching a tipping point, as in the past years sensors have become so cheap and small, there is a wide range of easy-to-connect to IoT platforms and the consumer is getting used to the idea of connected devices.

The next year we will therefore be flooded with new smart and connected devices. Either via crowd funding campaigns via for example Kickstarter or products from large corporations that hit the market via the normal way. All of these items will be linked to some sort of platform, such as for example Apple’s Homekit, or have easy to use dashboards that can be accessed via your smartphone, tablet or smartwatch.

All these connected devices offer unlimited possibilities for consumers and organizations and will infiltrate every aspect of our lives. Already today we see smart thermostats that know when you are in the house. We have smart toothbrushes that tell you if you are brushing your teeth correctly. There are even smart diapers that inform you about the health of your child. The possibilities for the Internet of Things are basically endless and expectations for the size of the market range from $3 to $ 14 trillion. All these connected devices will create massive amounts of data that can be used to improve our lives and make it more comfortable, efficient and effective. 2015 will be the first year that the Internet of Things hits the masses.

A Shift Towards Data-Driven Cultures

Big Data requires a culture shift if you truly want to reap all the benefits of a Big Data strategy. Shockingly, according to IBM, 1 in 3 business leaders do not trust the information they use in the decision-making and thus are reluctant to make data part of their company culture. However, in the past years, Big Data has gained so much traction and there are ample use cases of companies successfully implementing a data-driven culture that this is about to change.

In 2015 we will see a shift towards data-driven cultures. According to David Cearley, Gartner VP and Fellow, “analytics will become deeply, but invisibly embedded everywhere.” Big Data in the end remains a people’s questions and more and more organizations will understand the benefits of a Big Data strategy and thus will have to change their culture. Companies will see the benefits reaped by and the competitive advantage of large data-driven organizations and want to join the pack. When the IT infrastructure is ready for centralized data storage and processes are in place that ensure layered access to the data (based on user roles) and the right policies are in place including smart real-time dashboards, a data-driven culture is a logical consequence.

Owning Up to Your Own Identity – Claiming Your Personal Data

The debate on data ownership is not yet over. In fact, we are just getting started. Consumers around the world are waking up from the “free services/products” dream and understand that ‘free’ services like Gmail, Facebook or Twitter are actually paid for with data. Consumers are becoming more aware of the value of their data and they are less willing to give companies their sensitive data for free. According to a survey from TRUSTe, a privacy management firm, the percentage of US adults opting out of online behavioural advertising increased from 27% to 50%.

In 2015 we can expect a rise in startups that approach personal data from a new perspective. Already we see various personal data marketplaces that allow consumers to sell their data to companies. Examples include Handshake, who want to cut out data brokers such as Acxiom, Ctrlio, who are developing a platform for individuals to become more in control of their own data or DIME, a Dutch startup that wants to enable consumers to make money with their data. 2014 also saw the launch of Ello, an ad-free social network alternative to existing social networks like Facebook or Twitter. These initiatives are just the beginning and 2015 will see a rapid growth in such personal data startups.

Big Data Security Analytics Gaining Traction

When dealing with Big Data, security is key. In the Big Data trends overview for 2014, this was also an important aspect and in 2015 this will remain the case. High volumes of valuable data attract criminals and as long as quantum teleportation is not yet possible, data can and will be hacked. That offers big opportunities for Big Data Security Analytics. According to Alvaro Cardenas, lead author of the report in the CSA press release, "the goal of Big Data analytics for security is to obtain actionable intelligence in real time".

The coming year we will see a lot of different new tools that will leverage large quantities of structured and unstructured data to analyse and detect in real-time any possible security threat. Already there are quite a few Big Data startups that focus on security analytics such as 405labs, focusing on customized advanced security analytics, Sentinel Labs, which focuses on real-time forensics and 360-degrees attack visibility, or fiD3 that focuses on dark data discovery to protect organizations.

Although there are still quite some challenges that must be overcome to truly benefit from Big Data Security Analytics, for example how to analyse, detect and stop a security threat in real-time using massive amounts of different structured and unstructured data sources, 2015 will see an increase in traction for security analytics.

Time to Experiment with Data Lakes

Data storage has become so cheap these days, that organizations can almost store any data they want, whether they will immediately use it or not. Having a data lake, gives users instant and easy access to all that data and you don’t need to design a data model beforehand.

Pentaho CTO James Dixon is credited with coining the term "data lake".  As he described it in a blog entry, "If you think of a datamart as a store of bottled water – cleansed and packaged and structured for easy consumption. Translate this into the data version of the term and the contents of the data lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples.”

Apart from instant access to all data, another advantage of a data lake, is that it can contain any type of data from structured log data to unstructured audio, video or text data. Although there are many advantages, experts say that they organizations should use it in conjunction with traditional data warehouses. Data lakes are relatively new, but in 2015 we will see more organizations experiment with data lakes.

Well, that’s it for 2015. The overall Big Data trend and market is constantly evolving and organizations are getting better in benefiting from their data. Once again, an interesting year awaits us with fascinating new developments ahead of us. 

The Year in Review: How Did My 2014 Big Data Predictions Turn Out?

The Year in Review: How Did My 2014 Big Data Predictions Turn Out?

As is tradition, every year at the end of the year I give my predictions for the Big Data trends that we can expect to gain traction in the next year. With 2015 at the doorstep, it is the right moment to reflect at my Big Data predictions of 2014. Did they happen as I foresaw one year ago and what will be the Big Data trends we can expect in 2015? Last year I predicted the following Big Data trends:

  • The rise of the Industrial Internet;
  • It’s going to be cloudy: Big-Data-as-a-Service solutions;
  • Security to protect the privacy;
  • Personalization will become personal;
  • Education will be essential for success;
  • Big Data moves into Mixed Data;
  • It’s time for a Proof of Concept.

If we look back at 2014, we see that a lot has happened in the field of Big Data. According to Gartner’s 2014 Hype Cycle of Emerging Technologies, we are even past the hype. Big Data is on it’s way to the trough of disillusionment and it will take another 5-10 years before it will reach the plateau of productivity. In its Hype Cycle Special Report Gartner explains that “While interest in Big Data remains undiminished, it has moved beyond the peak because the market has settled into a reasonable set of approaches, and the new technologies and practices are additive to existing solutions.” What does that mean for my predictions?

The Industrial Internet

Looking back at my predictions of 2014 we see that the Industrial Internet has been taking off, but requires more time than anticipated. General Electric, who dubbed the term the Industrial Internet, is beginning to see strong returns on its Industrial Internet investments. The value created by the Industrial Internet is expected to be $ 1.3 trillion in 2020, while the total technology spend is expected to be $ 514 billion by 2020. This is a Return on Investment of 150%, creating a strong case for manufactures to also invest in Industrial Internet applications. However, the investments required take quite some to be carried out.

In 2014 we also saw the launch of the Industrial Internet Consortium, which aims to bring together the organizations and technologies necessary to accelerate growth of the Industrial Internet by identifying, assembling and promoting best practices. Currently the consortium is 100 members strong, including some of the largest organizations. This will definitely have a major impact on the growth of the Industrial Internet in the coming years.

Big-Data-as-a-Service Solutions

The amount of Big Data solutions available in the cloud has grown dramatically in 2014. Most of the major Big Data vendors nowadays offer a solution in the cloud that enables organizations to easily connect their data and gain insights from it. Data-Analytics-as-a-Service is expected to have a CAGR (Compound Annual Growth Rate) of 150% in the coming years, but we will also see other options hitting the market such as Logs-as-a-Service, BI-as-a-Service or Infrastructure-as-a-Service.

In the past year, cloud technologies have matured and companies such as Amazon have made it relatively easy to bring all your data into the cloud and connect with it via other platforms. This of course has contributed to the rise of Big-Data-as-a-Service solution providers. For the coming year, this trend will continue to expand.

Security

More than ever is security important for all kinds of organizations. With the amount of data that is created and has to be stored growing, organizations need to pay serious attention to protect that data and therefore the privacy of their customers. The first CEO who had to step down because of a data breach was a fact in 2014. Target’s CEO, President and Chairman Gregg Steinhafel resigned from all his positions after the company was hacked and personal information including credit/debit card details of close to 110 million individuals were stolen.

Organizations can secure themselves against such data breaches using Big Data Security Analytics. This trend, which combines a wide variety of data sources to discover security threats, promises great insights for organizations to battle cyber threats. Unfortunately, it is still immature and not yet widely adopted. With more data breaches to expect in the coming years, this will undoubtedly change.

Personalization becomes Personal

Personalization really took off in 2014. Many organizations in a wide range of industries have adopted Big Data to really get to know their customers. They use these insights to offer the right products/services to the right customers at the right moment for the right price via the right channel. Banks for example, started to use the vast amounts of data they have about their customers to create a 360-degree view of each customer based on how each and every one customer uses online and mobile banking, ATMs, branch banking or other channels. Retailers such as Walmart combine different data sources to know exactly what a customer wants. In fact it is the objective of Walmart to know what every product in the world is, they want to know who every person in the world is and they want to have the ability to connect them together in transaction, thereby bringing personalization to a new level.

Education is Essential

Although it will not solve the expected shortages in skilled Big Data talent, 2014 saw the launch of a large amount of Big Data programs at universities and colleges around the globe. In the USA alone already almost 60 programs have launched and also in Europe we saw a large amount of new Big Data programs. In addition, IBM is partnering with 1.000 universities in order to prepare students for Big Data careers. The amount of Big Data programs took off in 2014 and we can expect the first results of this growth in the coming years.

Big Data becomes Mixed Data

Although many Fortune 500 companies are faced with an ever-increasing amount of data flowing into their organization, a lot of Small en Medium Sized Enterprises are not yet faced with Petabytes of data. These organizations need to work with smaller amounts of data and in order to get insights from less data, they need more data sources. In 2014 we noticed that more and more SME’s are also developing a Big Data Strategy and they are combining a variety of different, smaller, data sources to gain insights. So in 2014, Big Data indeed became Mixed Data.

Proof of Concepts

Developing a Big Data Proof of Concept typically takes approximately 18 months and implementing a Big Data strategy is not easy. Therefore a lot of organizations have started with Proof of Concepts to better understand what Big Data can do for them and to learn from it. With the organizations that we advice, we noticed a steep increase in the amount that started with a Proof of Concept. For these companies, 2015 will be the year to move on and make Big Data part of their organization.

The year 2014 is almost finished and we saw a lot of interesting Big Data developments. We are still just at the beginning of an information revolution and therefore 2015 is going to be fascinating as well. In the next post I will share with you my ideas for the Big Data trends of 2015. 

8 Easy Steps to Become a Data Scientist

8 Easy Steps to Become a Data Scientist

Becoming a Big Data Scientist is not easy. It is hard work and it requires a vast array of skills. Although the Big Data Scientist is said to be the sexiest job of the 21st century, it requires a combination of skills that previously did not even exist in a complete department. A Big Data Scientist needs to be able to understand the technical aspects of data and have sufficient knowledge of mathematics and modelling. He or she needs to have knowledge of a wide range of tools and techniques ranging from Hadoop to Python and anything in between and at the same time needs to be able to understand the business in order to ask the right questions. Some time ago I wrote the ultimate job description of the Big Data scientist and that shows that these talents are scarce and difficult to achieve.

Last month, Datacamp confirmed this story. They created an infographic containing a guide with eight easy steps how to become a Big Data Scientist, including some additional resources for further perfection. Unfortunately, for Data Scientists to be who expect to become one at short notice, there is only utter disappointment. It still is a long road and requires a large amount of expertise that has to be learned.

Above anything, a Data Scientist needs to be able to learn and write code, preferably in many different languages. But sufficient knowledge of mathematics, statistics and machine learning to start with is also required.  Once these skills are mastered, it is time to learn various types of databases. Different types of data and different use cases require different databases. So it is important to know and be able to work with a wide range of databases. This knowledge comes in handy when you need to start mining the data within the various databases and when you want to visualize them in order to better understand the insights you can now derive from it. But when you need to handle such massive amounts of data that is just not enough. You should also learn typical Big Data tools and techniques such as Hadoop, MapReduce or Spark.

Once you have mastered all that, and which probably will take you some time, it is time to start working, get experience and meet fellow Data Scientists in order to learn from each other. Internships are a great way to work at large companies and learn from the best. Share your knowledge with the community, so that the entire community benefits from your newly gained knowledge.

Becoming a Big Data Scientist is difficult and it sure is a long way. But it is also very rewarding. Data Scientists will earn high salaries and work on challenging projects. So it definitely is worth the hard work and it will earn you the job title of the sexiest job in the 21st century. And let’s be honest, who does not want to have that job?

How to become a Data Scientist

 

SumoLogic

SumoLogic

CompanySumoLogic
Address605 Castro Street, Mountain View - USA
FoundersChristian Beedgen and Kumar Saurabh
FoundedMarch 2010
Funding$ 50.5 million
Employees60
Websitewww.sumologic.com
Rating7 bits

 

SumoLogic is a cloud-based service that collects, manages and analyses log data in real-time. It allows companies to analyse all their log data from different apps, networks or systems, regardless of type, volume or location.

Continue reading SumoLogic
Continuuity

Continuuity

CompanyContinuuity
Address150 Grant Avenue, Suite C, Palo Alto - USA
FoundersTodd Papaioannou, Jonathan Gray, Nitin Motgi
FoundedOktober 2011
Funding$ 12.5 million
Employees8
Websitewww.continuuity.com
Rating6 bits

 

Continuuity claims to democratize big data application development. They have built AppFabric, a Platform as a Service tool that hosts in the cloud and that allows developers to build their own big data tools. It is a development environment on top of a company’s Hadoop infrastructure. Developers do not have to worry about the back-end or the integration, as AppFabric takes care of this.

There is an SDK available that allows users to build apps with a drag-and-drop user interface. Before deploying them, they can run, test and debug the apps locally. When an app is ready it can be deployed easily and their AppFabric UI provides real-time information about the app, which can be useful if a developer needs to scale quickly.

Continuuity is founded by big data experts from Yahoo and Facebook, so they know their way around big data. Co-founder and CEO Todd Papaioannou says Continuuity wants to make it easy for developers at fast-follower companies to build big data applications without having to learn Hadoop’s low-level APIs and the general complexity of the distributed framework. Very few people, he said, really want to be part of the ”home-brew computing club experience.”

At the moment, Continuuity is only available in a private beta and developers need to sign-up if they are interested. It is expected that a private cloud version will be available in 2013, which will charge on a per-usage-basis.

Continuuity makes it a lot easier for developers to develop new applications for big data as engineers do not have to worry about nodes, clusters or data blocks anymore. Their PaaS service has received rather substantial funding from well-known venture capitalists in a short timeframe. Also they just opened their Beta to the public. However, they have not yet won any awards and the amount of customers is still small. Therefore, Continuuity receives a rating of 6 bits.

Platfora

Platfora

CompanyPlatfora
Address100 S Ellsworth St Suite 400 San Mateo - USA
FoundersBen Werther & John Eshleman
FoundedAugustus 2011
Funding$ 27.2 million
Employees16
Websitewww.platfora.com
Rating5 bits

 

More information is created every day and the need to understand this data is growing with a lot of companies. Big data provides valuable opportunities to better understand your company. Until recently, business users would have to wait ages to get answers from their massive amount of data. Not anymore as there is Platfora. Platfora creates clarity from big data and they have just received a $20 million funding led by Battery Ventures.

Continue reading Platfora
Metamarkets

Metamarkets

CompanyMetamarkets
Address625 2nd Street Suite 230 San Francisco - USA
FoundersMichael Driscoll & David Soloff
FoundedMay 2010
Funding$ 23.5 million
Employees25
Websitewww.metamarkets.com
Rating7 bits

 

Metamarkets helps buyers and sellers of online advertising to spot trends, drill down on data, visualize insights and make better decisions. They do this by ingesting and analyzing vast amounts of transactional data in real-time, as events are unfolding, and they present this information through an intuitive, interactive visual dashboard. Knowing why users cancel subscriptions or how they move through a platform can make a difference in being successful or not. Metamarkets offers a packaged Software-as-a-Service application that allows users to create an account, upload data and start analysing. It is a scalable service as it grows with the data volume growing.

The objective of Metamarkets is to help businesses understand the vast amounts of data and their visual analytics tool can be used without extensive statistics knowledge. The platform processes, queries and visualizes high volume and high frequency event streams such as payments and users can discover suitable insights in this data.

Part of the solution of Metamarkets is the open-source real-time streaming data store component called Druid. This is a distributed in-memory columnar database and it delivers dynamic and interactive user experiences across billions of records in no time. The fast response rate is possible because Metamarkets allows event data to be streamed and uploaded and at the same time performing aggregations and calculations on the fly.

A lot of companies working with big data use Hadoop. Unfortunately, Hadoop does not support real-time data queries. Therefore Metamarkets uses a specialized version of Hadoop for parallel processing. It also added a custom build in-memory database with predictive modelling capabilities. This means it can help companies determine the top content from e.g. the past few hours for a certain demographic across a large data set.

Metamarkets’ mission is to democratize data science by delivering powerful analytics that are easy and intuitive for everyone. It enables an increase in revenue, improved user engagement, and the ability to avoid errors. These aspects can make a difference within a company and its ability to deliver to the needs of the customer.

The big data phenomenon allows business to better understand their company. With Metamarkets it will be possible to understand real-time what customers want while deriving information from massive events data. Information that has always been available but only now can be analysed in great detail and in real-time. Furthermore, they have won the best Cloud Application Award at Cloud Computing World Series Awards. Therefore and they won the Top Innovator Award for Data Visualization Technology From DataWeek. Combined with their patent and the investors who have funded Metamarkets, they receive a 7 bits rating.

Tableau Software

Tableau Software

CompanyTableau Software
Address837 North 34th Street Suite 400 Seattle - USA
FoundersChristian Chabot, Dr. Chris Stolte, Dr. Pat Hanrahan
FoundedJanuary 2003
FundingIPO
Employees700
Websitewww.tableausoftware.com
Rating8 bits

 

Tableau Software provides easy-to-use software applications for fast analytics and visualization. Their goal is to help people see and understand data. In 2011 they were ranked by Gartner as the world's fastest growing business intelligence company.

Continue reading Tableau Software
Infinite Analytics

Infinite Analytics

CompanyInfinite Analytics
Address50 Memorial Drive, Cambridge, MA, 02139, United States
FoundersAkash Bhatia & Purushotham Botla
FoundedJune 2012
FundingUndisclosed
Employees8
Websitewww.InfiniteAnalytics.com/
Rating6 bits

 

Infinite Analytics is a cloud-based big data startup that uses the social graph of consumers to provide personalized recommendation. They have developed SocialGenomix, a tool that merges data from major social networks to create, as they like to call it, a “segment of 1”. The objective is to provide a complete and detailed overview of the customer based on his or her complete social profile. This allows companies in e-commerce in the retail or travel industry to deliver relevant and personalized recommendations.

The SocialGenomix takes into account likes, interests, activities, brand affinity, intent and spending potential and analyses all the data to create a personalized online experience. The platform has been developed at MIT and came to live during a class taught by Sir Tim Berners-Lee, who is also an advisor of the company. In order to predict user’s behaviour they use different big data techniques, among others Natural Language Processing, Machine Learning and predictive analytics. The algorithm follows the customers changing tastes, so that the information is always up-to-date.

According to Infinite Analytics, they have processed 90 million users, 5.3 billion data attributes and caused a 25% increase in site engagement for those companies using their tool.

It was founded by Akash Bhatia and Purushotham Botla, both MIT graduates. Currently they have 6 paying clients and they completed several accelerators and startup challenges in which they won $ 100k in the Startup Chile competition. Currently they are backed by various Angel Investors. They have patented their SocialGenomix.

Personalization of content and recommendation for online shoppers is extremely important for e-commerce companies, but still difficult to achieve, as there are so many variables. How Infinite Analytics will perform over time is still unclear as they are still a young big data startup with only a few customers. They have, however, an impressive board of advisors and therefore we are confident that they will live up to their promise. We therefore give them a 6 bits rating.

Aureus Analytics

Aureus Analytics

CompanyAureus Analytics
Address17 Phillip Street, #05-01 Grand Building, Singapore 048695
FoundersAnurag Shah, Ashish Tanna & Nitin Purohit
FoundedMay 2013
FundingSelf-funded
Employees12
Websitewww.AureusAnalytics.com/
Rating3 bits

 

Aureus Analytics is a Singapore based analytics service company that provides a platform that has been built using several big data (open source) frameworks such as Hadoop, Cloudera, Revolution, Datasift and Tableau Software to deliver a complete big data solution to organisations. This Aureus Statistical Analytics Platform (ASAP) combines industry specific best practices with relevant data, statistical & analytical models.

It enables organisations to develop a big data analytics solution across the organisation in a short time-frame, while still benefitting from the different tools in the market. The platform focuses on Customer, Risk and Operational Analytics in the Insurance Banking, Retail and Investment Healthcare industry. ASAP helps processing large volumes of data and analysing them to predict future trends and opportunities, build customer loyalty, prevent fraud, improve financial performance, and optimize operations.

They are a very young company, as they launched the big data startup in May 2013. Currently they have two ongoing pilots and a few more in the pipeline, before they open the platform for other organisations. It was founded by Anurag Shah, Ashish Tanna and Nitin Purohit, who self-funded the startup-phase they are currently in.

The platform that they have delivered looks promising, but there is yet still a lot unknown. Being launched so recently, they have not yet won any awards, nor do they currently have any patents for their platform. However, big data analytics is also gaining a lot of traction in Asia and it is good to see big data startups appearing in Asia. We therefore give them a 3 bits rating.



 
Quid

Quid

CompanyQuid
Address733 Front Street C1A San Francisco, CA, 94111 USA
FoundersSean Gourley & Bob Goodson
Founded2010
Funding$ 10 million
Employees40
Websitewww.Quid.com/
Rating8 bits

 

Looking to understand what’s happening in the world and where trends are heading? Than Quid is the big data startup to look for. Using trillions of different data points they combine information from among others research papers, patent applications, Twitter posts, press releases, funding information or news articles to deliver visualizations that show what’s happening in different technology sectors. These vizualizations can be used as multidimensional industry maps to find the organisations that can help you with your business.

Quid uses techniques like Natural Language Processing and semantic-clustering analysis to reveal which sectors contain a lot of competitors on the same topic as well as show potential business opportunities. The maps also show references and links between different companies across different industries. This can help organisations find new partners or markets as well as understand what the competition is doing. This is not only interesting for existing companies, but also for investors who want to find that early-stage technology that will be the next Google.

Companies that want to move into a new direction can use the tool to understand who is already present in the new area. Or Quid can help organisations find new technologies to better serve their customers. They have already worked for a variety of global organisations, including Samsun, BBVA or Intel and even the United States army that use Quid to take strategic decisions during war.

The objective of Quid is to make the complex world understandable with insights from comprehensive global real-time data. It was founded in 2010 by Bob Goodson and Sean Gourley and they call Quid “augmented intelligence”. Since then they have raised $ 10 million in funding from across the globe.

Quid is a very powerful visualization tool that gives organisations the ability to make sense of the vast amounts of public data today available. It was named one of the world’s most innovative companies in 2013. Within a few years they aim to create a user-friendly, affordable version that can also be used by smaller organisations or even individuals. We therefore give them an 8 bits rating.

Bizosys Technologies

Bizosys Technologies

CompanyBizosys Technologies
Address2630, 27th Main, Sector 1, HSR Layout, Bangalore - 560 102, India
FoundersSunil Guttula, Abinasha Karana, Sridhar Dhulipala
FoundedApril 10, 2010
FundingSelf-funded
Employees12
Websitewww.bizosys.com/
Rating6 bits

 

Bizosys Technologies is an Indian big data startup that delivers big data solutions for companies in the Pharma, Telecom or eCommerce industries. They have developed HSearch, which is a real-time search and analytics engine on Hadoop. HSearch works with text content as well as database records and it uses an indexing algorithm to process the data parallel across machines. It is available as an open source tool, but they offer a paid version that includes support and management features.

Next to their HSearch solution, Bizosys offers a consulting service to identify bottlenecks within an organisation and will suggest recommendations where necessary. In addition, they have developed a configurable framework that is capable of aggregating unstructured and semi-structured data from existing systems and make it searchable from within this application.

The three co-founders of Bizosys, Sunil Guttula, Abinasha Karana & Sridhar Dhulipala, all worked at Infosys in Bangalore. Founded in April 2010, it has been operational since without any outside funding. They primarily focus on Small and Medium sized enterprises. When they started, their objective was: “to develop a business operating system that is easy to build, robust, scalable and especially intended for frequently changing, rapid deploy, long tail of applications.”

As an open source tool, HSearch has currently been downloaded over 4.000 times in more than 80 different countries. They have developed the architecture in such a way that it is highly scalable and provides elasticity in sync with application usage growth. Currently they have filled for patent in the USA, which is still pending.

Byzosys Technologies has won several awards in India; among others they made the top 10 in India software product companies from Nasscom. Real-time analytics based on Hadoop is a powerful tool for organisations and with their open source tool being regularly downloaded in many countries, they have the possibility to leverage that and grow further by selling the enterprise editions to companies. They already have some customers in the USA, which is likely to grow. We therefore give them a 6 bits rating.

Guavus

Guavus

CompanyGuavus
Address1820 Gateway Blvd Suite 250 San Mateo CA 94404, USA
FoundersAnukool Lakhina
FoundedFebruary 2006
Funding$ 87 million
Employees450
Websitewww.Guavus.com/
Rating8 bits

 

Guavus is an integrated end-to-end big data solution with decision-making applications for network engineering, marketing and customer care for Telecom organisations. They fuse data together from multiple sources and analyse it before it is stored, which helps organisations make better decisions based on their different applications as well as eliminating the need to spend a lot of money on storage. Analysing the data provides business metrics that helps to optimize network capacity, improve revenue, and improve customer satisfaction as well as decision-making.

The platform is called the Guavus Reflex™ Platform and it is capable of creating actionable information from widely distributed, high volume data streams in near real-time. It uses algorithms and machine learning to create insights from extremely large datasets – hundreds of billions of events or petabytes of data per day. They have linked with Teradata to optimize their product.

Guavus focuses on Communications Service Providers (CSPs) to help them optimize network capacity and deliver a better customer experience and thus driving revenue. It provides insights across their network, (customer) devices, content and their customers. These insights a derived from combining vast amounts network data, deep packet inspection data or unstructured machine data with demographic and billing data. With CSPs dealing with large networks they deliver solutions to $ 100 million problems.

In order to provide the insights, they use a vast array of techniques, ranging from trend analysis, outlier and anomaly detection, pattern and clustering analysis to forecasting techniques. Guavus was founded in 2006 by Anukool Lakhina, who has a PhD in data science and previously worked at Sprint Labs and Intel Labs. Up till now they have raised & 87 million in venture capital from well-known investors like Goldman Sachs, Intel Capital, QuestMark Partners and Sofinnova Venture. Their latest round, a $30 million D round, was closed in January 2013. They have been granted 5 patents and 17 are pending. They made 2 acquisitions including Neuralitc, a leading provider of mobile marketing analytics, and Pipleine technology from Applied Broadband, a leading technology for IPDR.

The global telecom industry experiences a massive growth in data, thanks to the rise of the smartphones and tablets, the next generation mobile networks (4G) and the developed world that becomes connected to the mobile internet. Those telecom companies that are able to use these vast amounts of data efficiently will outperform their peers, grow their market share and improve their bottom line results. Guavus can help the Telecom companies with an innovative solution and therefore we give them an 8 bits rating.

Three Use Cases How Big Data Helps Save The Earth

Three Use Cases How Big Data Helps Save The Earth

The earth is having a difficult time, for quite some time already. Deforestation is still happening at a large scale across the globe. In Brazil alone 40,200 hectares were deforested in the past year. The great pacific garbage patch is still growing and smog in Beijing is more common than a normal bright day. This is nothing new unfortunately. A possible solution is however. Since a few years, scientists, companies and governments are turning to Big Data to solve such problems or even prevent them from happening. It turns out that Big Data can help save the earth and if done correctly, this could lead to significant results in the coming years. Let’s have a look at some fascinating use cases of how Big Data can contribute:

Monitoring Biodiversity Across the Globe

Conservation International, a non-profit environmental organization with a mission to protect nature and its biodiversity, crunches vast amounts of data from images to monitor biodiversity around the world. At 16 important sites across the continents, they have installed over a 1000 smart cameras. Thanks to the motion sensor, these cameras will captures images as soon as the sensor is triggered by animals passing by. Per site these cameras cover approximately 2.000 square kilometres.

In the past years Conversation International has captured massive amounts of data, even if that data had to be downloaded manually. The cameras are installed at remote locations and scientists need to download the images taken manually, at location, and subsequently upload it to the cloud. They automatically determine which species have appeared in the images and enrich the data with other information such as climate data, flora and fauna data and land use data to better understand how animal populations change over time.

Today, all that data is analysed via HP Vertica Analytics, which allows the researchers to get results in a short timespan once the data is uploaded. Together with HP engineers they have developed the Wildlife Picture Index (WPI) Analytics System, a project dashboard and analytics tool for visualizing user-friendly, near real-time data-driven insights on the biodiversity. The WPI monitors ground-dwelling tropical medium and large mammals and birds, species that are important economically, aesthetically and ecologically.

Using Satellite Imagery to Combat Deforestation

Mapping deforestation is becoming a lot easier today thanks to Big Data. Imagery analytics allows environmentalists and policy makers to monitor, almost in real-time, the status of forests around the globe with the help of satellite imagery. New tools like the Global Forest Watch uses a massive amount of high-resolution NASA satellite imagery to assist conservation organizations, governments and concerned citizens monitor deforestation in “near-real time.”

The tool was launched in the beginning of 2014 and thanks to massive computing power of Google, they were able to analyse over 700.000 satellite images. According to Rebecca Moore, Engineering Manager at Google Earth Outreach and Earth Engine, “It was a total of 20 terra-pixels of Landsat data and to do that we applied one million CPU hours on 10,000 computers in parallel”. That’s the equivalent of one normal computer making calculations from 15 years straight. What resulted was a high-resolution map that showed the annual change in forest cover since 2000. When combined with all kinds of other data sources, this can be used to combat deforestation.

But that’s not all. Planet Labs has developed a tiny satellite that they are currently sending into space, dozens at a time. The satellite measures only 10 by 10 by 30 centimeters but is outfitted with the latest technology. They aim to create a high-resolution image of every spot on the earth, updated daily. Once available, this will generate massive amounts of data that they will open source for others to develop applications that will improve earth. Check out his TED speech on this great initiative:

Monitoring and Predicting with Smart Oceans

Over 2/3 of the world consists of oceans and also these oceans can be monitored. Earlier this year, IBM Canada and Ocean Networks Canada announced a three-year program to better understand British Colombia’s Oceans. Using the latest technology and sensors, they want to predict offshore accidents, natural disasters and tsunamis and forecast the impact of these incidents. USings hundreds of cabled marine sensors they are capable of monitoring waves, currents, water quality and vessel traffic in some of the major shipping channels.

All data collected will be used to run simulations on tsunamis and earth quakes in order to determine the impact on coastal areas. Information like this can be very valuable to all kinds of public agencies, tourism and other industries operating in the area. The researchers will also use visual analytics and techniques like machine learning to develop, test and run systems that could monitor pollution, spill response and other important aspects of ocean management. The smart ocean has arrived.

These are just three examples of how Big Data can help save the planet. There are of course a lot more fascinating examples and here is list of 10 of such use cases. Big Data offers great possibilities for businesses and governments around the world, but Big Data provides also a solution to protect our planet. Therefore, let’s use it to improve our earth.

Where Does The Internet of Things Come From?

Where Does The Internet of Things Come From?

The term Internet of Things was first coined by Kevin Ashton in 1999, but actually, the Internet of Things goes back a lot further than merely 15 years. In fact, already in 1926 we saw the first predictions of an Internet of Things. Back then, Nikola Tesla told Colliers Magazine the following in an interview: "When wireless is perfectly applied the whole earth will be converted into a huge brain, which in fact it is, all things being particles of a real and rhythmic whole.........and the instruments through which we shall be able to do this will be amazingly simple compared with our present telephone. A man will be able to carry one in his vest pocket." In the coming years we will see an explosion of devices connected to the Internet and together we are creating a smart planet. But first, let’s have a look into the origin of the Internet of Things:

Before the Internet was developed in 1969, Alan Turing already proposed the question whether machines can think, in his 1950 article Computing Machinery and Intelligence. He stated that "...It can also be maintained that it is best to provide the machine with the best sense organs that money can buy, and then teach it to understand and speak English. This process could follow the normal teaching of a child." So, years before the first message was send across Internet, Alan Turing was already thinking about about smart machines communicating with each other.

In 1969, the Internet is born when UCLA and Stanford University establish the first nodes, known today as ARPANET, to send the first messages across.  Five years later, we see the beginnings of TCP/IP, which becomes the standard of communications across the Internet in 1982. In 1973, Mario Cardullo patented the first passive RFID tag, which he had come up with the a few years earlier. Still today, the RFID is widely used in many different industries, for example, to automatically control inventory in the retail industry.

In 1982 we saw the first connected Coke vending machine, when members of the Carnegie-Mellon Computer Science department installed micro-switches to see on their computer terminals how many bottles of Coke were left in the vending machine and whether they were cold or not.

The First Connected Devices

In 1989 we saw the first ‘House of the Future’ built in The Netherlands. This house, which was a project of Chriet Titulaer, was meant to give the consumer a view how the future might look like. The house had all kinds of smart domotica and focused on the interaction between man and machine. Voice recognition was an important aspect of the house.

In 1990 the first toaster was connected to the Internet. This toaster was developed by John Romkey and Simon Hacket and they revealed their Sunbeam Deluxe Automatic Radiant Control Toaster at a show in 1990. The only thing that could be done was turning it on and off, but in 1991 they added an automatic crane to also insert a slice of bread automatically.

From their on, things started to speed up and in 1999, after Kevin Ashton had coined the term Internet of Things, we saw the first Machine-to-Machine protocol developed by IBM, called MQ Telemetry Transport (MQTT). This protocol facilitates "connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium” as described by the website MQTT.org. This universally accepted and adopted Machine-to-Machine protocol has stimulated the development of new and more connected devices.

The Smart Refrigerator

In 2000, the smart refrigerator made its entry. For many years, the smart refrigerator has been the example of the Internet of Things and it was developed by LG. It had an LCD screen that was capable of showing information such as inside temperature, the freshness of stored foods, nutrition information and recipes. The refrigerator was called the Internet Digital DIOS and cost $ 20.000,-. Perhaps not surprisingly, it did not sell as well as LG had hoped.

In the years after, the Internet of Things is mentioned in several mainstream publications such as The Guardian, Boston Globe and Scientific American. In 2004, Walmart started to deploy RFID tags at large scale to improve their inventory.

In 2005, the United Nations first took notice of the Internet of Things went it was mentioned in a Telecom Union report. The first conference on the Internet of Things took place in Zurich in 2008. In 2010 Google launched their self-driving car concept, taking a huge leap forward in the development of connected and autonomous cars. Also in 2010, two former Apple engineers started Nest Labs, the company that produces smart thermostats and smoke detectors. In 2014, Google acquired Nest Labs for $ 3,2 billion to speed up their Internet of Things division. In 2014, Apple launched HomeKit, which is a framework in iOS 8 for communicating with and controlling connected accessories in a user's home, resulting probably in a lot more connected devices in the coming year.

Launch of IPV6

But the biggest enabler of the Internet of Things is the launch of IPV6 in 2011. Where IPV4 had only approximately 4 billion addresses, IPV6 has a total of 340 undecillion IP addresses, which is the equivalent of 3,4 with 38 zeros. This is more than enough to cope with the expanding Internet of Things in the coming years, even if projection are correct that predict that by 2030 we will have 100 trillion connected devices in the world.

We are at the forefront of a connected world and 2015 will probably be the year of the Internet of Things, with a huge amount of connected devices being developed and announced. What sort of connected device are you looking forward to?

Image source: Cisco
How Big Will The Internet of Things Be?

How Big Will The Internet of Things Be?

2014 was the year of Big Data, 2015 will be the year of the Internet of Things. More and more every day items are being connected to the Internet ranging from smart thermostats to smart toothbrushes. In the coming years, the amount of smart devices in our household could grow drastically as Gartner predicts that a typical home could contain more than 500 smart devices by 2022. The falling costs of sensors and the upcoming domotica platforms such as Apple’s Homekit will contribute to this growth. However, this is just the beginning. IDC expects the market for the Internet of Things to grow to $ 3 trillion by 2020, Garnter even predicts a $ 14 trillion market in 2022. So how big will the Internet of Things actually be?

Well, the Internet of Things is buzzing currently and the expectations are very high. The required infrastructure, cloud computing, is easily available, scalable and relatively cheap. The required sensors are becoming smaller, better and cheaper every year and all Internet of Things manufactures, of course, claim that it will make everything in our lives ‘smart’ and our lives easier.

The possible applications of the Internet of Things are more or less endless. Basically any device or product can be made ‘smart’ when several sensors are added and the device can connect to the Internet. Throw in some smart algorithms and the data can be analyzed and provide insights to the user as well as the device itself. With so many possibilities, cheap infrastructure and high demand it is very likely that we will see an explosion of IoT startups in the coming years. Already the list of Internet connected devices is long, very long. Ranging from smart baby monitoring shirts to smart light bulbs or connected carry-ons. And the list is expanding rapidly.

However, we not only see startups focus on the Internet of Things. Of course also investors have their focus on the Internet of Things. The Chinese Internet-security company Qihoo 360 Technology just announced a global, early-stage fund with a $60 million target size for investments in Internet of Things companies. Cisco Investments has created a $ 150 million fund to invest in Internet of Things startups. Cisco actually expects that in 2014 alone, Venture Capital firms have invested $ 1.6 billion in IoT startups around the globe and this number is expected to grow in the coming year.

But not only startups are moving into the direction of the Internet of Things. Of course, Google recently bought Nest to buy itself into this market and Apple has developed Homekit. Dell actually opened an Internet of Things lab to work together with software and hardware developers to build, test and release IoT products. Probably we will see this happen a lot more in the coming year.

Until now the buzz around the Internet of Things was still relatively low, but we are about to reach a tipping point. In 2015 we will be flooded with smart products that are connected to the Internet and the data will offer great insights for companies and consumers. Were the early-connected devices primarily mobile, wearable, devices, in the coming years we will see a lot more, larger connected devices such as vehicles that use telematics to connect to the Internet and offer insights.

We are just at the beginning of the Internet of Things and it will be huge. The below infographic, created by the FOW Community, offers and overview of some of the challenges and impact the IoT will have on our businesses and lives. 

How Big Will The Internet of Things Be?

Header image Source: CSIRO
BigData-Startups Becomes Datafloq: A One-Stop Shop Around Big Data

BigData-Startups Becomes Datafloq: A One-Stop Shop Around Big Data

The Number One Big Data Knowledge Platform Continues Under a New Brand

THE HAGUE – November 24, 2014 - Two years ago, BigData-Startups was founded in order to create a platform for organizations and consumers to learn more about Big Data. In the past years, the platform has grown to become the number one knowledge platform around Big Data. Today we are excited to announce that BigData-Startups is moving forward and has transformed into Datafloq.

Datafloq is the One-Stop Shop for Big Data, creating the Big Data ecosystem by connecting all stakeholders within the global Big Data market. Datafloq is the number one Big Data platform where organizations soon will be able to find a Big Data technology vendor for their Big Data strategy and where they can find the right Big Data talent. The platform also provides valuable knowledge around Big Data, including trends, best practices, organizational advice, events and trainings. The objective of Datafloq is to spur the global understanding and application of Big Data in order to drive innovation and growth.

At first, Datafloq is built up around three different pillars, each with their own objective. Together they form a complete platform that enables organisations and consumers to move ahead with Big Data. We will continue to expand the platform with more sections that could be of interest, such as a global index of the Big Data market. The current services that Datafloq offers are:

READ: Our Read section covers all sort of Big Data knowledge to help you better understand Big Data. Many bloggers provide you with the latest content and news on Big Data, the Internet of Things, Big Data Best Practices, trends, industry information, infographics and many more. Big Data thought leaders can register to also share their knowledge on Datafloq.

MEET: This is the events section of Datafloq. You can find here any Big Data related event from around the globe. Conference producers can easily submit their own events, in order to be listed on Datafloq as well.

LEARN: As one of the objectives of Datafloq is to spur the understanding of Big Data, this section will help you with that. We have developed a unique training that will help organizations develop and implement a winning Big Data strategy.

Big Data is gaining in importance across the globe and there is a real need for trustworthy information on how to use Big Data within your organization. As Mark van Rijmenam, Co-founder and CEO of Datafloq explains: “Big Data will drastically change how organizations work and many organizations still have no idea how or where to start with Big Data. With Datafloq we aim to help organizations develop and implement a Big Data strategy that helps them grow their business and be successful.”

Crowd Control Management in the Twente Region

Crowd Control Management in the Twente Region

The combined use of data can help companies achieve more information and make better business decisions, but big data will have also a major impact on they way public services like the police, health organizations or the fire brigade operate. In The Netherlands, a remarkable, and for The Netherlands unique, initiative took place in December 2012.

During the week before Christmas, a Dutch radio station called 3FM organizes Serious Request, an annual benefit project that collects money for charity. For nine years this event is organized now and every year it is in a different location. In 2012 the event took place in Enschede, in the Twente region.  This year the Police in Twente and the Safety Region Twente developed a Crowd Control Management tool to ensure the safety among all the visitors. In 6 days, around 500.000 visitors came to the centre of Enschede and the objective was to organize a pleasant party for everyone, without any incidents. Thanks to the Crowd Control Management, they succeeded in this objective as no incidents occurred.

What did they do?


They used three different tools to monitor what was going on in real-time in the centre of Enschede:

Twitcident: Developed in conjunction with the Delft University of Technology, Twitcident is a tool that can sift through massive amounts of local tweets to find information about emergencies happening. The tool detects, filters and analyses the tweets during massive public events and presents it in a structured way so first emergency responders can use it. Twitcident provided fast and reliable information about the real-time situation in the centre of Enschede, the sentiment among the public as well the need for information in the crowd. The following video provides an insight how Twitcident works:

http://youtu.be/CF6avObWiF0

During ’Serious Request’ Twitcident worked with a list of 533 search terms that resulted in 113.000 different combinations that were monitored by the system. In total around 1.1 billion tweets were scanned. This resulted in 12.000 tweets that were marked suspicious and those tweets were checked manually in the Crowd Control Room.

UrbanShield System: This system provides a real-time situational awareness overview of a complete area within a city. This system is based on a Geographical Information System and uses GPS to show the real-time location of all first responders in an area. All police officers, fire department, city security and private security guards who are part of the system are shown on a map. Based upon a situation that is noticed via the cameras on the street or via Twitcident the closest first responder can be alerted and he or she can take immediate action.

Blue Mark: a tool that can count the crowd. During large public events it is necessary to know the amount of people in a certain location to ensure that not too many people arrive and stay at a square within the city. Blue Mark makes it possible, using sensors, to monitor the amount of people and how they move through town based on their smartphone. Each smartphone broadcasts a digital signature on a regular basis and using Bluetooth or Wifi this can be counted. No private information such as account ID or phone ID was collected, so the privacy was protected.

Crowd Control Room


These three tools where used to achieve a multi-angle, real-time, high-over picture of the situation in Enschede around “Het Glazen Huis” and on the different city squares. From the Crowd Control Room, located in the city hall, they managed the situation and when necessary they came into action: Wilma van Raalte, Program Manager of the Safety Region Twente, explains: “At a certain moment messages came in of pickpockets being active at a certain location. They were immediately traced with the cameras and seen in action. The UrbanSheild sytem found the closest police officers who could take action and within no-time the criminal was arrested and taken out of the crowd without anyone noticing it.” That shows the power of combining different tools during such events.

During this event, the tools were not automatically integrated and there was no use of public data as well. It was used as a pilot for a larger project the Twente region is working on called “Tec4se”. Tec4se will integrate traffic management, crowd management, social media, object information and public data in order to create a ‘Common Operational Picture’; a single, identical real-time display of relevant (operational) information during an event. This is Big Data used at its best and the Twente Region aims to be live with Tec4se in 2015.

Pictures courtesy of Safety Region Twente
Copyright BigData-Startups 2013. You may share using our article tools. Please don't cut articles from BigData-Startups.com and redistribute by email or post to the web.

The Australian Open Gives Fans Deeper Insights With Data

The Australian Open Gives Fans Deeper Insights With Data

For 20 years now, IBM is the main technology partner for the Australian Open. In just two decades, the Australian Open went from a ‘watch-only’ tennis tournament to a full-interactive, data-rich tennis tournament that caters for the needs of millions of fans from around the world who want to know all about their favourite tennis player.

See their infographic here, which they released to celebrate 20 years of partnership.



During this year’s Australian Open, IBM has implemented new technologies to showcase their big data capabilities and these capabilities are definitely amazing. Big data has arrived in tennis and is here to stay for the long-term. During the tournament, data is captured, analysed and shared in real-time on multiple platforms and multiple devices.

Game statistics




Using IBM’s predictive analysis technology, fans and media can follow in real-time all statistics related to all games played at a certain moment. They developed the IBM SlamTracker<sup>TM</sup>, a real-time statistics and visualization platform. The tool combines real-time data delivered from the tennis court with 39 million data points collected in the past five years at different grand slam championships.

The umpire uses a touch-screen handheld device called ‘Chump’ to enter points, faults and other important match data. This data is cable-wired to the event server room from which it is sent to three US-based data centres, located in Raleigh, Boulder and St. Louis. Here the data is replicated three times to ensure data reliability and security.

IBM’s SlamTracker<sup>TM</sup> then delivers in-depth analysis of a player’s historical performance and identifies three ‘Keys to the Match’ that a player should achieve in order to improve their chance of winning. Combined with the real-time data this give fans a deeper insight in the best strategy for success for each player.

SlamTracker Australian Open


Sentiment Analysis


IBM’s real-time data analytics software determines the sentiment among the public present and anywhere else in the world. It is a combination of sophisticated analytics software and natural language processing to analyse Twitter, Facebook, YouTube, news websites and blogs. All this information is used to create a Social Leader Board that determines the popularity of a player. The Social Leader Board shows the amount of mentions a player has received as well as the sentiment. This tool measures, understands and shows fans’ views on players throughout the event.

The media room


All data that is collected is also transported by IPTV to the media room where 300 touchscreens view all available statistics around all matches. Journalists can view match statistics, review videos from all aspects that are important and use this to create a compelling story about the Australian Open.


Preventive Scaling


All data that is collected via social media is also used to predict spikes in activity across the web. The predictive cloud provisioning technology is an intelligent cloud system that combines social media, awareness with the schedule and historical data in order to predict the amount of traffic towards the website of the Australian Open. If the system recognizes a pattern, than the system automatically scales up capacity to serve all new visitors to the website. Last year the website welcome around 49 page views during the Australian Open and expectations are that this will rise significantly this year. In order to store all data, the Australian Open website is supported by an IBM Storewize v7000 200 TB server.

What the below video for some more information.

http://youtu.be/V-_fXFxYfT0

Big Data definitely changed the game of the Australian Open this year and we will most probably see these kinds of features appear at the different grand slam championships around the world as well. It brings a complete new layer of information to the fans and journalists to better understand and experience the Australian Open. A welcome experience made possible with smart use of big data.
Copyright BigData-Startups 2013. You may share using our article tools. Please don't cut articles from BigData-Startups.com and redistribute by email or post to the web.

Privacy Policy

Copyright © 2017 BBBT - All Rights Reserved
Powered by WordPress & Atahualpa
X