Sue Brandreth's Learning Resources

Apache Hadoop
(https://en.wikipedia.org/wiki/Apache_Hadoop)

Apache Hadoop is an open-source software framework written in Java for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware. All the modules in Hadoop are designed with a fundamental assumption that hardware failures are common and should be automatically handled by the framework.

The core of Apache Hadoop consists of a storage part, known as Hadoop Distributed File System (HDFS), and a processing part called MapReduce. Hadoop splits files into large blocks and distributes them across nodes in a cluster. To process data, Hadoop transfers packaged code for nodes to process in parallel based on the data that needs to be processed. This approach takes advantage of data locality — nodes manipulating the data they have access to— to allow the dataset to be processed faster and more efficiently than it would be in a more conventional supercomputer architecture that relies on a parallel file system where computation and data are distributed via high-speed networking.

The base Apache Hadoop framework is composed of the following modules:

• Hadoop Common – contains libraries and utilities needed by other Hadoop modules;
• Hadoop Distributed File System (HDFS) – a distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster;
• Hadoop YARN – a resource-management platform responsible for managing computing resources in clusters and using them for scheduling of users’ applications;^[6][7] and
• Hadoop MapReduce – an implementation of the MapReduce programming model for large scale data processing.

The term Hadoop has come to refer not just to the base modules above, but also to the ecosystem, or collection of additional software packages that can be installed on top of or alongside Hadoop, such as Apache Pig, Apache Hive, Apache HBase, Apache Phoenix, Apache Spark, Apache ZooKeeper, Cloudera Impala, Apache Flume, Apache Sqoop, Apache Oozie, Apache Storm.

Apache Hadoop’s MapReduce and HDFS components were inspired by Google papers on their MapReduce and Google File System.

The Hadoop framework itself is mostly written in the Java programming language, with some native code in C and command line utilities written as shell scripts. Though MapReduce Java code is common, any programming language can be used with “Hadoop Streaming” to implement the “map” and “reduce” parts of the user’s program.Other projects in the Hadoop ecosystem expose richer user interfaces.

---

MapReduce
(https://en.wikipedia.org/wiki/MapReduce)

One such project was an open-source web search engine called Nutch – the brainchild of Doug Cutting and Mike Cafarella. They wanted to return web search results faster by distributing data and calculations across different computers so multiple tasks could be accomplished simultaneously. During this time, another search engine project called Google was in progress. It was based on the same concept – storing and processing data in a distributed, automated way so that relevant web search results could be returned faster.

In 2006, Cutting joined Yahoo and took with him the Nutch project as well as ideas based on Google’s early work with automating distributed data storage and processing. The Nutch project was divided – the web crawler portion remained as Nutch and the distributed computing and processing portion became Hadoop (named after Cutting’s son’s toy elephant). In 2008, Yahoo released Hadoop as an open-source project. Today, Hadoop’s framework and ecosystem of technologies are managed and maintained by the non-profit Apache Software Foundation (ASF), a global community of software developers and contributors.

What are the challenges of using Hadoop?

MapReduce programming is not a good match for all problems. It’s good for simple information requests and problems that can be divided into independent units, but it’s not efficient for iterative and interactive analytic tasks. MapReduce is file-intensive. Because the nodes don’t intercommunicate except through sorts and shuffles, iterative algorithms require multiple map-shuffle/sort-reduce phases to complete. This creates multiple files between MapReduce phases and is inefficient for advanced analytic computing.

There’s a widely acknowledged talent gap. It can be difficult to find entry-level programmers who have sufficient Java skills to be productive with MapReduce. That’s one reason distribution providers are racing to put relational (SQL) technology on top of Hadoop. It is much easier to find programmers with SQL skills than MapReduce skills. And, Hadoop administration seems part art and part science, requiring low-level knowledge of operating systems, hardware and Hadoop kernel settings.

Data security. Another challenge centers around the fragmented data security issues, though new tools and technologies are surfacing. The Kerberos authentication protocol is a great step toward making Hadoop environments secure.

Full-fledged data management and governance. Hadoop does not have easy-to-use, full-feature tools for data management, data cleansing, governance and metadata. Especially lacking are tools for data quality and standardization.

---

What is Apache Hadoop?
(http://hortonworks.com/hadoop/)

The open source framework for storing and extracting insight from massive volumes of data.

Hortonworks Data Platform

In this guide, discover:

• The technology components of HDP and its blueprint for Enterprise Hadoop.
• How HDP integrates and complements your existing data systems.
• The versatility of applications enabled by Apache Hadoop YARN at the core of HDP.
• How HDP provides security, operations and governance capabilities.
• Deployment for HDP from Linux to Windows, On-Premise to In-Cloud.
• How HDP assists all stakeholders in data teams: from architects to scientists.

HDP is the 100% Open Source Blueprint for Apache Hadoop. Download the Guide Now.

---

Apache Hadoop – Cloudera
(https://www.cloudera.com/products/apache-hadoop.html)

Apache Hadoop
Hadoop is an ecosystem of open source components that fundamentally changes the way enterprises store, process, and analyze data. Unlike traditional systems, Hadoop enables multiple types of analytic workloads to run on the same data, at the same time. CDH, Cloudera’s open source platform, is the most popular distribution of Hadoop and related projects in the world (with support available via a Cloudera Enterprise subscription).

Why Cloudera?
(https://www.cloudera.com/why-cloudera.html)

---

What is Apache Hadoop?
(https://www.mapr.com/products/apache-hadoop)

Hadoop and Big Data
Apache Hadoop™ was born out of a need to process an avalanche of big data. The web was generating more and more information on a daily basis, and it was becoming very difficult to index over one billion pages of content. In order to cope, Google invented a new style of data processing known as MapReduce. A year after Google published a white paper describing the MapReduce framework, Doug Cutting and Mike Cafarella, inspired by the white paper, created Hadoop to apply these concepts to an open-source software framework to support distribution for the Nutch search engine project. Given the original case, Hadoop was designed with a simple write-once storage infrastructure.

Hadoop has moved far beyond its beginnings in web indexing and is now used in many industries for a huge variety of tasks that all share the common theme of lots of variety, volume and velocity of data – both structured and unstructured. It is now widely used across industries, including finance, media and entertainment, government, healthcare, information services, retail, and other industries with big data requirements but the limitations of the original storage infrastructure remain.

---

Hadoop Tutorial
(http://www.tutorialspoint.com/hadoop/)

Hadoop is an open-source framework that allows to store and process big data in a distributed environment across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

This brief tutorial provides a quick introduction to Big Data, MapReduce algorithm, and Hadoop Distributed File System.

---

Apache Hadoop
(https://developer.yahoo.com/hadoop/)

Hadoop at Yahoo!
Introduction
Apache Hadoop* is an open source Java framework for processing and querying vast amounts of data on large clusters of commodity hardware. Hadoop is a top level Apache project, initiated and led by Yahoo!. It relies on an active community of contributors from all over the world for its success.

With a significant technology investment by Yahoo!, Apache Hadoop has become an enterprise-ready cloud computing technology. It is becoming the industry de facto framework for big data processing.
The Hadoop project is an integral part of the Yahoo! cloud infrastructure — and is the heart of many of Yahoo!’s important business processes.

We run the world’s largest Hadoop clusters, work with academic institutions and other large corporations on advanced cloud computing research and our engineers are leading participants in the Hadoop community.

What’s new from Yahoo!?
Hadoop with security

Hadoop with security is a significant update to Apache Hadoop. This update integrates Hadoop with Kerberos, a mature open source authentication standard.
Hadoop with security:

• Prevents unauthorized access to data on Hadoop clusters
• Authenticates users sharing business sensitive data
• Reduces operational costs by consolidating Hadoop clusters
• Collocates data for new classes of applications

Oozie – Yahoo!’s workflow engine for Hadoop
Oozie, Yahoo!’s workflow engine for Hadoop is an open-source workflow solution to manage and coordinate jobs running on Hadoop, including HDFS, Pig and MapReduce.
Oozie was designed for Yahoo!’s complex workflows and data pipelines at global scale. It is integrated with the Yahoo! Distribution of Hadoop with security and is a primary mechanism to manage complex data analysis workloads across Yahoo!.

---

Hadoop: What it is, how it works, and what it can do
(https://www.oreilly.com/ideas/what-is-hadoop)

Cloudera CEO Mike Olson on Hadoop’s architecture and its data applications.
Hadoop gets a lot of buzz these days in database and content management circles, but many people in the industry still don’t really know what it is and or how it can be best applied.
Cloudera CEO and Strata speaker Mike Olson, whose company offers an enterprise distribution of Hadoop and contributes to the project, discusses Hadoop’s background and its applications in this interview.

---

Hadoop Platform and Application Framework
(https://www.coursera.org/learn/hadoop)

University of California, San Diego
Part of a 6-course series, the Big Data Specialization

Drive better business decisions with an overview of how big data is organized, analyzed, and interpreted. Apply your insights to real-world problems and questions. Do you need to understand big data and how it will impact your business? This Specialization is for you. You will gain an understanding of what insights big data can provide through hands-on experience with the tools and systems used by big data scientists and engineers. Previous programming experience is not required! You will be guided through the basics of using Hadoop with MapReduce, Spark, Pig and Hive. By following along with provided code, you will experience how one can perform predictive modeling and leverage graph analytics to model problems. This specialization will prepare you to ask the right questions about data, communicate effectively with data scientists, and do basic exploration of large, complex datasets. In the final Capstone Project, developed in partnership with data software company Splunk, you’ll apply the skills you learned to do basic analyses of big data.

---

Hadoop: What It Is And How It Works
(http://readwrite.com/2013/05/23/hadoop-what-it-is-and-how-it-works/)

You can’t have a conversation about Big Data for very long without running into the elephant in the room: Hadoop. This open source software platform managed by the Apache Software Foundation has proven to be very helpful in storing and managing vast amounts of data cheaply and efficiently.

But what exactly is Hadoop, and what makes it so special? Basically, it’s a way of storing enormous data sets across distributed clusters of servers and then running “distributed” analysis applications in each cluster.
It’s designed to be robust, in that your Big Data applications will continue to run even when individual servers — or clusters — fail. And it’s also designed to be efficient, because it doesn’t require your applications to shuttle huge volumes of data across your network.

Here’s how Apache formally describes it:

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly available service on top of a cluster of computers, each of which may be prone to failures.

Look deeper, though, and there’s even more magic at work. Hadoop is almost completely modular, which means that you can swap out almost any of its components for a different software tool. That makes the architecture incredibly flexible, as well as robust and efficient.

---

Spring for Apache Hadoop
(http://projects.spring.io/spring-hadoop/)

Introduction
Spring for Apache Hadoop simplifies developing Apache Hadoop by providing a unified configuration model and easy to use APIs for using HDFS, MapReduce, Pig, and Hive. It also provides integration with other Spring ecosystem project such as Spring Integration and Spring Batch enabling you to develop solutions for big data ingest/export and Hadoop workflow orchestration.

Features

• Support to create Hadoop applications that are configured using Dependency Injection and run as standard Java applications vs. using Hadoop command line utilities.
• Integration with Spring Boot to simply creat Spring apps that connect to HDFS to read and write data.
• Create and configure applications that use Java MapReduce, Streaming, Hive, Pig, or HBase
• Extensions to Spring Batch to support creating Hadoop based workflows for any type of Hadoop Job or HDFS operation.
• Script HDFS operations using any JVM based scripting language.
• Easily create custom Spring Boot based aplications that can be deployed to execute on YARN.
• DAO support (Template & Callbacks) for HBase.
• Support for Hadoop Security.

---

Spark and Hadoop on Google Cloud Platform (https://cloud.google.com/hadoop/)

You can run powerful and cost-effective Apache Spark and Apache Hadoop clusters easily on Google Cloud Platform. When you run Hadoop and Spark on Google Cloud Platform, you combine these powerful open source ecosystems with Google’s reliable and highly-scalable infrastructure, Google’s pricing philosophy, and a powerful portfolio of cloud technologies.

There are many easy ways to get started with Spark and Hadoop on Google Cloud Platform, including:

1. Google Cloud Dataproc — A managed Spark and Hadoop service that allows anyone to create and use fast, easy, and cost-effective clusters
2. Command line tools (bdutil) — A collection of shell scripts to manually create and manage Spark and Hadoop clusters
3. Third party Hadoop distributions:
   • Cloudera — Using the Cloudera Director Plugin for Google Cloud Platform
   • Hortonworks — Using bdutil support for Hortonworks HDP
   • MapR — Using bdutil support for MapR

---

Hadoop Course
(http://bigdatauniversity.com/courses/hadoop-course/)

Hadoop Fundamentals I
Hadoop Fundamentals I teaches you the basics of Apache Hadoop and the concept of Big Data. This Hadoop course is entirely free, and so are the materials and software provided. This is the third version of our most popular Hadoop course. Since Version 2 was published, several more detailed courses covering topics such as MapReduce, Hive, HBase, Pig, Oozie, and Zookeeper have been added.  We recommend you start here and then dig deeper into the specific Hadoop technology you wish to learn more about.

Learn Hadoop
This Hadoop course is designed to give you a basic understanding of key Big Data technologies. In this Hadoop tutorial, we first begin with describing what Big Data is and the need for Hadoop to be able to process that data in a timely manner. This is followed by describing the Hadoop architecture and how to work with the Hadoop Distributed File System (HDFS) both from the command line and using the BigInsights Console that is supplied with InfoSphere BigInsights.

Hadoop Developer Day Event
(http://bigdatauniversity.com/courses/hadoop-hackathon-onsite-and-live-stream-event/)

• Understand Hadoop and the Big Data Problems It Can Solve
• Know the key open source components related to Hadoop
• Understand how Big Data solutions can work in the Cloud
• Understand the components of Hadoop and how to leverage HDFS
• Know the essential steps involved with implementing a MapReduce program that runs in Hadoop
• Use Hive to implement data warehousing solutions in a Hadoop environment (using HiveQL)
• Understand the BigSQL approach to managing structured data in Hadoop
• Use BigSQL to store and query large data sets in Hadoop

---

---

The history of Hadoop: From 4 nodes to the future of data
(https://gigaom.com/2013/03/04/the-history-of-hadoop-from-4-nodes-to-the-future-of-data/)

Depending on how one defines its birth, Hadoop is now 10 years old. In that decade, Hadoop has gone from being the hopeful answer to Yahoo’s (s yhoo) search-engine woes to a general-purpose computing platform that’s poised to be the foundation for the next generation of data-based applications.

Alone, Hadoop is a software market that IDC predicts will be worth $813 million in 2016 (although that number is likely very low), but it’s also driving a big data market the research firm predicts will hit more than $23 billion by 2016. Since Cloudera launched in 2008, Hadoop has spawned dozens of startups and spurred hundreds of millions in venture capital investment since 2008.

In this four-part series, we’ll explain everything anyone concerned with information technology needs to know about Hadoop. Part I is the history of Hadoop from the people who willed it into existence and took it mainstream. Part II is more graphic; a map of the now-large and complex ecosystem of companies selling Hadoop products. Part III is a look into the future of Hadoop that should serve as an opening salvo for much of the discussion at our Structure: Data conference March 20-21 in New York. Finally, part IV will highlight some the best Hadoop applications and seminal moments in Hadoop history, as reported by GigaOM over the years.

---

Top 6 Hadoop Vendors providing Big Data Solutions in Open Data Platform
(https://www.dezyre.com/article/-top-6-hadoop-vendors-providing-big-data-solutions-in-open-data-platform/93)

With the demand for big data technologies expanding rapidly, Apache Hadoop is at the heart of the big data revolution. It is labelled as the next generation platform for data processing because of its low cost and ultimate scalable data processing capabilities. The open source framework hadoop is somewhat immature and big data analytics companies are now eyeing on Hadoop vendors- a growing community that delivers robust capabilities, tools and innovations for improvised commercial hadoop big data solutions. Here are top 6 big data analytics vendors that are serving Hadoop needs of various big data companies by providing commercial support.

Allied Market Research predicts that the “Hadoop-as-a-Service” market will grow to $50.2 billion by 2020. The Global Hadoop Market is anticipated to reach $8.74 billion by 2016, growing at a CAGR of 55.63 % from 2012–2016. Wikibon’s latest market analysis states that- spending on  Hadoop software and subscriptions accounted for less than 1% of $27.4 billion or approximately $187 million in 2014 in overall Big Data spending. Wikibon predicts that the spending on Hadoop software and subscriptions will increase to approximately $677 million by the end of 2017, with overall big data market anticipated to reach the $50 billion mark.

Big Data and Hadoop are on the verge of revolutionizing enterprise data management architectures. Cloud and enterprise vendors are competing to venture a claim in the big data ‘gold-rush market’ with pure plays of several top Hadoop Vendors. Apache Hadoop is an open source big data technology with HDFS, Hadoop Common, Hadoop MapReduce and Hadoop YARN as the core components .However, without the packaged solutions and support of commercial Hadoop vendors, Hadoop distributions can just go unnoticed.

Need for Commercial Hadoop Vendors
Today, Hadoop is an open-source, catch-all technology solution with incredible scalability, low cost storage systems and fast paced big data analytics with economical server costs.
Hadoop Vendor distributions overcome the drawbacks and issues with the open source edition of Hadoop. These distributions have added functionalities that focus on:
Support:
Most of the Hadoop vendors provide technical guidance and assistance that makes it easy for customers to adopt Hadoop for enterprise level tasks and mission critical applications.
Reliability:
Hadoop vendors promptly act in response whenever a bug is detected. With the intent to make commercial solutions more stable, patches and fixes are deployed immediately.
Completeness:
Hadoop vendors couple their distributions with various other add-on tools which help customers customize the Hadoop application to address their specific tasks.

Top Commercial Hadoop Vendors
Here is a list of top Hadoop Vendors who will play a key role in big data market growth for the coming years:

1) Amazon Web Services Elastic MapReduce Hadoop Distribution
The Amazon Hadoop Vendor has been there since the dawn of Hadoop, and Hadoopers boast of its success stories for the innovative Hadoop distributions in the open data platform. AWS Elastic MapReduce renders an easy to use and well organized data analytics platform built on the powerful HDFS architecture. With major focus on map/reduce queries, AWS EMR exploits Hadoop tools to a great extent by providing a high scale and secure infrastructure platform to its users. Amazon Web Services EMR is among one of the top commercial Hadoop distributions with the highest market share leading the global market.

AWS EMR handles important big data uses like web indexing, scientific simulation, log analysis, bioinformatics, machine learning, financial analysis and data warehousing. AWS EMR is the best choice for organizations who do not want to manage thousands of servers directly – as they can rent out this cloud ready infrastructure of Amazon for big data analysis.

DynamoDB is another major  NoSQL database offering by AWS Hadoop Vendor that was deployed to run its giant consumer website. Redshift is a completely managed petabyte scale data analytics solution that is cost effective in big data analysis with BI tools. Redshift has costs as low as $1000 per terabyte annually. According to Forrester, Amazon is the “King of the Cloud” – for companies in need of public cloud hosted Hadoop platforms for big data management services.

2) Hortonworks Hadoop Distribution
Hortonworks Hadoop vendor, features in the list of Top 100 winners of “Red Herring”. Hortonworks is a pure play Hadoop company that drives open source Hadoop distributions in the IT market. The main goal of Hortonworks is to drive all its innovations through the Hadoop open data platform and build an ecosystem of partners that speeds up the process of Hadoop adoption amongst enterprises.

Apache Ambari is an example of Hadoop cluster management console developed by Hortonworks Hadoop vendor for provision, managing and monitoring Hadoop clusters. The Hortonworks Hadoop vendor is reported to attract 60 new customers every quarter with some giant accounts like Samsung, Spotify, Bloomberg and eBay. Hortonworks has garnered strong engineering partnerships with RedHat, Microsoft, SAP and Teradata.

Hortonworks has grown its revenue at a rapid pace. The revenue generated by Hortonworks totaled $33.38 million in first nine months of 2013 which was a significant increase by 109.5% from the previous year. However, the professional services revenue generated by Hortonworks Hadoop vendor increases at a faster pace when compared to support and subscription services revenue.

3) Cloudera Hadoop Distribution
Cloudera Hadoop Vendor ranks top in the big data vendors list for making Hadoop a reliable platform for business use since 2008.Cloudera, founded by a group of engineers from Yahoo, Google and Facebook – is focused on providing enterprise ready solutions of Hadoop with additional customer support and training. Cloudera Hadoop vendor has close to 350 paying customers including the U.S Army, AllState and Monsanto. Some of them boast of deploying 1000 nodes on a Hadoop cluster to crunch big data analytics for one petabyte of data. Cloudera owes its long term success to corporate partners – Oracle, IBM, HP, NetApp and MongoDB that have been consistently pushing its services.

Cloudera Hadoop vendor is just on the right path towards its goal with 53% of the Hadoop market when compared to 11% of Hadoop Market possessed by MapR and 16% by Hortonworks Hadoop vendors. Forrester says “Cloudera’s approach to innovation is to be loyal to core Hadoop but to innovate quickly and aggressively to meet customer demands and differentiate its solution from those of other commercial Hadoop vendors.”

4) MapR Hadoop Distribution
MapR has been recognized extensively for its advanced distributions in Hadoop marking a place in the Gartner report “Cool Vendors in Information Infrastructure and Big Data, 2012.” MapR has scored the top place for its Hadoop distributions amongst all other vendors.

MapR has made considerable investments to get over the obstacles to worldwide adoption of Hadoop which include enterprise grade reliability, data protection, integrating Hadoop into existing environments with ease and infrastructure to render support for real time operations.

In 2015, MapR plans to make further investments to maintain its significance in the Big Data vendors list. Apart from this MapR is all set to announce its technical innovations for Hadoop with the intent of supporting ‘business-as-it-happens’- to increase revenue, mitigate risks and reduce costs.

5) IBM Infosphere BigInsights Hadoop Distribution
IBM Infosphere BigInsights is an industry standard IBM Hadoop distribution that combines Hadoop with enterprise grade characteristics.IBM provides BigSheets and BigInsights as a service via its Smartcloud Enterprise Infrastructure .With IBM Hadoop distributions users can easily set up and move data to Hadoop clusters in no more than 30 minutes with data processing rate of 60 cents per Hadoop cluster, per hour. With IBM BigInsights innovation, customers can get to market at a rapid pace with their applications that incorporate advanced Big Data analytics by harnessing the power of Hadoop.6) Microsoft Hadoop Distribution
Forrester rates Microsoft Hadoop Distribution as 4/5- based on the Big Data Vendor’s current Hadoop Distributions, market presence and strategy – with Cloudera and Hortonworks scoring 5/5.

Microsoft is an IT organization not known for embracing open source software solutions, but it has made efforts to run this open data platform software on Windows. Hadoop as a service offering by Microsoft’s big data solution is best leveraged through its public cloud product -Windows Azure’s HDInsight particularly developed to run on Azure. There is another production ready feature of Microsoft named Polybase that lets the users search for information available on SQL Server during the execution of Hadoop queries. Microsoft has great significance in delivering a growing Hadoop Stack to its customers.

Commercial Hadoop Vendors continue to mature overtime with increased worldwide adoption of Big Data technologies and growing vendor revenue. There are several top Hadoop vendors namely Hortonworks, Cloudera, Microsoft and IBM. These Hadoop vendors are facing a tough competition in the open data platform. With the war heating up amongst big data vendors, nobody is sure as to who will top the list of commercial Hadoop vendors. With Hadoop buying cycle on the upswing,  Hadoop vendors must capture the market share at a rapid pace to make the venture investors happy.

---

The Hadoop Ecosystem Table
(https://hadoopecosystemtable.github.io/)

This page is a summary to keep the track of Hadoop related projects, focused on FLOSS environment.

---

Udemy Free Online Video Courses
(https://www.udemy.com/)

Big Data and Hadoop Essentials
Essential Knowledge for everyone associated with Big Data & Hadoop
(https://www.udemy.com/big-data-and-hadoop-essentials-free-tutorial/learn/v4/overview)

Are you interested in the world of Big data technologies, but find it a little cryptic and see the whole thing as a big puzzle.

Are you looking to understand how Big Data impact large and small business and people like you and me?
Do you feel many people talk about Big Data and Hadoop, and even do not know the basics like history of Hadoop, major players and vendors of Hadoop. Then this is the course just for you!

This course builds a essential fundamental understanding of Big Data problems and Hadoop as a solution. This course takes you through:

1. Understanding of Big Data problems with easy to understand examples.
2. History and advent of Hadoop right from when Hadoop wasn’t even named Hadoop.
3. What is Hadoop Magic which makes it so unique and powerful.
4. Understanding the difference between Data science and data engineering, which is one of the big confusions in selecting a carrier or understanding a job role.
5. And most importantly, demystifying Hadoop vendors like Cloudera, MapR and Hortonworks by understanding about them.

Big Data Basics: Hadoop, MapReduce, Hive, Pig, & Spark
Learn about some of the most popular big data analysis frameworks in commercial use – and look at some real code!
(https://www.udemy.com/big-data-basics-hadoop-mapreduce-hive-pig-spark/learn/v4/overview)

Interested in analyzing big data sets? You should be – according to CareerCast, “data scientist” is the 8th most highly paid profession in the United States! If you have some coding or scripting background, you can make your experience even more valuable by understanding how to use Hadoop, MapReduce, Hive, Pig, and Spark to crunch huge data sets in parallel.

These techniques are used by some of the largest and most prestigious tech employers, including Google, Facebook, Twitter, Amazon, EBay, Yahoo, and many more. After this course, you’ll speak their language!

What We’ll Cover:

• What is MapReduce and Hadoop?
• What are some real-world applications of these technologies?
• A walk-through of designing, coding, and running a real example of MapReduce using real data.
• How Hadoop distributes computing across a cluster of machines
• An overview of Hive, Pig, and Spark along with a couple of small examples.

Hadoop Starter Kit
Hadoop learning made easy and fun. Learn HDFS, MapReduce and introduction to Pig and Hive with FREE cluster access.
(https://www.udemy.com/hadoopstarterkit/learn/v4/overview)

The objective of this course is to walk you through step by step of all the core components in Hadoop but more importantly make Hadoop learning experience easy and fun.

By enrolling in this course you can also get free access to our multi-node Hadoop training cluster so you can try out what you learn right away in a real multi-node distributed environment.

In the first section you will learn about what is big data with examples. We will discuss the factors to consider when considering whether a problem is big data problem or not. We will talk about the challenges with existing technologies when it comes to big data computation. We will breakdown the Big Data problem in terms of storage and computation and understand how Hadoop approaches the problem and provide a solution to the problem.
In the HDFS, section you will learn about the need for another file system like HDFS. We will compare HDFS with traditional file systems and its benefits. We will also work with HDFS and discuss the architecture of HDFS.

In the MapReduce section you will learn about the basics of MapReduce and phases involved in MapReduce. We will go over each phase in detail and understand what happens in each phase. Then we will write a MapReduce program in Java to calculate the maximum closing price for stock symbols from a stock dataset.

In the next two sections, we will introduce you to Apache Pig & Hive. We will try to calculate the maximum closing price for stock symbols from a stock dataset using Pig and Hive.

Hadoop Big Data – Must See Introduction to Big Data
Get Your Feet Wet in Big Data: What can you do with Big Data and Hadoop Developing
(https://www.udemy.com/hadoop-big-data-must-see-introduction-to-big-data/)

Welcome to this course on Big Data’s percolation into E-commerce: How Big Data in Cloud can be an added advantage. This course is targeted towards the tech enthusiasts who are willing to keep themselves updated in the technology space. At the end of this course, we aim to help the students Get an understanding of the practical implications of implementing Big Data projects in E-Commerce companies; Get a fair level knowledge on what is Big Data and how it evolved; Spot some of the latest updates in Big Data; Understand the level of maturity attained by Cloud Computing; Derive a high level knowledge on how cloud environment is favourable for implementing Big Data projects; Study some business drivers that is motivating firms to move into the Data-Cloud models.

Data Analytics using Hadoop eco system
Convert NYSE data into useful insights. In this course we will perform top down analysis of stock data based on volume.
(https://www.udemy.com/data-analytics-using-hadoop-in-2-hours/learn/v4/overview)

---

What is Hadoop?
Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a distributed computing environment. It is part of the Apache project sponsored by the Apache Software Foundation.

Big data applications: Real-world strategies for managing big data

(http://searchdatamanagement.techtarget.com/essentialguide/Big-data-applications-Real-world-strategies-for-managing-big-data)

Introduction
Big data management strategies and best practices are still evolving, but joining the big data movement has become an imperative for companies across a wide variety of industries. This guide delves into the experiences of early-adopter companies that have already deployed big data applications and technologies. IT professionals, C-level executives and industry analysts offer insights into what strategies work on big data projects and how to best integrate big data management initiatives with related processes such as data warehousing, data governance and data analytics.

The following stories explore the steps these companies took to set up big data systems and to update their approaches as needed. Readers will find practical information on implementing big data strategies, mixing Hadoop clusters and conventional data warehousing tools, incorporating big data analytics into the process and translating big data ideas into successful deployments.

Big data strategy

Advertising firm attracts clients with big data strategy
The search advertising company adMarketplace processes billions of ad requests daily in near real time using a pay-per-click platform. Due partly to the high level of data customization it offers to advertisers, adMarketplace processes 100 gigabytes of data per hour. The first article below explains how the company implemented a platform combining a traditional data warehouse and a NoSQL database to power the big data environment that feeds its search syndication system. Other stories in this section offer more insights into managing and using big data and how it fits into the data warehousing and data governance process.

---

Jhajj, Raman – Apache Hadoop Cookbook – Hot Recipes for Apache Hadoop
(JavaCodeGeeks)
Download from here