Alternatives to DAS in Hadoop storage - Hadoop Training in chennai

Alternatives to DAS in Hadoop storage

We concluded the last article in this series on managing "big data" with a question: DoesHadoop storage have to be direct-attached storage (DAS)? This article introduces some alternatives to the strict use of embedded DAS for hardware-based storage within theHadoop MapReduce framework.

Direct-attached storage (DAS) is computer storage that is directly attached to one computer or server and is not, without special support, directly accessible to other ones. The main alternatives to direct-attached storage are network-attached storage (NAS) and the storage area network (SAN).

For an individual computer user, the hard drive is the usual form of direct-attached storage. In an enterprise, providing for storage that can be shared by multiple computers and their users tends to be more efficient and easier to manage.

To do so, we'll examine the alternatives in terms of a three-stage model:

Stage one: DAS in the form of small amounts of disk (JBOD/RAID) embedded within each cluster node is replaced with larger, high-performance arrays that are external to the cluster nodes that are still directly attached to provide data locality. In a way, we’re rephrasing our original question:

Does Hadoop data storage have to be relatively small groupings of DASembedded within each cluster node? No, but the larger external storage arrays that replace embedded DAS still function as DAS.

Stage two: The node-based DAS layer used as primary storage by the cluster is augmented with the addition of a second storage layer consisting of network-attached storage (NAS) or a storage-area network (SAN).

Stage three: The node-based DAS layer used as primary storage is replaced by a networked storage layer consisting of NAS or a SAN.

Learn Big Data from Big Data Solutions Architects! Hadoop Training Chennai with Hands-On Practical 

 BigDataTraining.IN - India's Leading BigData Consulting & Training Provider

BigDataTraining.IN is a leading Global Talent Development Corporation, building skilled manpower pool for global industry requirements. BigData Training.in has today grown to be amongst world's leading talent development companies offering learning solutions to Individuals, Institutions & Corporate Clients.

BigDataTraining.IN is the only software institute which offers in-depth classroom & online training in different Cutting   

Hadoop Training  Conducted by Big Data Experts 

http://www.bigdatatraining.in/hadoop-development/training-schedule/

Classroom / Online / Corporate Training

 get connected with BigDataTraining.IN

http://www.hadooptrainingchennai.in/

http://www.hadooptrainingchennai.in/courses/

http://www.hadooptrainingchennai.in/hadoop-training/

http://www.hadooptrainingchennai.in/course-content/

Contact us:
Mail: info@bigdatatraining.in
Call: +91 97899 68765 / 044 – 42645495
Visit us: #67,2nd Floor, 1st Main Road, Gandhi Nagar, Adyar, Chennai- 600020

 

Apache Pig Hadoop Training in chennai - BigDataTraining.IN

Pig Latin Statements

A Pig Latin statement is an operator that takes a relation as input and produces another relation as output. (This definition applies to all Pig Latin operators except LOAD and STORE which read data from and write data to the file system.) Pig Latin statements can span multiple lines and must end with a semi-colon ( ; ). Pig Latin statements are generally organized in the following manner:

1. A LOAD statement reads data from the file system.
2. A series of "transformation" statements process the data.
3. A STORE statement writes output to the file system; or, a DUMP statement displays output to the screen.

 Pig Latin is a relatively simple language that executes statements. A statement is an operation that takes input (such as a bag, which represents a set of tuples) and emits another bag as its output. A bag is a relation, similar to table, that you'll find in a relational database (where tuples represent the rows, and individual tuples are made up of fields).

A script in Pig Latin often follows a specific format in which data is read from the file system, a number operations are performed on the data (transforming it in one or more ways), and then the resulting relation is written back to the file system.

 BigDataTraining.IN - India's Leading BigData Consulting & Training Provider, Request a Quote!

Pig has a rich set of data types, supporting not only high-level concepts like bags, tuples, and maps, but also simple data types such as ints, longs, floats, doubles, chararrays, and bytearrays. With the simple types, you'll find a range of arithmetic operators (such as add, subtract, multiply, divide, and module) in addition to a conditional operator called bincond that operates similar to the C ternary operator. And as you'd expect, a full suite of comparison operators, including rich pattern matching using regular expressions.

All Pig Latin statements operate on relations (and are called relational operators).  there's an operator for loading data from and storing data in the file system. There's a means to FILTER data by iterating the rows of a relation. This functionality is commonly used to remove data from the relation that is not needed for subsequent operations. Alternatively, if you need to iterate the columns of a relation instead of the rows, you can use the FOREACH operator. FOREACH permits nested operations such as FILTER and ORDER to transform the data during the iteration.

The ORDER operator provides the ability to sort a relation based on one or more fields. The JOIN operator performs an inner or outer join of two or more relations based on common fields. The SPLIT operator provides the ability to split a relation into two or more relations based on a user-defined expression. Finally, the GROUP operator groups the data in one or more relations based on some expression.

 http://www.bigdatatraining.in/contact/
http://www.bigdatatraining.in/hadoop-development/training-schedule/

Hadoop Training Chennai with Hands-On Practical Approach !

 

Mail:
info@bigdatatraining.in

Call:
+91 9789968765
044 - 42645495

Visit Us:
#67, 2nd Floor, Gandhi Nagar 1st Main Road, Adyar, Chennai - 20
[Opp to Adyar Lifestyle Super Market]

Apache Sqoop for importing data from a relational DB

Sqoop

Sqoop is a tool designed to transfer data between Hadoop and relational databases. You can use Sqoop to import data from a relational database management system (RDBMS) such as MySQL or Oracle into the Hadoop Distributed File System (HDFS), transform the data in Hadoop MapReduce, and then export the data back into an RDBMS.

Sqoop automates most of this process, relying on the database to describe the schema for the data to be imported. Sqoop uses MapReduce to import and export the data, which provides parallel operation as well as fault tolerance.

This document describes how to get started using Sqoop to move data between databases and Hadoop and provides reference information for the operation of the Sqoop 

command-line tool suite. 

This document is intended for:

• System and application programmers
• System administrators
• Database administrators
• Data analysts
• Data engineers

Using Apache Sqoop for Data Import from Relational DBs

 

Apache Sqoop can be used to import data from any relational DB into HDFS, Hive or HBase.

To import data into HDFS, use the sqoop import command and specify the relational DB table and connection parameters:

sqoop import --connect <JDBC connection string> --table <tablename> --username

                                            <username> --password <password>

This will import the data and store it as a CSV file in a directory in HDFS.

To import data into Hive, use the sqoop import command and specify the option ‘hive-import’.

sqoop import --connect <JDBC connection string> --table <tablename> --username

                               <username> --password <password> --hive-import

 

 Hadoop & Big Data Training | Development | Consulting | Projects

http://www.bigdatatraining.in/hadoop-training-chennai/

http://www.hadooptrainingchennai.in/hadoop-training-in-chennai/

http://www.bigdatatraining.in/

http://www.bigdatatraining.in/contact/

http://www.bigdatatraining.in/hadoop-development/training-schedule/

Mail:
info@bigdatatraining.in

Call:
+91 9789968765
044 - 42645495

Visit Us:
#67, 2nd Floor, Gandhi Nagar 1st Main Road, Adyar, Chennai - 20
[Opp to Adyar Lifestyle Super Market]

HBase -Big Data Hadoop Training in Bangalore @ BigDataTraining.IN

What is HBase?
“HBase is a distributed, persistent, strictly consistent storage system with near-optimal write-- in terms of  I/O channel saturation-- and excellent read  performance, and it makes efficient use of disk space  by supporting pluggable compression algorithms that  can be selected based on the nature of the data in  specific column families”

Why HBase?
Handles unstructured or semi-structured data.
Handles enormous data volumes.
Flexible. Ad-Hoc access as well as Full/Partial Table Scans.
Cost-effective scalability.
Near Linear scalability.
Part of the Hadoop Ecosystem.

HBase Architecture

HBase: HMaster

Responsible for assigning regions to Region Servers (RS)
• Handles load balancing of regions across Region Servers
• Has nothing to do with the actual data access.
• Schema and metadata management.
• Fairly Lightweight Process

HBase: Region Server
Responsible for the read/write requests.
• Handles region splits
• Clients communicate directly with the Region Servers
• Creates their own ephemeral node in ZooKeeper
• Region Servers contain the following:
• WAL (HLOG)
• Memstore
• HFile

Learn what the industry is in need of – Showcase required BigData Expertise with real experience integrating a BigData Workflow as Proof of Concept(PoC) Project work. Get hands-on Expertise – Powered by our Expert Team!

BigDataTraining.IN is an organization of learners who aim to advance individual and organizational competitiveness by acquiring and applying right knowledge.

24/7 Technical Support

To facilitate smooth training, In addition to our CloudLab Portal- we offer 24/7 Technical Support.

Our Expertise

* Work / Learn on production level Cloud Servers
* Big Data Thought Leadership
* Primary focus - hands-on sessions
* Leaders in Big Data Consulting Services in India.
http://www.bigdatatraining.in/bigdata-consulting/big-data-services/
* Founders of Big Data Real Time Analytics Platform
http://www.bigdatatraining.in/announcing-our-big-data-analytics-platform/

http://www.bigdatatraining.in/amazon-elastic-mapreduce-training/

http://www.bigdatatraining.in/launching-apache-cassandra-training/

http://www.bigdatatraining.in/launching-mongodb-training/

http://www.hadooptrainingindia.in/hadoop-bigdata-training-online/  BigDataTraining.IN is a leading Global Talent Development Corporation, building skilled manpower pool for global industry requirements. BigData Training.in has today grown to be amongst world's leading talent development companies offering learning solutions to Individuals, Institutions & Corporate Clients.

BigDataTraining.IN has a strong focus and established thought leadership in the area of Big Data and Analytics. We use a global delivery model to help you to evaluate and implement solutions tailored to your specific technical and business context.

http://www.bigdatatraining.in/hadoop-development/training-schedule/

http://www.bigdatatraining.in/contact/

Mail:
info@bigdatatraining.in

Call:
+91 9789968765
044 - 42645495

Visit Us:
#67, 2nd Floor, Gandhi Nagar 1st Main Road, Adyar, Chennai - 20
[Opp to Adyar Lifestyle Super Market]

Secondary Name Node - hadoop HDFS Training

Secondary Name Node
 

 

Hadoop has server role called the Secondary Name Node.  A common misconception is that this role provides a high availability backup for the Name Node.  This is not the case.

The Secondary Name Node occasionally connects to the Name Node (by default, ever hour) and grabs a copy of the Name Node’s in-memory metadata and files used to store metadata (both of which may be out of sync).  The Secondary Name Node combines this information in a fresh set of files and delivers them back to the Name Node, while keeping a copy for itself.

Should the Name Node die, the files retained by the Secondary Name Node can be used to recover the Name Node.  In a busy cluster, the administrator may configure the Secondary Name Node to provide this housekeeping service much more frequently than the default setting of one hour.  Maybe every minute.

Big Data to create a new boom in job market

  The 'Big Data' industry - the ability to access , analyze and use humungous 

volumes of data through    specific technology - will require a whole new army of 

 data workers globally. India itself will require a minimum of 1,00,000 data 

scientists in the next couple of years, in addition to scores of data managers and 

data analysts , to support the fast emerging Big Data space.


 The exponentially decreasing costs of data storage combined with the soaring 

volume of data being captured presents challenges and opportunities to those who 

work in the new frontiers of data science. Businesses, government agencies, and 

scientists leveraging data-based decisions are more successful than those relying 


on decades of trial-and-error. But taming and harnessing big data can be a 

herculean undertaking. The data must be collected, processed and distilled, 

analyzed, and presented in a manner humans can understand. Because there are no 

degrees in data science, data scientists must grow into their roles. If you are 

looking for resources to help you better understand big data and analytics, We have

 the knowledge and experience needed to help make your systems contribute to 

the success of your business. Form a tandem with us and take advantage of our 

capacity to manage, process and analyze big data effectively, quickly and 

economically.

BigDataTraining.IN has a strong focus and established thought leadership in the 

area of Big Data and Analytics. We use a global delivery model to help you to 

 evaluate and implement solutions tailored to your specific technical and business

 context.

Get Hands-on Training @ BigDataTraining.IN

BigDataTraining.IN - India's Leading BigData Consulting & Training Provider, Request a Quote!

Hadoop & Big Data Training | Development | Consulting | Projects

http://www.bigdatatraining.in/hadoop-training-chennai/

http://www.hadooptrainingchennai.in/hadoop-training-in-chennai/

email : info@bigdatatraining.in

Phone: +91 9789968765, 044-42645495

Contact us:
#67,2nd Floor, 1st Main Road, Gandhi Nagar, Adyar, Chennai- 600020

Name Node in HDFS - Hadoop Classroom / Online Training

Name Node in HDFS

The Name Node holds all the file system metadata for the cluster and oversees the health of Data Nodes and coordinates access to data.  The Name Node is the central controller of HDFS.  It does not hold any cluster data itself.  The Name Node only knows what blocks make up a file and where those blocks are located in the cluster.  The Name Node points Clients to the Data Nodes they need to talk to and keeps track of the cluster’s storage capacity, the health of each Data Node, and making sure each block of data is meeting the minimum defined replica policy.

Data Nodes send heartbeats to the Name Node every 3 seconds via a TCP handshake, using the same port number defined for the Name Node daemon, usually TCP 9000.  Every tenth heartbeat is a Block Report, where the Data Node tells the Name Node about all the blocks it has.  The block reports allow the Name Node build its metadata and insure (3) copies of the block exist on different nodes, in different racks.

The Name Node is a critical component of the Hadoop Distributed File System (HDFS).  Without it, Clients would not be able to write or read files from HDFS, and it would be impossible to schedule and execute Map Reduce jobs.  Because of this, it’s a good idea to equip the Name Node with a highly redundant enterprise class server configuration; dual power supplies, hot swappable fans, redundant NIC connections, etc.

Get Hands-on Training @ BigDataTraining.IN

BigDataTraining.IN - India's Leading BigData Consulting & Training Provider, Request a Quote!

Hadoop & Big Data Training | Development | Consulting | Projects

http://www.bigdatatraining.in/hadoop-training-chennai/

http://www.hadooptrainingchennai.in/hadoop-training-in-chennai/

http://www.bigdatatraining.in/

email : info@bigdatatraining.in

Phone: +91 9789968765, 044-42645495

Contact us:
#67,2nd Floor, 1st Main Road, Gandhi Nagar, Adyar, Chennai- 600020

HBase MapReduce Examples - Training

          HBase MapReduce Examples

        HBase MapReduce Read Example:

The following is an example of using HBase as a MapReduce source in read-only manner. Specifically, there is a Mapper instance but no Reducer, and nothing is being emitted from the Mapper. There job would be defined as follows...

Configuration config = HBaseConfiguration.create();
Job job = new Job(config, "ExampleRead");
job.setJarByClass(MyReadJob.class);     // class that contains mapper

Scan scan = new Scan();
scan.setCaching(500);        // 1 is the default in Scan, which 

                                              will be bad for MapReduce jobs
scan.setCacheBlocks(false);  // don't set to true for MR jobs
// set other scan attrs
...

TableMapReduceUtil.initTableMapperJob(
  tableName,        // input HBase table name
  scan,             // Scan instance to control CF and attribute selection
  MyMapper.class,   // mapper
  null,             // mapper output key
  null,             // mapper output value
  job);
job.setOutputFormatClass(NullOutputFormat.class);   // because we aren't

                                                 emitting anything from mapper

boolean b = job.waitForCompletion(true);
if (!b) {
  throw new IOException("error with job!");
}

 

public static class MyMapper extends TableMapper<Text, Text> {

  public void map(ImmutableBytesWritable row, Result value, Context context) 

                                 throws InterruptedException, IOException {
    // process data for the row from the Result instance.
   }
}
    

       HBase MapReduce Read/Write Example

The following is an example of using HBase both as a source and as a sink with MapReduce. This example will simply copy data from one table to another.

Configuration config = HBaseConfiguration.create();
Job job = new Job(config,"ExampleReadWrite");
job.setJarByClass(MyReadWriteJob.class);    // class that contains mapper

Scan scan = new Scan();
scan.setCaching(500);        // 1 is the default in Scan, which will be bad

                                              for MapReduce jobs
scan.setCacheBlocks(false);  // don't set to true for MR jobs
// set other scan attrs

TableMapReduceUtil.initTableMapperJob(
 sourceTable,      // input table
 scan,           // Scan instance to control CF and attribute selection
 MyMapper.class,   // mapper class
 null,           // mapper output key
 null,           // mapper output value
 job);
TableMapReduceUtil.initTableReducerJob(
 targetTable,      // output table
 null,             // reducer class
 job);
job.setNumReduceTasks(0);

boolean b = job.waitForCompletion(true);
if (!b) {
    throw new IOException("error with job!");
}
    

An explanation is required of what TableMapReduceUtil is doing, especially with the reducer. TableOutputFormat is being used as the outputFormat class, and several parameters are being set on the config (e.g., TableOutputFormat.OUTPUT_TABLE), as well as setting the reducer output key to ImmutableBytesWritable and reducer value to Writable. These could be set by the programmer on the job and conf, but TableMapReduceUtil tries to make things easier.

The following is the example mapper, which will create a Put and matching the input Result and emit it. Note: this is what the CopyTable utility does.

public static class MyMapper extends TableMapper<ImmutableBytesWritable, Put>  {

 public void map(ImmutableBytesWritable row, Result value, Context context) throws

IOException, InterruptedException {
  // this example is just copying the data from the source table...
     context.write(row, resultToPut(row,value));
    }

   private static Put resultToPut(ImmutableBytesWritable key, Result result)

throws IOException {
    Put put = new Put(key.get());
   for (KeyValue kv : result.raw()) {
   put.add(kv);
  }
  return put;
    }
}
    

There isn't actually a reducer step, so TableOutputFormat takes care of sending the Put to the target table.

Get Hands-on Training @ BigDataTraining.IN

BigDataTraining.IN - India's Leading BigData Consulting & Training Provider, Request a Quote!

Hadoop & Big Data Training | Development | Consulting | Projects

http://www.bigdatatraining.in/hadoop-training-chennai/

http://www.hadooptrainingchennai.in/hadoop-training-in-chennai/

http://www.bigdatatraining.in/

email : info@bigdatatraining.in

 Call:
+91 9789968765
044 - 42645495

Visit Us:
#67, 2nd Floor, Gandhi Nagar 1st Main Road, Adyar, Chennai - 20
[Opp to Adyar Lifestyle Super Market]