hadoop

Cloudera Enterprise VS Cloudera Express

Posted on Updated on

Cloudera Version Comparison Express Version Enterprise Version
Cluster Management
Number of hosts supported Unlimited Unlimited
Host inspector for determining CDH readiness Yes Yes
Multi-cluster management Yes Yes
Centralized view of all running commands Yes Yes
Resource management Yes Yes
Global time control for historical diagnosis Yes Yes
Cluster-wide configuration Yes Yes
Cluster-wide event management Yes Yes
Cluster-wide log search Yes Yes
Aggregate UI No Yes
Deployment
Support for CDH 4 and CDH 5 Yes Yes
Automated deployment and readiness checks Yes Yes
Installation from local repositories Yes Yes
Rolling upgrade of CDH No Yes
Service and Configuration Management
Manage Accumulo, Flume, HBase, HDFS, Hive, Hue, Impala, Isilon, Kafka, Kudu, MapReduce, Oozie, Sentry, Solr, Spark, Sqoop, YARN, and ZooKeeper services Yes Yes
Manage Key Trustee and Cloudera Navigator No Yes
Manage add-on services Yes Yes
Rolling restart of services No Yes
High availability (HA) support Yes Yes
CDH 4 – HDFS and MapReduce JobTracker (CDH 4.2) Yes Yes
CDH 5 – HDFS, Hive Metastore, Hue, Impala Llama ApplicationMaster, MapReduce JobTracker, Oozie, YARN ResourceManager Yes Yes
HBase co-processor support Yes Yes
Configuration audit trails Yes Yes
Client configuration management Yes Yes
Workflows (add, start, stop, restart, delete, and decommission services, hosts, and role instances) Yes Yes
Role groups Yes Yes
Host templates Yes Yes
Configuration versioning and history No Yes
Restoring a configuration using the API No Yes
Security
Kerberos authentication Yes Yes
LDAP authentication for CDH Yes Yes
LDAP authentication for Cloudera Manager No Yes
SAML authentication No Yes
Encrypted communication between Server and host Agents (TLS) Yes Yes
Sentry role-based access control Yes Yes
Password redaction Yes Yes
Data encryption with KMS No Yes
Cloudera Manager user roles No Yes
Monitoring and Diagnostics
Service, host, and activity monitoring Yes Yes
Proactive health tests Yes Yes
Health history Yes Yes
Advanced filtering and charting of metrics Yes Yes
Job monitoring for MapReduce jobs, YARN applications, and Impala queries Yes Yes
Similar activity performance for MapReduce jobs Yes Yes
Support for terminating activities Yes Yes
Alert Management
Alert by email Yes Yes
Alert by SNMP Yes Yes
User-defined triggers Yes Yes
Custom alert publish scripts Yes Yes
Advanced Management Features
Automated backup and disaster recovery No Yes
File browsing, searching, and disk quota management No Yes
HBase, MapReduce, Impala, and YARN usage reports No Yes
Support integration No Yes
Operational reports No Yes
Cloudera Navigator Data Management
Metadata management and augmentation No Yes
Ingest policies No Yes
Analytics No Yes
Auditing No Yes
Lineage No Yes
Advertisements

HDFS Balancer and HBase Data Locality

Posted on Updated on

HBase blocks file locality and HDFS Balancer could post some problems ūüė¶

There are 3 facts that I’ve learned from this stackoverflow post.

  1. The Hadoop (HDFS) balancer moves blocks around from one node to another to try to make it so each datanode has the same amount of data (within a configurable threshold). This messes up HBases’s data locality, meaning that a particular region may be serving a file that is no longer on it’s local host.
  2. HBase’s balance_switch balances the cluster so that each regionserver hosts the same number of regions (or close to). This is separate from Hadoop’s (HDFS) balancer.
  3. If you are running only HBase, I recommend not running Hadoop’s (HDFS) balancer as it will cause certain regions to lose their data locality. This causes any request to that region to have to go over the network to one of the datanodes that is serving it’s HFile.

HBase’s data locality is recovered though. Whenever compaction occurs, all the blocks are copied locally to the regionserver serving that region and merged. At that point, data locality is recovered for that region. With that, all you really need to do to add new nodes to the cluster is add them. Hbase will take care of rebalancing the regions, and once these regions compact data locality will be restored.

Don’t Upgrade Your Cloudera Manager When HDFS Rebalancer is Active

Posted on Updated on

This is just simple post and literally means the same thing as the title.

For Cloudera Hadoop Users:

Don’t Upgrade Your Cloudera Manager When HDFS Rebalancer is Active

If you accidently done it,¬†then the Cloudera Manager upgrade process will fail when it’s trying to¬†start the new version.
You need to revert it back to the previous working version.

How to connect PredictionIO 0.9.3 with Cloudera CDH 5.4.x HBase

Posted on Updated on

PredictionIO Image

  1. For some reason when I’m trying to use the Spark from CDH it doesn’t work with PredictionIO 0.9.3,
    So I use spark 1.3.1 binary with hadoop 2.6 support and I extracted mine to: SPARK_HOME=$PIO_HOME/vendors/spark-1.3.1-bin-hadoop2.6
  2. From CDH part I only use the HBase part as the event server storage.
  3. I use Elasticsearch as metadata storage.
  4. I use LocalFS as model storage.
  5. I installed spark standalone server manually (not from cdh) (spark 1.3.1 with hadoop 2.6 support)
    – For this test case I’m using a spark master with 4 workers node and let say I installed at spark://my.remote.sparkhost:7077
    – If you don’t know how to install a stand alone spark server, please read the spark manual.

My config file as below:

# pio-env.sh
#################################################################################################

#!/usr/bin/env bash
# spark 1.3.1-bin-hadoop2.6 binary downloaded from spark website
 SPARK_HOME=$PIO_HOME/vendors/spark-1.3.1-bin-hadoop2.6
# cloudera hadoop (CDH 5.4.x)
 HADOOP_CONF_DIR=/etc/hadoop/conf
# cloudera hbase (CDH 5.4.x)
 HBASE_CONF_DIR=/etc/hbase/conf
#######
PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
 PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
 PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
 PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
#######
# Elasticsearch
 PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
 PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
 PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
 PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
# LocalFS
 PIO_FS_BASEDIR=$HOME/.pio_store
 PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
 PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
 PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
 PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
# HBase CDH 5.4.x
 PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
 PIO_STORAGE_SOURCES_HBASE_HOME=/opt/cloudera/parcels/CDH-5.4.1-1.cdh5.4.1.p0.6/lib/hbase

#################################################################################################

pio@nn01:~$ pio status
[INFO] [Console$] Inspecting PredictionIO…
[INFO] [Console$] PredictionIO 0.9.3 is installed at /pio/PredictionIO-0.9.3
[INFO] [Console$] Inspecting Apache Spark…
[INFO] [Console$] Apache Spark is installed at /pio/PredictionIO-0.9.3/vendors/spark-1.3.1-bin-hadoop2.6
[INFO] [Console$] Apache Spark 1.3.1 detected (meets minimum requirement of 1.3.0)
[INFO] [Console$] Inspecting storage backend connections…
[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)…
[INFO] [Storage$] Verifying Model Data Backend (Source: LOCALFS)…
[INFO] [Storage$] Verifying Event Data Backend (Source: HBASE)…
[INFO] [Storage$] Test writing to Event Store (App Id 0)…
[INFO] [HBLEvents] The table pio_event:events_0 doesn’t exist yet. Creating now…
[INFO] [HBLEvents] Removing table pio_event:events_0…
[INFO] [Console$] (sleeping 5 seconds for all messages to show up…)
[INFO] [Console$] Your system is all ready to go.

so after you do “pio “build”, to make PredictionIO utilize remote spark server that I created before on point #5, I use this parameter when I want to “train and deploy”

$ pio train — –master spark://my.remote.sparkhost:7077 –driver-memory 4G –executor-memory 1G

$ pio deploy — –master spark://my.remote.sparkhost:7077 –driver-memory 4G –executor-memory 1G

if you have a large dataset, you might want to add:¬†–conf spark.akka.frameSize=1024 in “pio train”

So by using this setup, PredictionIO will utilize remote spark server and write event data to a Cloudera HBase Cluster.

This post is to answer Yanbo question at predictionio user group on how to integrate predictionIO with cloudera hadoop.

The Simplest Way to Generate CSV Output From Hive in Linux Shell

Posted on Updated on

If you are wondering the easiest way (at least IMHO) how to generate data output in HIVE in Excel-like CSV compatible format without modifying any table or using 3rd party java plugin is:

hive -e “SELECT col1, col2, … FROM table_name”¬†| perl -lpe ‘s//\\”/g; s/^|$//g; s/\t/,/g’ > output_file.csv

I know you can also use awk or some other shell commands, but perl regex is very POWERFUL and FAST.
I got this perl regex tips some time ago from stackoverflow link (i will put the link once i remember) and this method worked for me to convert the standard Tab separated output into CSV compatible ūüėČ