Month: February 2015

(Hadoop) Make Sure Your Datanode File System Have the Correct Permission!

Posted on Updated on

Kalau pernah ngalamin error seperti di bawah ini pas lagi mau jalanin MapReduce task, ini kemungkinan besar masalahnya ada pada file directory permission di salah satu datanode di-tempat MapReduce nya berjalan (via YARN).

Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/jars/hive-common-0.13.1-cdh5.3.0.jar!/hive-log4j.properties
Total jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapreduce.job.reduces=
Starting Job = job_1420709500935_0492, Tracking URL = http://**********:8088/proxy/application_1420709500935_0492/
Kill Command = /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/hadoop/bin/hadoop job  -kill job_1420709500935_0492
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1
2015-02-03 01:05:26,016 Stage-1 map = 0%,  reduce = 0%
2015-02-03 01:05:36,674 Stage-1 map = 50%,  reduce = 0%, Cumulative CPU 6.15 sec
2015-02-03 01:05:55,424 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 11.77 sec
2015-02-03 01:06:13,084 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 15.43 sec
MapReduce Total cumulative CPU time: 15 seconds 430 msec
Ended Job = job_1420709500935_0492
Launching Job 2 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapreduce.job.reduces=
Starting Job = job_1420709500935_0493, Tracking URL = http://**********:8088/proxy/application_1420709500935_0493/
Kill Command = /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/hadoop/bin/hadoop job  -kill job_1420709500935_0493
Hadoop job information for Stage-2: number of mappers: 0; number of reducers: 0
2015-02-03 01:06:29,383 Stage-2 map = 0%,  reduce = 0%
Ended Job = job_1420709500935_0493 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 2  Reduce: 1   Cumulative CPU: 15.43 sec   HDFS Read: 75107359 HDFS Write: 48814 SUCCESS
Stage-Stage-2:  HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 15 seconds 430 msec

Ketika kita melihat lebih jauh lagi error msg di:

 http://**********:8088/proxy/application_1420709500935_0493/
Application application_1420709500935_0493 failed 2 times due to AM Container for appattempt_1420709500935_0493_000002 exited with exitCode: -1000 due to: Not able to initialize distributed-cache directories in any of the configured local directories for user USERNAME
.Failing this attempt.. Failing the application.

Nah kalau gue, datanode root directory-nya di filesystem ada di:

/disk1/, /disk2/, /disk3/

Jadi.. supaya gue terbebas dari error msg diatas, gue harus make sure directories ini mempunyai permission yang cocok (yarn:yarn) (Karena MapReduce gue di manage oleh YARN, supaya default user nya bisa create file cache nya)

/disk1/yarn/nm/usercache
/disk2/yarn/nm/usercache
/disk3/yarn/nm/usercache

Cara nya gimana?

chown -R yarn:yarn /disk1/yarn/nm/usercache
chown -R yarn:yarn /disk2/yarn/nm/usercache
chown -R yarn:yarn /disk3/yarn/nm/usercache

Advertisements

Unusual Java Jenkins Problem (Play Framework)

Posted on Updated on

Sudah beberapa hari ini server Jenkins di kantor selalu komplain ‘build failed’ terus tanpa error yang jelas seperti ini:

------------------------------------------
[...truncated 85 lines...]

[info] Resolving joda-time#joda-time;2.1 ...
[info] Resolving org.joda#joda-convert;1.2 ...
[info] Resolving org.apache.commons#commons-lang3;3.1 ...
[info] Resolving com.ning#async-http-client;1.7.6 ...
[info] Resolving oauth.signpost#signpost-core;1.2.1.2 ...
[info] Resolving oauth.signpost#signpost-commonshttp4;1.2.1.2 ...
[info] Resolving org.apache.httpcomponents#httpcore;4.0.1 ...
[info] Resolving org.apache.httpcomponents#httpclient;4.0.1 ...
[info] Resolving commons-logging#commons-logging;1.1.1 ...
[info] Resolving org.codehaus.jackson#jackson-core-asl;1.9.10 ...
[info] Resolving org.codehaus.jackson#jackson-mapper-asl;1.9.10 ...
[info] Resolving net.sf.ehcache#ehcache-core;2.6.0 ...
[info] Resolving javax.transaction#jta;1.1 ...
[info] Resolving org.scala-lang#scala-reflect;2.10.0 ...
[info] Resolving play#play-java_2.10;2.1.3 ...
[info] Resolving org.hibernate#hibernate-validator;4.3.0.Final ...
[info] Resolving javax.validation#validation-api;1.0.0.GA ...
[info] Resolving org.jboss.logging#jboss-logging;3.1.0.CR2 ...
[info] Resolving org.springframework#spring-core;3.1.2.RELEASE ...
[info] Resolving org.springframework#spring-beans;3.1.2.RELEASE ...
[info] Resolving org.reflections#reflections;0.9.8 ...
[info] Resolving com.google.guava#guava;13.0.1 ...
[info] Resolving com.google.code.findbugs#jsr305;2.0.1 ...
[info] Resolving javax.servlet#javax.servlet-api;3.0.1 ...
[info] Resolving play#play-java-jdbc_2.10;2.1.3 ...
[info] Resolving play#play-jdbc_2.10;2.1.3 ...
[info] Resolving com.jolbox#bonecp;0.7.1.RELEASE ...
[info] Resolving com.h2database#h2;1.3.168 ...
[info] Resolving tyrex#tyrex;1.0.1 ...
[info] Resolving play#play-java-ebean_2.10;2.1.3 ...
[info] Resolving org.hibernate.javax.persistence#hibernate-jpa-2.0-api;1.0.1.Final ...
[info] Resolving mysql#mysql-connector-java;5.1.18 ...
[info] Resolving org.springframework#spring-context;3.2.4.RELEASE ...
[info] Resolving org.springframework#spring-aop;3.2.4.RELEASE ...
[info] Resolving aopalliance#aopalliance;1.0 ...
[info] Resolving org.springframework#spring-beans;3.2.4.RELEASE ...
[info] Resolving org.springframework#spring-core;3.2.4.RELEASE ...
[info] Resolving org.springframework#spring-expression;3.2.4.RELEASE ...
[info] Resolving com.mfizz#mfz-ruby-marshal;1.0.2 ...
[info] Resolving org.mockito#mockito-all;1.8.4 ...
[info] Resolving org.powermock#powermock-module-junit4;1.5.1 ...
[info] Resolving junit#junit;4.11 ...
[info] Resolving org.hamcrest#hamcrest-core;1.3 ...
[info] Resolving org.powermock#powermock-module-junit4-common;1.5.1 ...
[info] Resolving org.powermock#powermock-core;1.5.1 ...
[info] Resolving org.powermock#powermock-reflect;1.5.1 ...
[info] Resolving org.objenesis#objenesis;1.2 ...
[info] Resolving org.javassist#javassist;3.18.0-GA ...
[info] Resolving org.powermock#powermock-api-mockito;1.5.1 ...
[info] Resolving org.mockito#mockito-all;1.9.5 ...
[info] Resolving org.powermock#powermock-api-support;1.5.1 ...
[info] Resolving com.fasterxml.jackson.core#jackson-core;2.2.3 ...
[info] Resolving org.simpleframework#simple-xml;2.7.1 ...
[info] Resolving stax#stax-api;1.0.1 ...
[info] Resolving stax#stax;1.2.0 ...
[info] Resolving xpp3#xpp3;1.1.3.3 ...
[info] Resolving net.greghaines#jesque;1.3.0 ...
[info] Resolving redis.clients#jedis;2.1.0 ...
[info] Resolving commons-pool#commons-pool;1.6 ...
[info] Resolving com.fasterxml.jackson.core#jackson-databind;2.1.2 ...
[info] Resolving com.fasterxml.jackson.core#jackson-annotations;2.1.1 ...
[info] Resolving commons-codec#commons-codec;1.8 ...
[info] Resolving org.avaje.ebeanorm#avaje-ebeanorm;3.2.4 ...
[info] Resolving javax.persistence#persistence-api;1.0 ...
[info] Resolving com.github.mumoshu#play2-memcached_2.10;0.3.0.3 ...
[info] Resolving net.spy#spymemcached;2.9.0 ...
[info] Resolving com.maxmind.geoip#geoip-api;1.2.11 ...
[info] Resolving org.yaml#snakeyaml;1.13 ...
[info] Resolving org.apache.commons#commons-collections4;4.0 ...
[info] Resolving org.fluentlenium#fluentlenium-parent;0.9.1 ...
[info] Resolving com.datastax.cassandra#cassandra-driver-core;2.1.4 ...
[info] Resolving com.google.guava#guava;14.0.1 ...
[info] Resolving com.codahale.metrics#metrics-core;3.0.2 ...
[info] Resolving org.slf4j#slf4j-api;1.7.5 ...
[info] Resolving com.datastax.cassandra#cassandra-driver-mapping;2.1.4 ...
[info] Resolving commons-io#commons-io;2.4 ...
[info] Resolving play#play-test_2.10;2.1.3 ...
[info] Resolving junit#junit-dep;4.10 ...
[info] Resolving org.specs2#specs2_2.10;1.13 ...
[info] Resolving org.specs2#scalaz-core_2.10;7.0.0 ...
[info] Resolving org.specs2#scalaz-concurrent_2.10;7.0.0 ...
[info] Resolving org.specs2#scalaz-effect_2.10;7.0.0 ...
[info] Resolving com.novocode#junit-interface;0.10 ...
[info] Resolving org.scala-tools.testing#test-interface;0.5 ...
[info] Resolving org.fluentlenium#fluentlenium-festassert;0.7.3 ...
[info] Resolving org.fluentlenium#fluentlenium-core;0.7.3 ...
[info] Resolving org.seleniumhq.selenium#selenium-java;2.25.0 ...
[info] Resolving org.seleniumhq.selenium#selenium-android-driver;2.25.0 ...
[info] Resolving org.seleniumhq.selenium#selenium-remote-driver;2.25.0 ...
[info] Resolving cglib#cglib-nodep;2.1_3 ...
[info] Resolving org.json#json;20080701 ...
[info] Resolving org.seleniumhq.selenium#selenium-api;2.25.0 ...
[info] Resolving org.apache.httpcomponents#httpclient;4.1.2 ...
[info] Resolving org.apache.httpcomponents#httpcore;4.1.3 ...
[info] Resolving org.apache.commons#commons-exec;1.1 ...
[info] Resolving net.java.dev.jna#jna;3.4.0 ...
[info] Resolving net.java.dev.jna#platform;3.4.0 ...
[info] Resolving org.seleniumhq.selenium#selenium-chrome-driver;2.25.0 ...
[info] Resolving org.seleniumhq.selenium#selenium-htmlunit-driver;2.25.0 ...
[info] Resolving net.sourceforge.htmlunit#htmlunit;2.9 ...
[info] Resolving xalan#xalan;2.7.1 ...
[info] Resolving xalan#serializer;2.7.1 ...
[info] Resolving xml-apis#xml-apis;1.3.04 ...
[info] Resolving commons-collections#commons-collections;3.2.1 ...
[info] Resolving commons-lang#commons-lang;2.6 ...
[info] Resolving org.apache.httpcomponents#httpmime;4.1.2 ...
[info] Resolving net.sourceforge.htmlunit#htmlunit-core-js;2.9 ...
[info] Resolving xerces#xercesImpl;2.9.1 ...
[info] Resolving net.sourceforge.nekohtml#nekohtml;1.9.15 ...
[info] Resolving net.sourceforge.cssparser#cssparser;0.9.5 ...
[info] Resolving org.w3c.css#sac;1.3 ...
[info] Resolving org.seleniumhq.selenium#selenium-firefox-driver;2.25.0 ...
[info] Resolving org.seleniumhq.selenium#selenium-ie-driver;2.25.0 ...
[info] Resolving org.seleniumhq.selenium#selenium-iphone-driver;2.25.0 ...
[info] Resolving org.seleniumhq.selenium#selenium-safari-driver;2.25.0 ...
[info] Resolving org.webbitserver#webbit;0.4.6 ...
[info] Resolving org.seleniumhq.selenium#selenium-support;2.25.0 ...
[info] Resolving org.easytesting#fest-assert;1.4 ...
[info] Resolving org.easytesting#fest-util;1.1.6 ...

 [0m[ [0minfo [0m]  [0mDone updating. [0m
 [0m[ [0minfo [0m]  [0mCompiling 12 Scala sources and 297 Java sources to <http://**********:8080/job/AdsServing%20Engine/ws/target/scala-2.10/classes... [0m>
 [0m[ [31merror [0m]  [0m<http://**********:8080/job/AdsServing%20Engine/ws/app/models/rtb/twoone/Banner.java>:1: package rtb contains object and package with same name: v21 [0m
 [0m[ [31merror [0m]  [0mone of them needs to be removed from classpath [0m
 [0m[ [31merror [0m]  [0mpackage models.rtb.twoone; [0m
 [0m[ [31merror [0m]  [0m                   ^ [0m
 [0m[ [31merror [0m]  [0mone error found [0m
 [0m[ [31merror [0m]  [0m(compile: [31mcompile [0m) Compilation failed [0m
 [0m[ [31merror [0m]  [0mTotal time: 10 s, completed Feb 3, 2015 2:22:48 PM [0m
Build step 'Execute shell' marked build as failure

Setelah beberapa jam dikutak-katik ternyata solusi-nya cuma hanya perlu re-download dan re-compile play framework nya (kita pake java play framework) :/ Kemungkinan besar ini dikarenakan 1-2minggu yang lalu server jenkins hdd-nya sempat corrupted lalu di repair (dengan software) tapi ada beberapa broken files/directories structure-nya.