Eclipse怎么远程执行MapReduce程序

本篇内容介绍了"Eclipse怎么远程执行MapReduce程序"的有关知识，在实际案例的操作过程中，不少人都会遇到这样的困境，接下来就让小编带领大家学习一下如何处理这些情况吧！希望大家仔细阅读，能够学有所成！

1、环境说明

Win7 64位

Hadoop-1.2.1 Virtualbox三台主机bd11，bd12，bd13

Eclipse Indigo 必须用3系，插件放到4.x中会报错，是META-INF/MANIFEST.MF中classpath的问题，会找不到类

自编译hadoop-1.2.1的插件

2、插件参数配置

2.1、Eclipse配置

配置本地hadoop路径

2.2、插件配置-General

2.3、Advanced parameters

2.3、可能报的错

当以上配置没有配好时，可能会报一下错误

3、运行

3.1、运行参数

是的，你可以把参数写死。

3.2、运行过程

3.3、运行结果文件

4、可能出现的错误

4.1、打开/刷新目录/创建目录没反应，也不报错

可能是引用的jar包或者以上配置的参数不对

4.2、运行mapreduce程序，抛出serverIPC版本和client不同的异常

Server IPC version 9 cannot communicate with client version 4

这种情况是由于引用的hadoop相关jar的版本与服务器上不一致导致的。

4.3、权限问题1

问题如下

14/08/21 21:39:29 ERROR security.UserGroupInformation: PriviledgedActionException as:zhaiph cause:java.io.IOException: Failed to set permissions of path: tmphadoop-xxx.staging to 0700Exception in thread "main" java.io.IOException: Failed to set permissions of path: tmphadoop-xxx.staging to 0700        at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)        at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)        at java.security.AccessController.doPrivileged(Native Method)        at javax.security.auth.Subject.doAs(Subject.java:415)        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)        at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)        at MaxTemperature.main(MaxTemperature.java:36)

解决办法

hadoop-core-1.2.1.jar的问题。方法有二，要么重新编译jar，要么改源文org.apache.hadoop.fs.FileUtil.java。改源文件内容如下

private static void checkReturnValue(boolean rv， File p，FsPermission permission)                                     throws IOException {//    if (!rv) {//      throw new IOException("Failed to set permissions of path: " + p + //                            " to " + //                            String.format("o", permission.toShort()));//    }  }

4.4、权限问题2

错误描述

14/08/21 21:43:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable14/08/21 21:43:12 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.14/08/21 21:43:12 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).14/08/21 21:43:12 INFO input.FileInputFormat: Total input paths to process : 114/08/21 21:43:12 WARN snappy.LoadSnappy: Snappy native library not loaded14/08/21 21:43:13 INFO mapred.JobClient: Running job: job_local1395395251_000114/08/21 21:43:13 WARN mapred.LocalJobRunner: job_local1395395251_0001org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=Johnson, access=WRITE, inode="ch02":wukong:supergroup:rwxr-xr-x    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)    at java.lang.reflect.Constructor.newInstance(Unknown Source)    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)    at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1459)    at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:362)    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1161)    at org.apache.hadoop.mapred.FileOutputCommitter.setupJob(FileOutputCommitter.java:52)    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:319)Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Permission denied: user=Johnson, access=WRITE, inode="ch02":wukong:supergroup:rwxr-xr-x    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:217)    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:197)    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:141)    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5758)    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5731)    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:2502)    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2469)    at org.apache.hadoop.hdfs.server.namenode.NameNode.mkdirs(NameNode.java:911)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke(Method.java:606)    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)    at java.security.AccessController.doPrivileged(Native Method)    at javax.security.auth.Subject.doAs(Subject.java:415)    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)     at org.apache.hadoop.ipc.Client.call(Client.java:1113)    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)    at com.sun.proxy.$Proxy1.mkdirs(Unknown Source)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)    at java.lang.reflect.Method.invoke(Unknown Source)    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)    at com.sun.proxy.$Proxy1.mkdirs(Unknown Source)    at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1457)    ... 4 more14/08/21 21:43:14 INFO mapred.JobClient:  map 0% reduce 0/08/21 21:43:14 INFO mapred.JobClient: Job complete: job_local1395395251_000114/08/21 21:43:14 INFO mapred.JobClient: Counters: 0

或者是直接在eclipse的DFS目录树中某个节点提示permission ^&^((%^&%%R*^*

解决办法：在hdfs-site.xml中加入以下部分。使得dfs不校验权限。

        dfs.permissions    false

4.5、Win7中执行报winutils.exe找不到

错误如下：

2015-02-05 15:19:43,080 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable2015-02-05 15:19:44,257 ERROR [main] util.Shell (Shell.java:getWinUtilsPath(336)) - Failed to locate the winutils binary in the hadoop binary pathjava.io.IOException: Could not locate executable D:\Lab\lib\hadoop\hadoop-2.4.1\bin\winutils.exe in the Hadoop binaries.        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)        at org.apache.hadoop.util.Shell.(Shell.java:326)        at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)        at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)        at org.apache.hadoop.security.Groups.(Groups.java:77)        at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232)        at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718)        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703)        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605)        at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2554)        at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2546)        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2412)        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)        at wukong.hadoop.tools.HdfsTool.rm(HdfsTool.java:26)        at sample.FlowMR.main(FlowMR.java:21)2015-02-05 15:19:51,141 INFO  [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1009)) - session.id is deprecated. Instead, use dfs.metrics.session-id2015-02-05 15:19:51,154 INFO  [main] jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=

"Eclipse怎么远程执行MapReduce程序"的内容就介绍到这里了，感谢大家的阅读。如果想了解更多行业相关的知识可以关注网站，小编将为大家输出更多高质量的实用文章！