Windows下在Cygwin中Hadoop格式化以及运行Nutch命令

lizhonghai 发布于 2013/08/01 14:55
阅读 1K+
收藏 0

下面是Windows下在Cygwin中Hadoop格式化时的问题:

$ bin/hadoop namenode -format

Warning: $HADOOP_HOME is deprecated.


13/08/01 14:30:14 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ZHENZHEN/192.168.14.163
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.0.4-SNAPSHOT
STARTUP_MSG:   build =  -r ; compiled by 'Administrator' on Thu Aug  1 12:23:19                                                                                                                                2013
************************************************************/
Re-format filesystem in \tmp\hadoop-Administrator\dfs\name ? (Y or N) Y
13/08/01 14:30:17 INFO util.GSet: VM type       = 32-bit
13/08/01 14:30:17 INFO util.GSet: 2% max memory = 19.33375 MB
13/08/01 14:30:17 INFO util.GSet: capacity      = 2^22 = 4194304 entries
13/08/01 14:30:17 INFO util.GSet: recommended=4194304, actual=4194304
13/08/01 14:30:17 INFO namenode.FSNamesystem: fsOwner=Administrator
13/08/01 14:30:17 INFO namenode.FSNamesystem: supergroup=supergroup
13/08/01 14:30:17 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/08/01 14:30:17 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/08/01 14:30:17 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessK                                                                                                                           eyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/08/01 14:30:17 INFO namenode.NameNode: Caching file names occuring more than                                                                                                                            10 times
13/08/01 14:30:17 ERROR namenode.NameNode: java.io.IOException: Cannot remove cu                                                                                                                           rrent directory: \tmp\hadoop-Administrator\dfs\name\current
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDi                                                                                                                           rectory(Storage.java:295)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:13                                                                                                                           20)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:13                                                                                                                           39)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:                                                                                                                           1164)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNo                                                                                                                           de.java:1271)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:12                                                                                                                           88)


13/08/01 14:30:17 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ZHENZHEN/192.168.14.163

************************************************************/

下面是Windows下在Cygwin中Nutch执行命令时的问题:

$ bin/nutch crawl urls -dir crawl -depth 3 -topN 5
cygpath: can't convert empty path
13/07/31 11:55:55 WARN crawl.Crawl: solrUrl is not set, indexing will be skipped...
13/07/31 11:55:55 INFO crawl.Crawl: crawl started in: crawl
13/07/31 11:55:55 INFO crawl.Crawl: rootUrlDir = urls
13/07/31 11:55:55 INFO crawl.Crawl: threads = 10
13/07/31 11:55:55 INFO crawl.Crawl: depth = 3
13/07/31 11:55:55 INFO crawl.Crawl: solrUrl=null
13/07/31 11:55:55 INFO crawl.Crawl: topN = 5
13/07/31 11:55:55 INFO crawl.Injector: Injector: starting at 2013-07-31 11:55:55
13/07/31 11:55:55 INFO crawl.Injector: Injector: crawlDb: crawl/crawldb
13/07/31 11:55:55 INFO crawl.Injector: Injector: urlDir: urls
13/07/31 11:55:55 INFO crawl.Injector: Injector: Converting injected urls to crawl db entries.
13/07/31 11:55:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/07/31 11:55:55 ERROR security.UserGroupInformation: PriviledgedActionException as:Administrator cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-Administrator\mapred\staging\Administrator341668738\.staging to 0700
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-Administrator\mapred\staging\Administrator341668738\.staging to 0700
        at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
        at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
        at org.apache.nutch.crawl.Injector.inject(Injector.java:281)
        at org.apache.nutch.crawl.Crawl.run(Crawl.java:132)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)


加载中
0
十万情缘

为什么只是说明问题,没有说如何解决


0
十万情缘
官方的安装是在linux,我查到是hadoop的包在widow上就是不兼容,要改下,再编译生成


返回顶部
顶部