当前访客身份:游客 [ 登录 | 加入 OSCHINA ]

代码分享

当前位置:
代码分享 » Java  » 服务端开发
分享到: 
收藏 +0
2
使用Hadoop的HDFS API示例
标签: Hadoop

代码片段(2) [全屏查看所有代码]

1. [代码][Java]代码     跳至 [1] [全屏预览]

package cn.kissoft.hadoop.week04;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.IOUtils;

import java.io.*;
import java.net.URI;

/**
 * 2 自行在本地文件系统生成一个大约100字节的文本文件,写一段程序(可以利用Java API或C API),
 * 读入这个文件,并将其第101-120字节的内容写入HDFS成为一个新文件,提供代码和执行结果演示抓图
 */
public class Homework2 {

    /**
     *
     * @param as as[0] hdfs uri
     * @throws IOException
     */
    public static void main(String[] as) throws IOException {
        //1、创建文件
        File file = createFile("poem.txt");
        //2、读取101-12-bytes的数据
        byte[] bytes = read101to120Bytes(file);
        //3、上传到hdfs
        upload2Hdfs(bytes, as[0]);
    }

    /**
     * 创建文件
     * @param fileName
     * @return
     */
    private static File createFile(String fileName) {
        File file = new File(fileName);
        try {
            //打开一个写文件器,构造函数中的第二个参数true表示以追加形式写文件
            FileWriter writer = new FileWriter(file, true);
            writer.write(writelyrics());
            writer.close();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        return file;
    }

    /**
     * 读取101-120字节的数据
     * @param file
     * @return
     * @throws IOException
     */
    private static byte[] read101to120Bytes(File file) throws IOException {
        InputStream in = new BufferedInputStream(new FileInputStream(file));
        byte b[]=new byte[1024];
        in.skip(100);
        in.read(b,0,20);
        return b;
    }

    /**
     * 上传到hdfs
     * @param bytes 需要上传的数据
     * @param uri 上传的hdfs路径
     * @throws IOException
     */
    private static void upload2Hdfs(byte[] bytes, String uri) throws IOException {
        InputStream in = new ByteArrayInputStream(bytes);
        Configuration conf = new Configuration();
        FileSystem fs = FileSystem.get(URI.create(uri), conf);
        OutputStream out = fs.create(new Path(uri));
        IOUtils.copyBytes(in, out, 4096, true);
    }

    /**
     * 生成文件内容
     * @return
     */
    private static String writelyrics() {
        return "Ome, country roads albums.\n" +
                "I watch you sleeping little angel face\n" +
                "And on behalf of the human race\n" +
                "Welcome to this crazy place\n" +
                "I watch you sleeping innocent and free\n" +
                "I dont know what your dreams may be\n" +
                "You dont know what you mean to me\n" +
                "You have never heard thunder\n" +
                "You have never seen the rain\n" +
                "But you can still feel the wonder\n" +
                "You can still feel the pain\n" +
                "And sometimes you look at me\n" +
                "So wise and so sure\n" +
                "I could easily believe you have been here before\n" +
                "I watch you sleeping little angel face\n" +
                "And on behalf of the human race\n" +
                "Welcome to this crazy place\n" +
                "I wont be here forever\n" +
                "But as long as Im around\n" +
                "I promise you I will never let you down\n" +
                "To help make a world you can feel worthy of\n" +
                "I will teach you to fly on the wings of my love\n" +
                "I watch you sleeping brave and unaware\n" +
                "You dont know yet so you dont care\n" +
                "I want you to know its hard out there\n" +
                "I watch you sleeping little angel face\n" +
                "And on behalf of the human race\n" +
                "Welcome to this crazy place\n" +
                "I watch you sleeping\n" +
                "I watch you sleeping\n" +
                "I watch you sleeping\n" +
                "I watch you sleeping\n" +
                "I watch you sleeping\n" +
                "I watch you sleeping ";
    }
}

2. [图片] 1.png    



开源中国-程序员在线工具:Git代码托管 API文档大全(120+) JS在线编辑演示 二维码 更多»

发表评论 回到顶部 网友评论(1)

  • 1楼:青阳老焦 发表于 2015-07-08 11:57 回复此评论
    如果在写入过程中 突然断网、断电、程序强制退出,那么你正在写的那个HDFS文件就傻了,不能用了,要很久才能自动恢复。。。。。。
开源从代码分享开始 分享代码
悟空太多啦的其它代码 全部(3)...