10-Oracle大数据OLH连接

参考：

os:rhel-server-5.4-x86_64-dvd

jdk版本为:jdk-6u31-linux-i586-rpm.bin

Hadoop版本为：hadoop-0.20.2.tar.gz hadoop-0.20.2-CDH3B4.tar

oraloader版本为：oraloader-2.0.0

oracle的版本为：Version 11.2.0.1.0

JDBC:ojdbc6.jar

解压：oraloader

Oracle大数据OLH连接

有二个文件

10-Oracle大数据OLH连接

查看 README.TXT

10-Oracle大数据OLH连接

所以我们要用CHD 的版本hadoop 现有的版本：hadoop-0.20.2-CDH3B4.tar

解压oraloader

oraloader-2.0.0-1

修改环境变量

[[email protected] home]# vi /etc/profile

10-Oracle大数据OLH连接

配置 Oracle 相关驱动包

ojdbc6.jar 包在/hadoop/oraloader-2.0.0-1/jlib里面有一个

[[email protected] jlib]$ cp ojdbc6.jar /hadoop/hadoop-0.20.2/lib

启动hadoop集群

创建目标表

create table tab_lx(id number,name varchar2(50),address varchar2(100));

在Hadoop中放入示例文件

10-Oracle大数据OLH连接

[[email protected] bin]$ cat data.dat

1,zzg,湖南

2,zjy,北京

3,ldh,美国

拷贝国际化文件

cp /hadoop/oraloader-2.0.0-1/jlib/orai18n* /hadoop/hadoop-0.20.2/lib

10-Oracle大数据OLH连接

不然报错：

10-Oracle大数据OLH连接

配置 Oracle Loader for Hadoop

[[email protected] bin]$ vi MyConf.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

        <property>
                <name>mapred.input.dir</name>
                <value>olh_exercise1_in/data.dat</value>
        </property>
        <property>
                <name>mapreduce.inputformat.class</name>
                <value>oracle.hadoop.loader.lib.input.DelimitedTextInputFormat</value>
        </property>

        <property>
                <name>mapreduce.outputformat.class</name>
                <value>oracle.hadoop.loader.lib.output.JDBCOutputFormat</value>
        </property>




        <property>
                <name>mapred.output.dir</name>
                <value>olh_exercise1_out</value>
        </property>


        <property>
                <name>oracle.hadoop.loader.connection.url</name>
                <value>jdbc:oracle:thin:@192.168.80.70:1521/hadoop64</value>
        </property>

        <property>
                <name>oracle.hadoop.loader.connection.user</name>
                <value>SCOTT</value>
        </property>

        <property>
                <name>oracle.hadoop.loader.connection.password</name>
                <value>oracle</value>
        </property>

        <property>
                <name>oracle.hadoop.loader.loaderMapFile</name>
                <value>file:////hadoop/hadoop-0.20.2/bin/loaderMap_exercise1.xml</value>
        </property>



</configuration>

~

[[email protected] bin]$ vi loaderMap_exercise1.xml

<?xml version="1.0" encoding="UTF-8" ?>
<LOADER_MAP>
        <SCHEMA>SCOTT</SCHEMA>
        <TABLE>tab_lx</TABLE>
        <COLUMN field="F0">id</COLUMN>
        <COLUMN field="F1">name</COLUMN>
        <COLUMN field="F2">address</COLUMN>
</LOADER_MAP>

运行

[[email protected] bin]$ hadoop jar ${OLH_HOME}/jlib/oraloader.jar oracle.hadoop.loader.OraLoader -conf MyConf.xml

Oracle Loader for Hadoop Release 2.0.0 - Production

13/02/01 09:34:25 INFO loader.OraLoader: Oracle Loader for Hadoop Release 2.0.0 - Production

13/02/01 09:34:25 INFO loader.OraLoader: Built-Against: not available

13/02/01 09:34:28 INFO loader.OraLoader: oracle.hadoop.loader.loadByPartition is disabled because table: TAB_LX is not partitioned

13/02/01 09:34:28 INFO loader.OraLoader: oracle.hadoop.loader.enableSorting disabled, no sorting key provided

13/02/01 09:34:28 INFO output.DBOutputFormat: Setting reduce tasks speculative execution to false for : oracle.hadoop.loader.lib.output.JDBCOutputFormat

13/02/01 09:34:28 WARN loader.OraLoader: Sampler error: the number of reduce tasks must be greater than one; the configured value is 1 . Job will continue without sampled information.

13/02/01 09:34:28 INFO loader.OraLoader: Sampling time=0D:0h:0m:0s:14ms (14 ms)

13/02/01 09:34:28 INFO loader.OraLoader: Submitting OraLoader job OraLoader

13/02/01 09:34:29 INFO input.FileInputFormat: Total input paths to process : 1

13/02/01 09:34:30 INFO loader.OraLoader: map 0% reduce 0%

13/02/01 09:34:38 INFO loader.OraLoader: map 100% reduce 0%

13/02/01 09:34:50 INFO loader.OraLoader: map 100% reduce 100%

13/02/01 09:34:52 INFO loader.OraLoader: Job complete: OraLoader (null)

13/02/01 09:34:52 INFO loader.OraLoader: Counters: 17

FileSystemCounters

FILE_BYTES_READ=114

FILE_BYTES_WRITTEN=260

HDFS_BYTES_READ=39

HDFS_BYTES_WRITTEN=1858

Job Counters

Data-local map tasks=1

Launched map tasks=1

Launched reduce tasks=1

Map-Reduce Framework

Combine input records=0

Combine output records=0

Map input records=3

Map output bytes=102

Map output records=3

Reduce input groups=1

Reduce input records=3

Reduce output records=3

Reduce shuffle bytes=0

Spilled Records=6

测试ORACLE

10-Oracle大数据OLH连接