BlueXIII's Blog

热爱技术,持续学习

基础准备

  • JDK
  • Hadoop
  • SSH Server

下载HBase

https://hbase.apache.org/

1
2
tar -zxvf hbase-1.4.2-bin.tar.gz
sudo mv hbase-1.4.2 /opt/hbase

系统环境变量

export HBASE_MANAGES_ZK=true
export HBASE_HOME=/opt/hbase
export PATH=$PATH:/opt/hbase/bin

写死JAVA_HOME

vi ./conf/hbase-env.sh

1
export JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre/

单机模式配置

vi hbase-site.xml

1
2
3
4
5
6
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///tmp/hbase-${user.name}/hbase</value>
</property>
</configuration>

伪分布模式配置

vi hbase-site.xml

1
2
3
4
5
6
7
8
9
10
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
</configuration>

启动Hadoop:

1
./start-all.sh

Hbase启停

启停:

1
2
./bin/start-hbase.sh
./bin/stop-hbase.sh

查看进程:

1
jps

WEB控制台:
http://localhost:45005/master-status

HBase Shell

1
2
./bin/hbase shell
help

常用命令:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
status
create 'test', 'cf'
list 'test'
describe 'test'

disable 'test'
drop 'test'
exists 'test'

put 'test', 'row1', 'cf:a', 'value1'
put 'test', 'row2', 'cf:b', 'value2'
put 'test', 'row3', 'cf:c', 'value3'

scan 'test'
get 'test', 'row1'

参考文档

https://www.jianshu.com/p/8818e8d40da4
https://www.jianshu.com/p/352213f109e5
https://www.shiyanlou.com/courses/37
https://www.ibm.com/developerworks/cn/analytics/library/ba-cn-bigdata-hbase/index.html

下载

http://www.oracle.com/technetwork/database/database-technologies/express-edition/downloads/index.html

Download
Oracle Database Express Edition 11g Release 2 for Linux x64

Copy the downloaded file and paste it in home directory.

Unzip using the command:

unzip oracle-xe-11.2.0-1.0.x86_64.rpm.zip
Install required packages using the command:

sudo apt-get install alien libaio1 unixodbc
Enter into the Disk1 folder using command:

cd Disk1/
Convert RPM package format to DEB package format (that is used by Ubuntu) using the command:

sudo alien –scripts -d oracle-xe-11.2.0-1.0.x86_64.rpm
Create the required chkconfig script using the command:

sudo pico /sbin/chkconfig
The pico text editor is started and the commands are shown at the bottom of the screen. Now copy and paste the following into the file and save:

#!/bin/bash

Oracle 11gR2 XE installer chkconfig hack for Ubuntu

file=/etc/init.d/oracle-xe
if [[ ! tail -n1 $file | grep INIT ]]; then
echo >> $file
echo ‘### BEGIN INIT INFO’ >> $file
echo ‘# Provides: OracleXE’ >> $file
echo ‘# Required-Start: $remote_fs $syslog’ >> $file
echo ‘# Required-Stop: $remote_fs $syslog’ >> $file
echo ‘# Default-Start: 2 3 4 5’ >> $file
echo ‘# Default-Stop: 0 1 6’ >> $file
echo ‘# Short-Description: Oracle 11g Express Edition’ >> $file
echo ‘### END INIT INFO’ >> $file
fi
update-rc.d oracle-xe defaults 80 01
Change the permission of the chkconfig file using the command:

sudo chmod 755 /sbin/chkconfig
Set kernel parameters. Oracle 11gR2 XE requires additional kernel parameters which you need to set using the command:

sudo pico /etc/sysctl.d/60-oracle.conf
Copy the following into the file and save:

Oracle 11g XE kernel parameters

fs.file-max=6815744
net.ipv4.ip_local_port_range=9000 65000
kernel.sem=250 32000 100 128
kernel.shmmax=536870912
Verify the change using the command:

sudo cat /etc/sysctl.d/60-oracle.conf
You should see what you entered earlier. Now load the kernel parameters:

sudo service procps start
Verify the new parameters are loaded using:

sudo sysctl -q fs.file-max
You should see the file-max value that you entered earlier.

Set up /dev/shm mount point for Oracle. Create the following file using the command:

sudo pico /etc/rc2.d/S01shm_load
Copy the following into the file and save.

#!/bin/sh
case “$1” in
start)
mkdir /var/lock/subsys 2>/dev/null
touch /var/lock/subsys/listener
rm /dev/shm 2>/dev/null
mkdir /dev/shm 2>/dev/null
*)
echo error
exit 1
;;

esac
Change the permissions of the file using the command:

sudo chmod 755 /etc/rc2.d/S01shm_load
Now execute the following commands:

sudo ln -s /usr/bin/awk /bin/awk
sudo mkdir /var/lock/subsys
sudo touch /var/lock/subsys/listener
Now, Reboot Your System

Step 3: Install Oracle
Install the oracle DBMS using the command:

sudo dpkg –install oracle-xe_11.2.0-2_amd64.deb
Configure Oracle using the command:

sudo /etc/init.d/oracle-xe configure
Setup environment variables by editting your .bashrc file:

pico ~/.bashrc
Add the following lines to the end of the file:

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
export ORACLE_SID=XE
export NLS_LANG=$ORACLE_HOME/bin/nls_lang.sh
export ORACLE_BASE=/u01/app/oracle
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
export PATH=$ORACLE_HOME/bin:$PATH
Load the changes by executing your profile:

. ~/.bashrc
Start the Oracle 11gR2 XE:

sudo service oracle-xe start
Add user YOURUSERNAME to group dba using the command:

sudo usermod -a -G dba YOURUSERNAME
Step 4: Using the Oracle XE Command Shell
Start the Oracle XE 11gR2 server using the command:

sudo service oracle-xe start
Start command line shell as the system admin using the command:

sqlplus sys as sysdba
Enter the password that you gave while configuring Oracle earlier. You will now be placed in a SQL environment that only understands SQL commands.

Create a regular user account in Oracle using the SQL command:

create user USERNAME identified by PASSWORD;
Replace USERNAME and PASSWORD with the username and password of your choice. Please remember this username and password. If you had error executing the above with a message about resetlogs, then execute the following SQL command and try again:

alter database open resetlogs;
Grant privileges to the user account using the SQL command:

grant connect, resource to USERNAME;
Replace USERNAME and PASSWORD with the username and password of your choice. Please remember this username and password.

Exit the sys admin shell using the SQL command:

exit;
Start the commandline shell as a regular user using the command:

sqlplus
Now, you can run sql commands…

Installing Oracle XE by hand
Installing Oracle XE with Vagrant and Puppet
Installing Oracle XE with Docker

docker方式

https://tuhrig.de/3-ways-of-installing-oracle-xe-11g-on-ubuntu/
https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1
https://hub.docker.com/r/alexeiled/docker-oracle-xe-11g/

如果涉及macOS与Windows交换文件的话,在Windows会有一些无用的.DS_Store文件需要清理。

可以在gitbash/cygwin/cmder下执行如下命令清理:

1
find . -name '*.DS_Store' -type f -delete

最优配置

为了最大限度兼容macOS以及Linux,需要:

  1. 提交时转换为LF,检出时不转换
  2. 拒绝提交包含混合换行符的文件
1
2
git config --global core.autocrlf input
git config --global core.safecrlf true

批量将CRLF转换成LF

如果在Windows下不慎将部分文本的换行符写为CRLF,可以使用dos2unix工具修复回LF

在Cmder或Cywin下执行:

1
find . -type f|xargs dos2unix

IDE配置

注意及时将VS Code、IntelliJ IDEA等编辑器默认配置修改,将换行符置为LF!

附:autocrlf参数说明

1
2
3
4
5
6
7
8
# 提交时转换为LF,检出时转换为CRLF
git config --global core.autocrlf true

# 提交时转换为LF,检出时不转换
git config --global core.autocrlf input

# 提交检出均不转换
git config --global core.autocrlf false

附:safecrlf参数说明

1
2
3
4
5
6
7
8
# 拒绝提交包含混合换行符的文件
git config --global core.safecrlf true

# 允许提交包含混合换行符的文件
git config --global core.safecrlf false

# 提交包含混合换行符的文件时给出警告
git config --global core.safecrlf warn

安装

1
brew install kafka

如果缺少JDK8依赖,或安装了JDK10,则需要先通过brew cask插件安装JDK8:

1
brew cask install java8

安装位置

1
2
/usr/local/Cellar/zookeeper
/usr/local/Cellar/kafka

配置文件位置

1
2
/usr/local/etc/kafka/server.properties
/usr/local/etc/kafka/zookeeper.properties

启动

直接启动:

1
2
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties &
kafka-server-start /usr/local/etc/kafka/server.properties &

或通过brew services插件启动

1
2
brew services start zookeeper
brew services start kafka

创建topic

1
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

查看创建的topic

1
kafka-topics.sh --list --zookeeper localhost:2181

发送一些消息

1
kafka-console-producer.sh --broker-list localhost:9092 --topic test 

消费消息

1
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

附:brew services说明

https://github.com/Homebrew/homebrew-services

附:brew cask说明

https://caskroom.github.io

GitHub地址

https://github.com/wnameless/docker-oracle-xe-11g

安装

Ubuntu 16.04环境:

1
docker pull wnameless/oracle-xe-11g:16.04

Ubuntu 18.04环境:

1
docker pull wnameless/oracle-xe-11g

启动

1
2
docker run -d -p 10022:22 -p 1521:1521 -e ORACLE_ALLOW_REMOTE=true wnameless/oracle-xe-11g:16.04
docker run -d -p 10022:22 -p 1521:1521 -e ORACLE_ALLOW_REMOTE=true myoracle

开放22和1521端口

查看启动日志

1
docker logs -f c154cf832b7a1c77092b2cbe47457153bf853d223d861d0f1cac6af4f73e3ea1

sqlplus测试

Oracle默认连接参数:

1
2
3
4
5
hostname: localhost
port: 1521
sid: xe
username: system
password: oracle
1
2
3
4
5
6
7
本地连接
su - oracle
cd $ORACLE_HOME
bin/sqlplus / as sysdba

或远程连接
sqlplus system/oracle@127.0.0.1:1521/xe

SSH连接

1
2
ssh-copy-id -i -p10022 root@127.0.0.1   #admin
ssh root@127.0.0.1 -p 10022

docker exec连接

1
docker exec -it 5d9430198388 bash

挂载宿主磁盘

1
2
3
4
docker run -d -v /Users/bluexiii/extra/dump:/dump -p 10022:22 -p 1521:1521 -e ORACLE_ALLOW_REMOTE=true wnameless/oracle-xe-11g:16.04

docker run -d -v /Users/bluexiii/extra/dump:/dump -p 10022:22 -p 1521:1521 -e ORACLE_ALLOW_REMOTE=true myoracle

之后就可以impdp了

保存变更

1
2
docker diff 5d9430198388
docker commit 5d9430198388 myoracle

参考文档

https://my.oschina.net/Cleverdada/blog/801124
https://hacpai.com/article/1492914187689

本文简要整理一下Kettle学习过程中的相关文档

官网

https://community.hitachivantara.com/docs/DOC-1009855-data-integration-kettle
https://github.com/pentaho/pentaho-kettle

下载

https://sourceforge.net/projects/pentaho/files/Data%20Integration/

JDK10下启动报错,需切至低版本JDK

官方文档

https://wiki.pentaho.com/display/EAI/Pan+User+Documentation

中文教程

https://www.jianshu.com/p/4d8171150faf
https://juejin.im/entry/590c215ca22b9d0058e8e56b
https://ask.hellobi.com/blog/yuguiyang1990/category/1532

ktr执行

./pan.sh -file=”./scripts/APP_CODE.ktr”
./pan.sh -file=”./scripts/APP_ZHIJI_CORRESPONDING.ktr”
./pan.sh -file=”./scripts/USER_BASIC_INFO.ktr”

kjb执行

./kitchen.sh -file=”./scripts/heimdall.kjb”

命令行输出乱码

1
export LANG=zh_CN.UTF-8

MySQL插表乱码

数据源选项中
Advanced,添加:
set names utf8;

Options,添加:
characterEncoding = utf8

官网

https://www.influxdata.com/time-series-platform/influxdb/

InfluxDB是InfluxData的核心产品。InfluxDB是一个开源分布式时序、时间和指标数据库,使用Go语言编写,无需外部依赖。其设计目标是实现分布式和水平伸缩扩展。

教程

https://jasper-zhang1.gitbooks.io/influxdb/content/Concepts/key_concepts.html
https://docs.influxdata.com/influxdb/v1.5/
https://www.jianshu.com/p/48104975d60a
https://www.jianshu.com/p/a373784c0bf9
https://www.jianshu.com/p/b51ba7f88fb0
https://jkzhao.github.io/2017/12/15/时序数据库InfluxDB/
http://www.ywnds.com/?p=10763
https://anyof.me/articles/390
https://xtutu.gitbooks.io/influxdb-handbook/content/an_zhuang_shi_yong.html

https://www.jianshu.com/p/a1344ca86e9b
https://code-examples.net/zh-CN/docs/influxdata/influxdb/v1.3/troubleshooting/frequently-asked-questions/index

安装方式

https://influxdata.com/downloads/

macOS

1
2
brew update
brew install influxdb

1
2
https://dl.influxdata.com/influxdb/releases/influxdb-1.5.4_darwin_amd64.tar.gz
tar zxvf influxdb-1.5.4_darwin_amd64.tar.gz

Docker

1
docker pull influxdb

Ubuntu

1
2
wget https://dl.influxdata.com/influxdb/releases/influxdb_1.5.4_amd64.deb
sudo dpkg -i influxdb_1.5.4_amd64.deb

CentOS

1
2
wget https://dl.influxdata.com/influxdb/releases/influxdb-1.5.4.x86_64.rpm
sudo yum localinstall influxdb-1.5.4.x86_64.rpm

配置

配置文件路径

1
2
3
/usr/local/etc/influxdb.conf

/etc/influxdb/influxdb.conf

生成默认配置

1
influxd config > /etc/influxdb/influxdb.generated.conf

名词

name desc
database 数据库
measurement 数据库中的表
points 表里面的一行数据
time 每个数据记录时间,是数据库中的主索引(会自动生成)
fields 各种记录值(没有索引的属性)也就是记录的值:温度, 湿度
tags 各种有索引的属性:地区,海拔
series tags的组合,类似复合索引

命令行方式登入

1
2
influx
influx -precision rfc3339 # 显示标准时间

创建数据库

1
2
3
show databases
create database mydb
use mydb

显示所有表

1
show measurements

新建表/插入数据

无建表语句,第一次insert后自动创建 p

1
2
3
4
5
6
7
8
INSERT cpu,host=serverA,region=us_west value=0.64
INSERT temperature,machine=unit42,type=assembly external=25,internal=37

其中:
cpu # 表名
host=serverA,region=us_west # tag
value=0.64 # field

删除表

1
drop measurement weather 

series操作

series表示这个表里面的数据,可以在图表上画成几条线,series主要通过tags排列组合算出来。

1
show series from weather

查询数据

1
2
3
SELECT * FROM /.*/ LIMIT 1
SELECT * FROM cpu_load_short
SELECT * FROM cpu_load_short WHERE value > 0.9

查询/修改RETENTION

1
2
3
4
SHOW RETENTION POLICIES ON "mydb"
单位有 w/d/h/s
ALTER RETENTION POLICY "autogen" ON "mydb" DURATION 100d DEFAULT
ALTER RETENTION POLICY "autogen" ON "mydb" DURATION 0s DEFAULT

用户管理

1
2
3
4
5
6
7
8
显示用户
SHOW USERS
创建用户
CREATE USER "username" WITH PASSWORD 'password'
创建管理员权限的用户
CREATE USER "username" WITH PASSWORD 'password' WITH ALL PRIVILEGES
删除用户
DROP USER "username"

HTTP创建和删除数据库

curl -i -XPOST http://localhost:8086/query –data-urlencode “q=CREATE DATABASE mydb”
curl -POST http://localhost:8086/query –data-urlencode “q=DROP DATABASE mydb”

HTTP添加数据

单条

1
curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary 'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000'

多条

1
2
3
curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary 'cpu_load_short,host=server02 value=0.67
cpu_load_short,host=server02,region=us-west value=0.55 1422568543702900257
cpu_load_short,direction=in,host=server01,region=us-west value=2.0 1422568543702900257'

使用HTTP查询数据

1
curl -GET 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" --data-urlencode "q=show measurements"

或直接在浏览器中:

1
http://localhost:8086/query?pretty=true&db=mydb&q=show%20measurements
1
curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=mydb" --data-urlencode "q=SELECT \"value\" FROM \"cpu_load_short\" WHERE \"region\"='us-west'"

查询多条用分号分割:

1
curl -GET 'http://localhost:8086/query?db=_internal' --data-urlencode "q=show databases;show measurements"

时间格式

epoch=[h,m,s,ms,u,ns]

1
curl -G 'http://localhost:8086/query' --data-urlencode "db=mydb" --data-urlencode "epoch=s" --data-urlencode "q=SELECT value FROM cpu_load_short WHERE region='us-west'"

指定每次查询数据大小

chunk_size

1
curl -G 'http://localhost:8086/query' --data-urlencode "db=mydb" --data-urlencode "chunk_size=200" --data-urlencode "q=SELECT value FROM cpu_load_short WHERE region='us-west'"

WEB控制台

1.3内置8086WEB管理已经移除,需要使用TICK工具栈中的Chronograf来进行管理。

下载安装说明:
https://portal.influxdata.com/downloads

1
brew install chronograf

使用:
http://localhost:8888

测试数据导入

https://docs.influxdata.com/influxdb/v1.6/query_language/data_download/

1
2
3
4
5
6
CREATE DATABASE NOAA_water_database

curl https://s3.amazonaws.com/noaa.water-database/NOAA_data.txt -o NOAA_data.txt
influx -import -path=NOAA_data.txt -precision=s -database=NOAA_water_database

influx -precision rfc3339 -database NOAA_water_database

数据查询详细语法

https://jasper-zhang1.gitbooks.io/influxdb/content/Query_language/data_exploration.html

备份

1
2
3
4
5
6
# 全量
influxd backup -portable /mnt/disk/bak/influxdb/total
# 指定库
influxd backup -portable -database heimdall /mnt/disk/bak/influxdb/heimdall
# 指定时间
influxd backup -portable -database heimdall -start 2018-09-01T00:00:00Z -end 2018-09-30T23:59:59Z /mnt/disk/bak/influxdb/heimdall

恢复

1
2
3
4
5
6
7
8
# 恢复到名为heimdall_bak的库
influxd restore -portable -db heimdall -newdb heimdall_bak /mnt/disk/bak/influxdb/heimdall

# 移动库
create database heimdall_new
use heimdall_bak
SELECT * INTO heimdall_new..:MEASUREMENT FROM /.*/ GROUP BY *
drop heimdall_bak

连续查询

自动缩小取样存储到新的measurement中

1
2
3
4
CREATE CONTINUOUS QUERY "cq_basic" ON "transportation"
BEGIN
SELECT mean("passengers") INTO "average_passengers" FROM "bus_data" GROUP BY time(1h)
END

简介

官网:
https://github.com/Homebrew/homebrew-services

macOS使用launchctl命令加载开机自动运行的服务,brew service可以简化lauchctl的操作。

以MySQL为例,使用launchctl启动:

1
2
ln -sfv /usr/local/opt/mysql/*.plist ~/Library/LaunchAgents
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist

如使用brew service可以简化为:

1
brew services start mysql

常用命令

1
2
3
4
5
6
brew services list  # 查看使用brew安装的服务列表
brew services run formula|--all # 启动服务(仅启动不注册)
brew services start formula|--all # 启动服务,并注册
brew services stop formula|--all # 停止服务,并取消注册
brew services restart formula|--all # 重启服务,并注册
brew services cleanup # 清除已卸载应用的无用的配置

配置文件目录

1
2
/Library/LaunchDaemons # 开机自启,需要sudo
~/Library/LaunchAgents # 用户登录后自启

以homebrew.mxcl.kafka.plist为例:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>homebrew.mxcl.kafka</string>
<key>WorkingDirectory</key>
<string>/usr/local</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/opt/kafka/bin/kafka-server-start</string>
<string>/usr/local/etc/kafka/server.properties</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
<key>StandardErrorPath</key>
<string>/usr/local/var/log/kafka/kafka_output.log</string>
<key>StandardOutPath</key>
<string>/usr/local/var/log/kafka/kafka_output.log</string>
</dict>
</plist>

在这里可以找到服务路径、启动参数、日志路径等

一些废话

起因是macOS下安装Oracle官方的JDK10后,kettle启动报错,需要切至JDK8。
于是需要借助brew cask来安装不同版本的JDK,并自已写脚本切换。

安装

使用brew cask安装JDK

1
2
3
4
brew tap caskroom/versions
brew cask install java6 #JDK6
brew cask install java8 #JDK8
brew cask install java #JDK10

切换脚本

vi .zshrc

1
2
3
4
5
6
7
8
# Switch JDK
export JAVA_6_HOME="/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home"
export JAVA_8_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_172.jdk/Contents/Home"
export JAVA_10_HOME="/Library/Java/JavaVirtualMachines/jdk-10.0.1.jdk/Contents/Home"
export JAVA_HOME=$JAVA_8_HOME # Default JDK8
alias jdk6="export JAVA_HOME=$JAVA_6_HOME;java -version" # to JDK6
alias jdk8="export JAVA_HOME=$JAVA_8_HOME;java -version" # to JDK8
alias jdk10="export JAVA_HOME=$JAVA_8_HOME;java -version" # to JDK10
0%