Ubuntu18搭建CDH6环境03

1、确保cdt01可以ssh联通cdt02和cdt03

#这个userid与可以无密码使用sudo的userid相同
ssh -l userid cdh02
ssh -l userid cdh03

2、浏览器访问(以后都是界面了)
http://172.16.172.101:7180
用户名:admin
密码:admin

3、根据引导界面,新建Cluster
将172.16.172.101-172.16.172.103都安装好cloudera-manager-agent

4、根据引导界面,选用需要的软件进行安装
安装时,注意合理分配角色,也就是合理分配内存资源

5、依次安装
hdfs
zookeeper
hbase
yarn
hive
spark

6、安装完毕

PS:
1、如果出现找不到jdbc driver的情况

sudo apt-get install libmysql-java

Ubuntu18搭建CDH6环境02

1、cdt01安装

#添加cloudera仓库
wget https://archive.cloudera.com/cm6/6.3.0/ubuntu1804/apt/archive.key
sudo apt-key add archive.key
wget https://archive.cloudera.com/cm6/6.3.0/ubuntu1804/apt/cloudera-manager.list
sudo mv cloudera-manager.list /etc/apt/sources.list.d/

#更新软件清单
sudo apt-get update

#安装jdk8
sudo apt-get install openjdk-8-jdk

#安装cloudera
sudo apt-get install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

2、安装及配置mysql
2.1、安装mysql

sudo apt-get install mysql-server mysql-client libmysqlclient-dev libmysql-java

2.2、停止mysql

sudo service mysql stop

2.3、删除不需要的文件

sudo rm /var/lib/mysql/ib_logfile0
sudo rm /var/lib/mysql/ib_logfile1

2.4、修改配置文件

sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf

#修改或添加以下信息
[mysqld]
transaction-isolation = READ-COMMITTED
max_allowed_packet = 32M
max_connections = 300
innodb_flush_method = O_DIRECT

2.5、启动mysql

sudo service mysql start

2.6、初始化mysql

sudo mysql_secure_installation

3、创建数据库并授权

sudo mysql -uroot -p
-- 创建数据库
-- Cloudera Manager Server
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Activity Monitor
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Reports Manager
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Hue
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Hive Metastore Server
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Sentry Server
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Cloudera Navigator Audit Server
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Cloudera Navigator Metadata Server
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- Oozie
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

#创建用户并授权
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm123456';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon123456';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman123456';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue123456';
GRANT ALL ON hive.* TO 'hive'@'%' IDENTIFIED BY 'hive123456';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry123456';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav123456';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms123456';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie123456';

4、初始化数据库

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm scm123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql amon amon amon123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql rman rman rman123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql hue hue hue123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql hive hive hive123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql sentry sentry sentry123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql nav nav nav123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql navms navms navms123456
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql oozie oozie oozie123456

5、启动

#启动cloudera-scm-server
sudo systemctl start cloudera-scm-server

#查看启动日志,等待Jetty启动完成
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

6、启动
浏览器访问
http://172.16.172.101:7180
用户名:admin
密码:admin

Ubuntu18搭建CDH6环境01

1、环境准备

VirtualBox 6
Ubuntu 18
Cloudera CDH 6.3

2、虚拟机安装Ubuntu18,配置为
1CPU
4G内存
300G硬盘
两块网卡,一块为HostOnly,一块为NAT

3、将虚拟机克隆为三份
如果是手工拷贝,记得修改硬盘UUID、虚拟机UUID、网卡硬件ID

4、设置IP地址、hostname及hosts文件

机器名 HostOnly IP
cdh01 172.16.172.101
cdh02 172.16.172.102
cdh03 172.16.172.103

5、允许无密码使用sudo,至少修改cdh02和cdh03

#edit /etc/sudoers
userid ALL=(ALL:ALL) NOPASSWD: ALL

SonarQube集成代码覆盖率

Java项目

#清理,并构建
mvn clean install javadoc:aggregate -Dadditionalparam=-Xdoclint:none surefire-report:report -Daggregate=true

#sonar收集信息
mvn sonar:sonar -Dsonar.host.url=http://127.0.0.1:9000 -Dsonar.scm.disabled=True -Dsonar.junit.reportPaths=target/surefire

.Net项目

#准备收集信息
MSBuild.SonarQube.Runner.exe begin /k:"MyProject" /n:"MyProject" /v:"1.0" /d:sonar.host.url=http://127.0.0.1:9000 /d:sonar.scm.disabled=True /d:sonar.cs.vstest.reportsPaths="%CD%\*.trx" /d:sonar.cs.vscoveragexml.reportsPaths="%CD%\MyProject.coverage.xml"

#清理并构建项目
msbuild MyProject /t:Clean /t:Rebuild

#进行单元测试并生成测试报告
"%VSINSTALLDIR%\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" collect /output:"%CD%\MyProject.coverage" /verbose "%VSINSTALLDIR%\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" /Logger:trx "MyProject\bin\Debug\MyProject.dll"

#报告转换为xml格式
"%VSINSTALLDIR%\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" analyze /output:"%CD%\MyProject.coverage.xml" "%CD%\MyProject.coverage"

#结束分析
MSBuild.SonarQube.Runner.exe end

SonarQube集成单元测试结果

对于Java来说,最简单的就是用maven进行构建

#surefire-report:report -Daggregate=true要求多个模块的报告合成一个
#可以用site作为替代,比这个更简单
mvn clean install javadoc:aggregate -Dadditionalparam=-Xdoclint:none surefire-report:report -Daggregate=true

#-Dsonar.host.url指定SonarQube地址
#-Dsonar.scm.disabled要求SonarQube不要自己去更新SVN或GIT
#-Dsonar.junit.reportPaths是单元测试报告的路径
mvn sonar:sonar -Dsonar.host.url=http://IP:9000 -Dsonar.scm.disabled=True -Dsonar.junit.reportPaths=Path_To_Surefire_Report

对于CSharp项目来说,用VS自带的CodeCoverage最方便了

#准备
#我一般只指定项目key就好了,多数情况下就是Solution名称
#d:sonar.cs.vscoveragexml.reportsPath一定要指定到正确位置
MSBuild.SonarQube.Runner.exe begin /k:"项目Key" /n:"项目名称" /v:"项目版本" /d:sonar.cs.vscoveragexml.reportsPaths="%CD%\MyProject.coverage.xml"

#构建Solution
msbuild 项目名称

#生成coverage报告
"%VSINSTALLDIR%\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" collect /output:"%CD%\MyProject.coverage" "PATH_TO_TEST_DLL1" "PATH_TO_TEST_DLL2" "PATH_TO_TEST_DLL3"

#coverage报告转为XML版本
"%VSINSTALLDIR%\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" analyze /output:"%CD%\MyProject.coverage.xml" "%CD%\MyProject.coverage"

#完成
MSBuild.SonarQube.Runner.exe end

查看Docker daemon日志

Ubuntu (old using upstart )
/var/log/upstart/docker.log

Ubuntu (new using systemd )
journalctl -u docker.service

Boot2Docker
/var/log/docker.log

Debian GNU/Linux
/var/log/daemon.log

CentOS
/var/log/daemon.log | grep docker

CoreOS
journalctl -u docker.service

Fedora
journalctl -u docker.service

Red Hat Enterprise Linux Server
/var/log/messages | grep docker

OpenSuSE
journalctl -u docker.service

OSX
~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/log/d‌​ocker.log

Windows
Get-EventLog -LogName Application -Source Docker -After (Get-Date).AddMinutes(-5) | Sort-Object Time, as mentioned here.

Selenium入门04

接上一节,说一下Grid的用法

1、启动hub

set JAVA_HOME=C:\NeoLanguages\Java\JDK\jdk_x86_1.8.0_77
set PATH=%JAVA_HOME%\bin;%PATH%;C:\NeoTest\TestSelenium\trunk\bin\x86\;
set webdriver.gecko.driver=C:\NeoTest\TestSelenium\trunk\bin\x86\geckodriver.exe

java -jar ../lib/selenium-server-standalone-3.3.1.jar -role hub -hubConfig hubConfig.json -debug true

pause

hubConfig.json

{
  "port": 4444,
  "newSessionWaitTimeout": -1,
  "servlets" : [],
  "withoutServlets": [],
  "custom": {},
  "capabilityMatcher": "org.openqa.grid.internal.utils.DefaultCapabilityMatcher",
  "throwOnCapabilityNotPresent": true,
  "cleanUpCycle": 5000,
  "role": "hub",
  "debug": false,
  "browserTimeout": 0,
  "timeout": 1800
}

2、启动node

set JAVA_HOME=C:\NeoLanguages\Java\JDK\jdk_x86_1.8.0_77
set PATH=%JAVA_HOME%\bin;%PATH%;C:\NeoTest\TestSelenium\trunk\bin\x86\;
set webdriver.gecko.driver=C:\NeoTest\TestSelenium\trunk\bin\x86\geckodriver.exe

java -jar ../lib/selenium-server-standalone-3.3.1.jar -role node -nodeConfig node01Config.json -debug true

pause

node01Config.json

{
 "capabilities":
  [
    {
      "browserName": "firefox",
      "maxInstances": 5,
      "seleniumProtocol": "WebDriver"
    }
  ],
  "proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
  "maxSession": 5,
  "port": 5555,
  "register": true,
  "registerCycle": 5000,
  "hub": "http://192.168.130.178:4444",
  "nodeStatusCheckTimeout": 5000,
  "nodePolling": 5000,
  "role": "node",
  "unregisterIfStillDownAfter": 60000,
  "downPollingLimit": 2,
  "debug": false,
  "servlets" : [],
  "withoutServlets": [],
  "custom": {}
}

3、可以在hub中看到注册情况
http://localhost:4444/grid/console
这个地方一定要注意,每个node上的浏览器及操作系统信息,向hub发送指令时,必须符合该信息

4、运行脚本
TestHelloWorldHub.py

# !C:\Languages\Python\Python27\python.exe
# -*- coding: utf-8 -*-
'''
Created on 2016-10-22
@author: Hansen
HelloWorld sample for NeoSelenium
'''

from NeoSelenium import initEngine
from NeoSelenium import deInitEngine
from NeoSelenium import initEngineHub

#一个简单的查询测试
def neohope_search_test():
    try:
        #myEngine = initEngine('ie32')
        #myEngine = initEngine('ie64')
        #myEngine = initEngine('chrome')
        #myEngine = initEngine('ff32')
        #myEngine = initEngine('ff64')
        #myEngine = initEngineRemote()
	myEngine = initEngineHub()

        base_url = "http://www.neohope.com"
        myEngine.get(base_url + "/")
        myEngine.find_element_by_name("s").clear()
        myEngine.find_element_by_name("s").send_keys("Metabase")
        myEngine.find_element_by_css_selector("button.search-submit").click()
        myEngine.implicitly_wait(1000)
        #print(myEngine.find_elements_by_xpath("//div[@id='content']/article"))
        queryResultLenght = len(myEngine.find_elements_by_xpath("//div[@id='content']/article"))
        #print(queryResultLenght)
        #应该是1但现在是10
        assert queryResultLenght==1
    finally:
        #deInitEngine(myEngine)
        print("test end")

#start here
neohope_search_test()

NeoSelenium.py

#!C:\Languages\Python\Python27\python.exe
# -*- coding: utf-8 -*-
'''
Created on 2017-04-07
@author: Hansen
NeoSelenium
'''

import os
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

NPath = None
x86Path = "C:\NeoTest\TestSelenium\\trunk\\bin\\x86;"
x64Path = "C:\NeoTest\TestSelenium\\trunk\\bin\\x64;"

'''
selenium remote engine init
'''
def initEngineRemote():
    engine = webdriver.Remote(command_executor="http://192.168.130.178:4444/wd/hub", desired_capabilities=DesiredCapabilities.FIREFOX)
    return engine

'''
selenium remote engine init
'''
def initEngineHub():
    engine = webdriver.Remote(command_executor="http://192.168.130.178:5555/wd/hub", desired_capabilities={
        "browserName": "firefox",
        "platform": "VISTA"
    })
    return engine   

'''
selenium engine deInit
'''
def deInitEngine(engine):
    engine.close()
    engine.quit()


Selenium入门03

接上一节,说一下如何进行远程测试

1、下载selenium-server-standalone的jar包
http://www.seleniumhq.org/download/

2、运行selenium-server-standalone

set JAVA_HOME=C:\NeoLanguages\Java\JDK\jdk_x86_1.8.0_77
set PATH=%JAVA_HOME%\bin;%PATH%;C:\NeoTest\TestSelenium\trunk\bin\x86\;
set webdriver.gecko.driver=C:\NeoTest\TestSelenium\trunk\bin\x86\geckodriver.exe

java -jar ../lib/selenium-server-standalone-3.3.1.jar -role standalone

pause

3、远程测试脚本
TestHelloWorldRemote.py

# !C:\Languages\Python\Python27\python.exe
# -*- coding: utf-8 -*-
'''
Created on 2016-10-22
@author: Hansen
HelloWorld sample for NeoSelenium
'''

from NeoSelenium import deInitEngine
from NeoSelenium import initEngineRemote

#一个简单的查询测试
def neohope_search_test():
    try:
        #myEngine = initEngine('ie32')
        #myEngine = initEngine('ie64')
        #myEngine = initEngine('chrome')
        #myEngine = initEngine('ff32')
        #myEngine = initEngine('ff64')
	myEngine = initEngineRemote()

        base_url = "http://www.neohope.com"
        myEngine.get(base_url + "/")
        myEngine.find_element_by_name("s").clear()
        myEngine.find_element_by_name("s").send_keys("Metabase")
        myEngine.find_element_by_css_selector("button.search-submit").click()
	myEngine.implicitly_wait(1000)
	#print(myEngine.find_elements_by_xpath("//div[@id='content']/article"))
	queryResultLenght = len(myEngine.find_elements_by_xpath("//div[@id='content']/article"))
	#print(queryResultLenght)
	#应该是1但现在是10
        assert queryResultLenght==1
    finally:
        #deInitEngine(myEngine)
	print("test end")

#start here
neohope_search_test()

NeoSelenium.py

#!C:\Languages\Python\Python27\python.exe
# -*- coding: utf-8 -*-
'''
Created on 2017-04-07
@author: Hansen
NeoSelenium
'''

import os
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

NPath = None
x86Path = "C:\NeoTest\TestSelenium\\trunk\\bin\\x86;"
x64Path = "C:\NeoTest\TestSelenium\\trunk\\bin\\x64;"

'''
selenium remote engine init
'''
def initEngineRemote():
    engine = webdriver.Remote(command_executor="http://192.168.130.178:4444/wd/hub", desired_capabilities=DesiredCapabilities.FIREFOX)
    return engine

'''
selenium engine deInit
'''
def deInitEngine(engine):
    engine.close()
    engine.quit()