Installation of CDH 5.7.6 on Ubuntu 14.04

Record the process of installing CDH on Ubuntu 14.04, no guarantee that it will work flawlessly on all machines...
The following commands are recommended to be run under su.

System Installation#

Omitted

System Configuration#

Static IP#

Edit /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto p2p1
iface p2p1 inet static
address 172.16.1.68
netmask 255.255.255.0
gateway 172.16.1.1

Hosts#

Set /etc/hosts, use names instead of IP addresses

172.16.1.68     master
172.16.1.69     data1
172.16.1.70     data2

Hostname#

Add the future machine name to /etc/hostname on each machine.

The machine name should be the same as the name in the hosts file.

SSH passwordless communication#

Go to ~/.ssh, generate keys with ssh-kengen (press enter when prompted).

Send the keys to other machines using ssh-copy-id [hostname]

For example: ssh-copy-id localhost sends a key to yourself.

Disable firewall#

sudo ufw disable

sudo reboot

Time synchronization#

apt-get install ntp

Update sources#

This is about Cloudera's source, others are omitted.

cd /etc/apt/sources.list.d
sudo wget http://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/cloudera.list
sudo apt-get update

Get apt key

curl -s https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/archive.key| sudo apt-key add -

Install Cloudera Server#

On the Cloudera Manager machine

sudo apt-get -y --force-yes install cloudera-manager-daemons cloudera-manager-server

MySQL#

Configure MySQL only on the master, apt-get install mysql-server

Stop the MySQL service
Add configuration file, vim /etc/mysql/conf.d/mysql_cloudera_manager.cnf

Content:

[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

# For MySQL version 5.1.8 or later. Comment out binlog_format for older versions.
binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

Restart the service, service mysql start
Log in to MySQL and create databases

create database amon DEFAULT CHARACTER SET utf8;  
grant all on amon.* TO 'amon'@'localhost' IDENTIFIED BY 'amon_password';  
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';  
create database smon DEFAULT CHARACTER SET utf8;  
grant all on smon.* TO 'smon'@'localhost' IDENTIFIED BY 'smon_password';  
grant all on smon.* TO 'smon'@'%' IDENTIFIED BY 'smon_password';  
create database rman DEFAULT CHARACTER SET utf8;  
grant all on rman.* TO 'rman'@'localhost' IDENTIFIED BY 'rman_password';  
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';  
create database hmon DEFAULT CHARACTER SET utf8;  
grant all on hmon.* TO 'hmon'@'localhost' IDENTIFIED BY 'hmon_password';  
grant all on hmon.* TO 'hmon'@'%' IDENTIFIED BY 'hmon_password';  
create database hive DEFAULT CHARACTER SET utf8;  
grant all on hive.* TO 'hive'@'localhost' IDENTIFIED BY 'hive_password';  
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';  
create database oozie;  
grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie';  
grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie';

Enable remote access to the database, comment out the line bind-host:localhost in my.cnf

JDK installation#

apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install oracle-j2sdk1.7

Master and agent both need to be installed, assuming the source has been added.

Cloudera Manager initialize database#

/usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p --scm-host localhost scm scm scm_password

Start CM on master#

service cloudera-scm-server start

Deploy CDH using the web page#

The address is http://host:7180

The login account and password are both admin