Kafa Hadoop Abubuwan da ake buƙata da Hardarfafa Tsaro - Kashi na 2


Hadoop Cluster Building tsari ne mataki-mataki inda tsarin zai fara daga siyan sabobin da ake buƙata, hawa cikin rake, cabling, da sauransu da sanyawa a cikin Datacentre. Don haka muna buƙatar shigar da OS, ana iya yin amfani da kickstart a cikin yanayin lokaci na ainihi idan girman tarin ya girma. Da zarar an shigar da OS, to muna buƙatar shirya sabar don shigar da Hadoop kuma muna buƙatar shirya sabobin bisa ga manufofin securityungiyar tsaro.

  • Mafi Kyawawan Ayyuka don Sanya Hadoop Server akan CentOS/RHEL 7 - Sashe na 1

A cikin wannan labarin, zamu bi cikin matakan OS-waɗanda aka ba da shawarar Cloudera. Hakanan, mun haskaka wasu mahimmin nasihu game da eningarfafa Tsaro bisa ga CIS Benchmark don Sabobin samarwa. Wadannan tsaro Hardening na iya zama daban-daban gwargwadon bukatun.

Kafa Cloudera Hadoop Abubuwan da ake buƙata

Anan, zamu tattauna abubuwanda ake buƙata na matakin OS wanda Cloudera ta bada shawarar.

Ta hanyar tsoho, Transparent Huge Page (THP) ana aiki dashi a cikin injunan Linux waɗanda basa iya hulɗa tare da kayan aikin Hadoop kuma yana ƙasƙantar da aikin Cluster gabaɗaya. Don haka muna buƙatar musaki wannan don samun ingantaccen aiki ta amfani da umarnin echo mai zuwa.

# echo never > /sys/kernel/mm/transparent_hugepage/enabled 
# echo never > /sys/kernel/mm/transparent_hugepage/defrag 

Ta hanyar tsoho, ƙimar vm.swappiness ita ce 30 ko 60 don yawancin injunan Linux.

# sysctl vm.swappiness

Samun mafi darajar swappiness ba'a ba da shawarar ga sabobin Hadoop ba saboda yana iya haifar da tsawan tarkacen datti. Kuma, tare da mafi girman darajar canzawa, za a iya adana bayanai don sauya ƙwaƙwalwar ajiya koda kuwa muna da isasshen ƙwaƙwalwar ajiya. Valueara darajar sauyawa na iya yin ƙwaƙwalwar ajiyar jiki don ƙunsar ƙarin shafukan ƙwaƙwalwar ajiya.

# sysctl vm.swappiness=1

Ko kuma, zaku iya buɗe fayil ɗin /etc/sysctl.conf kuma ƙara \"vm.swappiness = 1 \" a karshen.

vm.swappiness=1

Kowane sabar Hadoop zai kasance yana da nasa nauyin tare da ayyuka da yawa (daemons) da ke gudana akan hakan. Duk sabobin za su kasance suna sadarwa da juna ta hanyar da ta dace don dalilai daban-daban.

Misali, Datanode zai turawa Namenode bugun zuciya kowane dakika 3 domin Namenode ya tabbatar Datanode yana raye.

Idan duk sadarwa ta faru tsakanin diyar da ke tsakanin sabar daban ta hanyar Firewall, zai zama ƙarin nauyi ga Hadoop. Don haka ya fi dacewa a dakatar da katangar bango a cikin sabobin mutum a cikin Cluster.

# iptables-save > ~/firewall.rules
# systemctl stop firewalld
# systemctl disable firewall

Idan muka ci gaba da kunna SELinux, zai haifar da matsala yayin shigar da Hadoop. Kamar yadda Hadoop kwamfyuta ce mai tattara abubuwa, Cloudera Manager zai isa ga dukkan sabobin da ke cikin rukunin don shigar da Hadoop da ayyukanta kuma hakan zai ƙirƙiri kundayen sabis na sabis a duk inda ake buƙata.

Idan an kunna SELinux, ba zai bar Cloudera Manager ya mallaki shigarwa yadda yake so ba. Don haka, kunna SELinux zai zama cikas ga Hadoop kuma zai haifar da al'amuran aiki.

Kuna iya bincika matsayin SELinux ta amfani da umarnin da ke ƙasa.

# sestatus

Yanzu, buɗe fayil ɗin/sauransu/selinux/config kuma musaki SELINUX kamar yadda aka nuna.

SELinux=disabled

Bayan katse SELinux, kuna buƙatar sake kunna tsarin don yin aiki.

# reboot

A cikin Hadoop Cluster, duk sabobin yakamata a haɗa su da Lokaci don kauce wa kurakurai masu daidaita agogo. RHEL/CentOS 7 yana da chronyd inbuilt don aiki tare agogo/aiki tare, amma Cloudera ya bada shawarar amfani da NTP.

Muna buƙatar shigar da NTP kuma saita ta. Da zarar an shigar, dakatar da 'chronyd' kuma a kashe. Domin, idan sabar da take da ntpd da chronyd masu gudana, Cloudera Manager zaiyi la'akari da chronyd don aiki tare lokaci, to zai jefa kuskure koda kuwa muna da lokacin aiki ta hanyar ntp.

# yum -y install ntp
# systemctl start ntpd
# systemctl enable ntpd
# systemctl status ntpd

Kamar yadda muka ambata a sama, ba mu buƙatar chronyd mai aiki kamar yadda muke amfani da ntpd. Bincika halin chronyd, idan yana gudana tasha kuma a kashe shi. Ta hanyar tsoho, an dakatar da chronyd har sai mun fara shi bayan girke-girke na OS, kawai muna buƙatar musaki don gefen aminci.

# systemctl status chronyd
# systemctl disable chronyd

Dole ne mu saita sunan mai masauki tare da FQDN (cikakken ualarancin Sunan Yanki). Kowane sabar ya zama yana da suna na Canonical na musamman. Don warware sunan mai masauki, ko dai muna buƙatar daidaita DNS ko/sauransu/runduna. A nan, za mu saita/sauransu/runduna.

Adireshin IP da FQDN na kowane sabar yakamata a shiga cikin/sauransu/rundunonin duk sabobin. Bayan haka Manajan Cloudera ne kawai ke iya sadarwa da duk sabobin tare da sunan mai masaukin sa.

# hostnamectl set-hostname master1.linux-console.net

Na gaba, saita fayil/sauransu/runduna. Misali: - Idan muna da tarin kumburi 5 tare da iyaye 2 da ma'aikata 3, zamu iya saita/etc/runduna kamar yadda ke ƙasa.

Kamar yadda Hadoop ya kasance daga Java, duk masu watsa shiri ya kamata a sanya Java tare da sigar da ta dace. Anan zamu sami OpenJDK. Ta hanyar tsoho, Cloudera Manager zai girka OracleJDK amma, Cloudera tana bada shawarar samun OpenJDK.

# yum -y install java-1.8.0-openjdk-devel
# java -version

Hadoop Tsaro da Hardening

A wannan sashin, zamu tafi Harden Hadoop yanayin tsaro environment

Ountirƙirar 'autofs' yana ba da damar hawa kayan aiki ta atomatik kamar USB, CD/DVD. Mai amfani tare da samun damar zahiri zai iya haɗa USB ɗin su ko kowane matsakaicin Ma'aji don samun damar shigar da bayanai. Yi amfani da umarnin da ke ƙasa don tabbatar ko yana da nakasa ko a'a, idan ba musaki shi ba.

# systemctl disable autofs
# systemctl is-enabled autofs

Fayil ɗin girke-girke yana ƙunshe da bayanai masu mahimmanci game da saitunan taya da takardun shaidarka don buɗe zaɓukan taya. Fayil din grub config 'grub.cfg' wanda yake a/boot/grub2 kuma an danganta shi azaman /etc/grub2.conf kuma tabbatar da cewa grub.cfg mallakar mai amfani ne.

# cd /boot/grub2

Yi amfani da umarnin da ke ƙasa don bincika Uid da Gid duka 0/tushe ne kuma 'rukuni' ko 'wasu' bai kamata su sami izini ba.

# stat /boot/grub2/grub.cfg

Yi amfani da umarnin da ke ƙasa don cire izini daga wasu da rukuni.

# chmod og-rwx /boot/grub2/grub.cfg

Wannan saitin zai guji sauran rebooting na sabar. watau, Yana buƙatar kalmar wucewa don sake yi sabar. Idan ba'a saita shi ba, masu amfani da izini zasu iya kora da sabar kuma zasu iya yin canje-canje ga ɓangarorin taya.

Yi amfani da umarnin da ke ƙasa don saita kalmar sirri.

# grub2-mkpasswd-pbkdf2

Passwordara kalmar sirri da aka kirkira sama zuwa /etc/grub.d/01_users fayil.

Na gaba, sake samar da fayil ɗin sanyi.

# grub2-mkconfig > /boot/grub2/grub.cfg

Prelink shiri ne na software wanda zai iya haɓaka rashin ƙarfi a cikin sabar idan masu amfani da ƙeta zasu iya kawo cikas ga ɗakunan karatu na yau da kullun kamar su libc.

Yi amfani da umarnin da ke ƙasa don cire shi.

# yum remove prelink

Ya kamata muyi la’akari da kashe wasu sabis/ladabi don kauce wa yuwuwar kai hari.

# systemctl disable <service name>

  • Kashe Sabis ɗin Hanyar Sadarwa - Tabbatar da sabis na hanyar sadarwa - caji, rana, jefar, amo, ba a kunna lokaci ba. Waɗannan ayyukan hanyar sadarwar sune don gyarawa da gwaji, ana bada shawara don musaki wanda zai iya rage harin nesa.
  • Kashe TFTP & FTP - Dukansu yarjejeniya ba za ta goyi bayan sirrin bayanai ko takardun shaidarka ba. Mafi kyawun aiki ne kada a sami a cikin sabar sai dai idan an buƙaci a bayyane. Mafi yawa waɗannan ladabi an girka kuma an kunna su akan Fayil.
  • Kashe DHCP - DHCP shine ladabi wanda zai rarraba adireshin IP da ƙarfi. An ba da shawarar a dakatar da shi sai dai idan ya kasance uwar garken DHCP ne don kauce wa yiwuwar kai hari.
  • Kashe HTTP - HTTP ita ce yarjejeniya da za a iya amfani da ita don karɓar bakuncin abubuwan yanar gizo. Baya ga sabar Jagora/Gudanarwa (inda WebUI na ayyuka za a daidaita su kamar CM, Hue, da sauransu), za mu iya katse HTTP a kan wasu ƙwararrun ma'aikata waɗanda ke iya guje wa yiwuwar kai hari.

Takaitawa

Mun wuce cikin shirye-shiryen sabar wanda ya kunshi abubuwanda ake bukata na Cloudera Hadoop da wasu lamuran tsaro. Abubuwan buƙatun farko na OS waɗanda aka bayyana ta Cloudera sune tilas don sauƙin shigar da Hadoop. Yawancin lokaci, za a shirya rubutu mai taurin gwiwa tare da amfani da CIS Benchmark kuma ana amfani da shi don dubawa da kuma daidaita rashin bin doka a ainihin lokacin.

A cikin ƙaramin shigarwa na CentOS/RHEL 7, ana shigar da ayyukan yau da kullun/software kawai, wannan zai guje wa haɗarin da ba'a so. Kodayake imalaramin allationaramin shigarwa za ayi maganganu da yawa na binciken tsaro kafin sanya Hadoop, koda bayan gina tarin, kafin motsa movingungiyar zuwa Aiki/Samarwa.