Another Word For It Patrick Durusau on Topic Maps and Semantic Diversity

March 14, 2012

HBase + Hadoop + Xceivers

Filed under: Hadoop,HBase — Patrick Durusau @ 7:35 pm

HBase + Hadoop + Xceivers by Lars George.

From the post:

Introduction

Some of the configuration properties found in Hadoop have a direct effect on clients, such as HBase. One of those properties is called “dfs.datanode.max.xcievers”, and belongs to the HDFS subproject. It defines the number of server side threads and – to some extent – sockets used for data connections. Setting this number too low can cause problems as you grow or increase utilization of your cluster. This post will help you to understand what happens between the client and server, and how to determine a reasonable number for this property.

The Problem

Since HBase is storing everything it needs inside HDFS, the hard upper boundary imposed by the ”dfs.datanode.max.xcievers” configuration property can result in too few resources being available to HBase, manifesting itself as IOExceptions on either side of the connection.

This is a true sysadmin type post.

Error messages say “DataXceiver,” but set the “dfs.datanode.max.xcievers” property. Post notes “xcievers” is misspelled.

Detailed coverage of the nature of the problem, complete with sample log entries. Along with suggested solutions.

And, word of current work to improve the current situation.

If you are using HBase and Hadoop, put a copy of this with your sysadmin stuff.

1 Comment

  1. […] } #themeHeader #titleAndDescription * { color: black; } tm.durusau.net – Today, 6:41 […]

    Pingback by HBase + Hadoop + Xceivers « Another Word For It | Gestion de l'Information - Gouvernance et Big Data | Scoop.it — March 15, 2012 @ 5:42 am

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Powered by WordPress