Jonas-Kress.de

Hive Table Column Mapping

admin — Fri, 16 Dec 2011 11:56:13 +0000

When creating a column mapping in hive all the data is being saved to the metastore.

Example of a hive hbase column mapping:
CREATE EXTERNAL TABLE hivetableName(Col String, Col1 String) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = "colFamily:Col, colFamily:Col") TBLPROPERTIES("hbase.table.name" = "hbasetableName")

If you use a vast amount of columns that are being mapped the table column of the serde params will reach its limit.

You can fix this problem with the following statement if your metastore uses a mysql database:
ALTER TABLE hive.`SERDE_PARAMS` MODIFY COLUMN `PARAM_VALUE` text;

Virtualbox Image Resize Disk (VDI, VMDK)

admin — Fri, 16 Dec 2011 11:29:08 +0000

When resizing a VirtualBox image you may face the error “Resize hard disk operation for this format is not implemented yet!”.
You can fix this by doing a convert to the VDI disk format before trying to resize it.
It could be necessary to boot the converted image before doing a resize!

Convert format of virtual hard disk:
VBoxManage clonehd disk.vmdk disk.vdi --format VDI

Resize virtual hard disk:
VBoxManage modifyhd disk.vdi --resize [SIZE_IN_MB]

You may also want to resize your partitions.
This can be easily done by using the GParted live cd

Master Thesis in Progress

admin — Mon, 28 Nov 2011 15:53:25 +0000

Started working on my master thesis with the title:

Analysis of migrating a MySQL based data warehouse and monitoring system to hadoop

Most relevant books and papers:

Hadoop The Definitive Guide 2nd Edition and
HBase The Definitive Guide
Hive – A Petabyte Scale Data Warehouse
High Performance MySQL

Further links:
http://www.cloudera.com/ Hadoop distribution and installation package
http://hadoop.apache.org/ Hadoop is an open source Apache project