ArrayIndexOutOfBoundsException

[flamingofugang]

I got the following exception error when I run hadoop mr:

Sampling started
16/07/14 09:25:29 INFO input.FileInputFormat: **Total input paths to process : 0**
16/07/14 09:25:29 INFO partition.InputSampler: Using 0 samples
16/07/14 09:25:29 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
16/07/14 09:25:29 INFO compress.CodecPool: Got brand-new compressor [.deflate]
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
        at org.apache.hadoop.mapreduce.lib.partition.InputSampler.writePartitionFile(InputSampler.java:340)
        at org.rdfhdt.mrbuilder.HDTBuilderDriver.runDictionaryJob(HDTBuilderDriver.java:242)
        at org.rdfhdt.mrbuilder.HDTBuilderDriver.main(HDTBuilderDriver.java:112)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Here is the code snippet causing exception:
InputSampler.writePartitionFile(job, new InputSampler.IntervalSampler<Text, Text>(this.conf.getSampleProbability()));

It seems the input files are not found... I created 'input' directory, and put ntriples '.nt' files in it.

Any idea?

Best,
Gang

[artob]

@flamingofugang Just to check, is this a regression given the changes in the last two months, or is this the furthest as yet in making HDT-MR actually work?

[flamingofugang]

There is still lzo compression library issue, I will report later on.

[artob]

Related pull request: #4

[artob]

@flamingofugang Does your pull request #4 resolve this?

[flamingofugang]

The java program takes in lzo compressed ntriples file as input, and the lzo file should be indexed as far as I understand.

I changed the pom.xml to make dependency on a locally build hadoop lzo package with native lzo library available.

I recommend this should be explained in the README file:

First the user need to install lzo and lzop
Second, build hadoop lzo package: https://github.com/twitter/hadoop-lzo
Then register that jar in the local .m2 repository, then build this hdt-mr package.

[tangina-sultana]

Hi, can you share the installation process of HDT-MR?