Skip to content
This repository was archived by the owner on Jan 2, 2023. It is now read-only.

Could not load job setup script ("word-count") #35

Open
janpieper opened this issue Oct 5, 2015 · 10 comments
Open

Could not load job setup script ("word-count") #35

janpieper opened this issue Oct 5, 2015 · 10 comments

Comments

@janpieper
Copy link

If I try to run the word-count example, I keep getting the following error and I don't know how to get it running...

$ git clone [email protected]:iconara/rubydoop.git
$ cd rubydoop/examples/word_count
$ bundle install
$ rake package
$ hadoop jar build/word_count.jar word-count conf/hadoop-local.xml README.md output                                                                        
LoadError: no such file to load -- word-count
  require at org/jruby/RubyKernel.java:1065
  require at file:/tmp/hadoop-unjar2289886275803509919/lib/jruby-complete-1.7.12.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
Exception in thread "main" rubydoop.RubydoopRunnerException: Could not load job setup script ("word-count"): "(LoadError) no such file to load -- word-count"
        at rubydoop.RubydoopJobRunner.run(RubydoopJobRunner.java:36)
        at rubydoop.RubydoopJobRunner.run(RubydoopJobRunner.java:18)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at rubydoop.RubydoopJobRunner.main(RubydoopJobRunner.java:50)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.jruby.embed.InvokeFailedException: (LoadError) no such file to load -- word-count
        at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.call(EmbedRubyObjectAdapterImpl.java:317)
        at org.jruby.embed.internal.EmbedRubyObjectAdapterImpl.callMethod(EmbedRubyObjectAdapterImpl.java:249)
        at org.jruby.embed.ScriptingContainer.callMethod(ScriptingContainer.java:1369)
        at rubydoop.RubydoopJobRunner.run(RubydoopJobRunner.java:33)
        ... 10 more
Caused by: org.jruby.exceptions.RaiseException: (LoadError) no such file to load -- word-count
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1065)
        at RUBY.require(file:/tmp/hadoop-unjar2289886275803509919/lib/jruby-complete-1.7.12.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55)

$ ruby -v
jruby 1.7.12 (1.9.3p392) 2014-04-15 643e292 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 +indy [linux-amd64]
@jowl
Copy link
Collaborator

jowl commented Oct 5, 2015

It seems as if you're not running the latest Rubydoop, but follow the latest instructions? Either pull origin/master and re-run your commands or try running

$ hadoop jar build/word_count.jar word_count_job path/to/input path/to/output

which I think is how the previous version worked.

Having that said, it appears as if the instructions in examples/word_count/README.md are a bit off; you shouldn't create the output directory and ../README.md should probably be README.md, unless I'm missing something. But I'm guessing that you've already figured that out as your commands look sensible, apart from the part that runs Hadoop.

@jowl jowl mentioned this issue Oct 5, 2015
Merged
@janpieper
Copy link
Author

I've changed my Gemfile to use the latest code:

gem "rubydoop",
  :git => "[email protected]:iconara/rubydoop.git",
  :branch => "master"

Now I get the following errors:

NoMethodError: undefined method `run' for Rubydoop:Module
  (root) at /tmp/hadoop-unjar1573602031273691108/META-INF/app.home/bin/word-count:10
    load at org/jruby/RubyKernel.java:1081
  (root) at classpath:jar-bootstrap.rb:14
    each at org/jruby/RubyArray.java:1613
  (root) at classpath:jar-bootstrap.rb:10

@jowl
Copy link
Collaborator

jowl commented Oct 5, 2015

Ah, now I think I understand. You must have checked out the latest version of Rubydoop, but not re-built lib/rubydoop.jar. You should revert your word count Gemfile to use the path dependency like it did before, git won't work because the aforementioned JAR file isn't checked in, and then run rake build in the project's root folder (i.e. rubydoop/ in your setup). This will re-build the JAR file which should enable you to run the examples using the same commands as you described first.

@janpieper
Copy link
Author 

$ cd rubydoop
$ git checkout v2.0.0.pre0
$ bundle install
$ rake build
mkdir -p ext/build
rake aborted!
No such file or directory - /home/jan/workspace/rubydoop/.classpath
org/jruby/RubyIO.java:3663:in `foreach'
/home/jan/workspace/rubydoop/Rakefile:14:in `(root)'
org/jruby/RubyBasicObject.java:1533:in `instance_eval'
/home/jan/workspace/rubydoop/Rakefile:12:in `(root)'
org/jruby/RubyProc.java:271:in `call'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:246:in `execute'
org/jruby/RubyArray.java:1613:in `each'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:241:in `execute'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:184:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:205:in `invoke_prerequisites'
org/jruby/RubyArray.java:1613:in `each'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:203:in `invoke_prerequisites'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:183:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:205:in `invoke_prerequisites'
org/jruby/RubyArray.java:1613:in `each'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:203:in `invoke_prerequisites'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:183:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:205:in `invoke_prerequisites'
org/jruby/RubyArray.java:1613:in `each'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:203:in `invoke_prerequisites'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:183:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:170:in `invoke'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:143:in `invoke_task'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:101:in `top_level'
org/jruby/RubyArray.java:1613:in `each'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:101:in `top_level'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:110:in `run_with_threads'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:95:in `top_level'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:73:in `run'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:160:in `standard_exception_handling'
/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:70:in `run'
org/jruby/RubyKernel.java:1081:in `load'
org/jruby/RubyKernel.java:1101:in `eval'
/home/jan/.rvm/gems/jruby-1.7.12/bin/jruby_executable_hooks:15:in `(root)'
Tasks: TOP => build => build:jars => build:compile => build:setup
(See full trace by running task with --trace)

@janpieper
Copy link
Author

Was able to generate a .classpath file, but the build still breaks...

$ rake setup:hadoop
$ rake setup:classpath
$ rake build
$ rake build
mkdir -p ext/build
rake aborted!
Unable to find a javac compiler;
com.sun.tools.javac.Main is not on the classpath.
Perhaps JAVA_HOME does not point to the JDK.
It is currently set to "/usr/lib/jvm/java-8-oracle/jre"
org.apache.tools.ant.taskdefs.compilers.CompilerAdapterFactory.getCompiler(org/apache/tools/ant/taskdefs/compilers/CompilerAdapterFactory.java:129)
org.apache.tools.ant.taskdefs.Javac.findSupportedFileExtensions(org/apache/tools/ant/taskdefs/Javac.java:979)
org.apache.tools.ant.taskdefs.Javac.scanDir(org/apache/tools/ant/taskdefs/Javac.java:956)
org.apache.tools.ant.taskdefs.Javac.execute(org/apache/tools/ant/taskdefs/Javac.java:927)
org.apache.tools.ant.UnknownElement.execute(org/apache/tools/ant/UnknownElement.java:292)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:497)
java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)
RUBY.(root)(/home/jan/workspace/rubydoop/Rakefile:21)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:271)
RUBY.execute(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:246)
org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
RUBY.execute(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:241)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:184)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177)
RUBY.invoke_prerequisites(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:205)
org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
RUBY.invoke_prerequisites(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:203)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:183)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177)
RUBY.invoke_prerequisites(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:205)
org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
RUBY.invoke_prerequisites(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:203)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:183)
RUBY.invoke_with_call_chain(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:177)
RUBY.invoke(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/task.rb:170)
RUBY.invoke_task(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:143)
RUBY.top_level(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:101)
org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
RUBY.top_level(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:101)
RUBY.run_with_threads(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:110)
RUBY.top_level(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:95)
RUBY.run(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:73)
RUBY.standard_exception_handling(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:160)
RUBY.run(/home/jan/.rvm/gems/jruby-1.7.12/gems/rake-10.0.4/lib/rake/application.rb:70)
org.jruby.RubyKernel.load(org/jruby/RubyKernel.java:1081)
org.jruby.RubyKernel.eval(org/jruby/RubyKernel.java:1101)
java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)
home.jan.$_dot_rvm.gems.jruby_minus_1_dot_7_dot_12.bin.jruby_executable_hooks.(root)(/home/jan/.rvm/gems/jruby-1.7.12/bin/jruby_executable_hooks:15)
home.jan.$_dot_rvm.gems.jruby_minus_1_dot_7_dot_12.bin.jruby_executable_hooks.(root)(home/jan/$_dot_rvm/gems/jruby_minus_1_dot_7_dot_12/bin//home/jan/.rvm/gems/jruby-1.7.12/bin/jruby_executable_hooks:15)
Tasks: TOP => build => build:jars => build:compile
(See full trace by running task with --trace)

@iconara
Copy link
Owner

iconara commented Oct 6, 2015

Perhaps JAVA_HOME does not point to the JDK.
It is currently set to "/usr/lib/jvm/java-8-oracle/jre"

You don't have a JDK installed, only a JRE. You can find JDK 8 here: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

However, there's no need to do this if you just want to write your own Rubydoop jobs. It looks like you just want to use the latest version, and if that is the case there's a prerelease of it on Rubygems: gem 'rubydoop', '= 2.0.0.pre0'

I'm not sure you want to use the latest version, though. Unless you absolutely need packaging with Puck you should use the latest stable version, 1.2.0.

@janpieper
Copy link
Author

I think I got it working... These are the steps that was needed to get the word_count example running.

$ git clone [email protected]:iconara/rubydoop.git
$ cd rubydoop
$ git checkout v2.0.0.pre0
$ bundle install
$ rake setup:hadoop
$ rake setup:classpath
$ JAVA_HOME=/usr/lib/jvm/jdk1.8.0_60 rake build
$ cd examples/word_count
$ bundle install
$ rake package
$ hadoop jar build/word_count.jar word-count -conf conf/hadoop-local.xml README.md output

Now there is a output directory which contains the following files:

part-r-00000
.part-r-00000.crc
_SUCCESS
._SUCCESS.crc

@hudsonsferreira
Copy link

@janpieper I followed the same steps of you and it didn't worked.

➜  word_count git:(55aa17c) ✗ hadoop jar build/word_count.jar word-count -conf conf/hadoop-local.xml README.md output
LoadError: no such file to load -- rubydoop
                              require at org/jruby/RubyKernel.java:939
                              require at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
                                <top> at uri:classloader:/META-INF/gem.home/rubydoop-2.0.0.pre0/lib/rubydoop.rb:4
                              require at org/jruby/RubyKernel.java:939
                               (root) at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
                                <top> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
                                 load at org/jruby/RubyKernel.java:957
  block in classpath:jar-bootstrap.rb at classpath:jar-bootstrap.rb:14
                                 each at org/jruby/RubyArray.java:1560
                                <top> at classpath:jar-bootstrap.rb:10

@iconara
Copy link
Owner

iconara commented Nov 10, 2015

Both of you: why are you building your own version, why are you not installing the gem?

@hudsonsferreira
Copy link

@iconara Because this is how the example dir is.
https://github.com/iconara/rubydoop/blob/master/examples/word_count/Gemfile

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@iconara @janpieper @jowl @hudsonsferreira