UndefinedConversionError "\xC3" from ASCII-8BIT to UTF-8 within JRuby/Rails/Sprockets dev mode #71

chadlwilson · 2022-10-18T16:00:17Z

Hi there - similar to #2 after some change I can't trace, I have started getting a persistent failure of a particular stylesheet to compile dynamically in a dev environment. It seems to be fine when precompiling assets, and I can't quite trace what is different between the two setups.

Environment:
MacOS 12.6 Monterey
JRuby 9.3.8.0 under Java 17
rails 6.1.7
sprockets 4.1.1
sassc-embedded 1.54.0 installed per https://github.com/ntkme/sassc-embedded-shim-ruby

Possible problem
I am guessing from #2 that there is some default encoding problem somewhere, but I am not quite sure where to look as the code seems rather different than when #2 was discussed.

I tried forcing Java file.encoding=UTF-8 in case there is a JRuby default thing going on here, as well as forcing the default encodings at Rails level - seems no difference.

The issue seems to be with debug assets as changing to
config.assets.debug = false in application.rb (or for the specific problematic stylesheet's tag) seems to allow render to work, at least for this particular page.

# Fails
<%= stylesheet_link_tag "new-theme", {media: "all", debug: Rails.env.development? && params[:debug_assets]} %>

# Works OK
<%= stylesheet_link_tag "new-theme", {media: "all", debug: false} %>

Any workarounds or help to narrow the problem down would be appreciated.

Error Detail

The same top level SCSS asset (new-theme) fails to compile with

ActionView::Template::Error ("\xC3" from ASCII-8BIT to UTF-8):
    4: <%= yield :before_stylesheets %>
    5: <%= stylesheet_link_tag "application", {media: "all", debug: Rails.env.development? && params[:debug_assets]} %>
    6: <%= stylesheet_link_tag "patterns/application", {media: "all", debug: Rails.env.development? && params[:debug_assets]} %>
    7: <%= stylesheet_link_tag "new-theme", {media: "all", debug: Rails.env.development? && params[:debug_assets]} %>
    8:
    9: <%= javascript_include_tag "application", debug: Rails.env.development? && params[:debug_assets] %>
   10:

app/views/shared/_head.html.erb:7
app/views/layouts/pipelines.html.erb:4
app/controllers/stages_controller.rb:165:in `block in render_stage'
app/controllers/stages_controller.rb:164:in `render_stage'
app/controllers/stages_controller.rb:41:in `overview'

Full stack trace is

org/jruby/RubyString.java:6473:in `encode'
com/google/protobuf/jruby/RubyMessage.java:168:in `[]='
com/google/protobuf/jruby/RubyMessage.java:99:in `initialize'
org/jruby/RubyClass.java:890:in `new'
sass-embedded (1.55.0) lib/sass/embedded/host.rb:55:in `block in compile_request'
sass-embedded (1.55.0) lib/sass/embedded/host.rb:133:in `await'
sass-embedded (1.55.0) lib/sass/embedded/host.rb:43:in `compile_request'
sass-embedded (1.55.0) lib/sass/embedded.rb:209:in `compile_string'
sass-embedded (1.55.0) lib/sass/embedded.rb:47:in `compile_string'
sassc-embedded (1.54.0) lib/sassc/embedded.rb:17:in `render'
sassc-rails (2.1.2) lib/sassc/rails/template.rb:40:in `block in call'
sprockets (4.1.1) lib/sprockets/utils.rb:141:in `module_include'
sassc-rails (2.1.2) lib/sassc/rails/template.rb:39:in `call'
sprockets (4.1.1) lib/sprockets/processor_utils.rb:84:in `call_processor'
sprockets (4.1.1) lib/sprockets/processor_utils.rb:66:in `block in call_processors'
org/jruby/RubyArray.java:1947:in `reverse_each'
sprockets (4.1.1) lib/sprockets/processor_utils.rb:65:in `call_processors'
sprockets (4.1.1) lib/sprockets/processor_utils.rb:22:in `block in CompositeProcessor'
sprockets (4.1.1) lib/sprockets/processor_utils.rb:33:in `call'
sprockets (4.1.1) lib/sprockets/processor_utils.rb:84:in `call_processor'
sprockets (4.1.1) lib/sprockets/processor_utils.rb:66:in `block in call_processors'
org/jruby/RubyArray.java:1947:in `reverse_each'
sprockets (4.1.1) lib/sprockets/processor_utils.rb:65:in `call_processors'
sprockets (4.1.1) lib/sprockets/loader.rb:182:in `load_from_unloaded'
sprockets (4.1.1) lib/sprockets/loader.rb:59:in `block in load'
sprockets (4.1.1) lib/sprockets/loader.rb:337:in `fetch_asset_from_dependency_cache'
sprockets (4.1.1) lib/sprockets/loader.rb:43:in `load'
sprockets (4.1.1) lib/sprockets/cached_environment.rb:44:in `block in load'
org/jruby/RubyHash.java:1354:in `fetch'
sprockets (4.1.1) lib/sprockets/cached_environment.rb:44:in `load'
sprockets (4.1.1) lib/sprockets/add_source_map_comment_to_asset_processor.rb:48:in `call'
sprockets (4.1.1) lib/sprockets/processor_utils.rb:84:in `call_processor'
sprockets (4.1.1) lib/sprockets/processor_utils.rb:66:in `block in call_processors'
org/jruby/RubyArray.java:1947:in `reverse_each'
sprockets (4.1.1) lib/sprockets/processor_utils.rb:65:in `call_processors'
sprockets (4.1.1) lib/sprockets/loader.rb:182:in `load_from_unloaded'
sprockets (4.1.1) lib/sprockets/loader.rb:59:in `block in load'
sprockets (4.1.1) lib/sprockets/loader.rb:337:in `fetch_asset_from_dependency_cache'
sprockets (4.1.1) lib/sprockets/loader.rb:43:in `load'
sprockets (4.1.1) lib/sprockets/cached_environment.rb:44:in `block in load'
org/jruby/RubyHash.java:1354:in `fetch'
sprockets (4.1.1) lib/sprockets/cached_environment.rb:44:in `load'
sprockets (4.1.1) lib/sprockets/base.rb:81:in `find_asset'
sprockets (4.1.1) lib/sprockets/base.rb:119:in `[]'
sprockets-rails (3.4.2) lib/sprockets/rails/helper.rb:363:in `find_asset'
sprockets-rails (3.4.2) lib/sprockets/rails/helper.rb:348:in `find_debug_asset'
sprockets-rails (3.4.2) lib/sprockets/rails/helper.rb:230:in `block in lookup_debug_asset'
sprockets-rails (3.4.2) lib/sprockets/rails/helper.rb:243:in `block in resolve_asset'
org/jruby/RubyEnumerable.java:626:in `detect'
sprockets-rails (3.4.2) lib/sprockets/rails/helper.rb:242:in `resolve_asset'
sprockets-rails (3.4.2) lib/sprockets/rails/helper.rb:229:in `lookup_debug_asset'
sprockets-rails (3.4.2) lib/sprockets/rails/helper.rb:171:in `block in stylesheet_link_tag'
org/jruby/RubyArray.java:2667:in `map'
sprockets-rails (3.4.2) lib/sprockets/rails/helper.rb:170:in `stylesheet_link_tag'
org/jruby/RubyKernel.java:2003:in `public_send'
actionview (6.1.7) lib/action_view/base.rb:247:in `_run'
actionview (6.1.7) lib/action_view/template.rb:154:in `block in render'
activesupport (6.1.7) lib/active_support/notifications.rb:205:in `instrument'
actionview (6.1.7) lib/action_view/template.rb:345:in `instrument_render_template'
actionview (6.1.7) lib/action_view/template.rb:152:in `render'
actionview (6.1.7) lib/action_view/renderer/partial_renderer.rb:285:in `block in render_partial_template'
activesupport (6.1.7) lib/active_support/notifications.rb:203:in `block in instrument'
activesupport (6.1.7) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.1.7) lib/active_support/notifications.rb:203:in `instrument'
actionview (6.1.7) lib/action_view/renderer/partial_renderer.rb:280:in `render_partial_template'
actionview (6.1.7) lib/action_view/renderer/partial_renderer.rb:271:in `render'
actionview (6.1.7) lib/action_view/renderer/renderer.rb:81:in `render_partial_to_object'
actionview (6.1.7) lib/action_view/renderer/renderer.rb:27:in `render_to_object'
actionview (6.1.7) lib/action_view/renderer/renderer.rb:22:in `render'
actionview (6.1.7) lib/action_view/helpers/rendering_helper.rb:38:in `block in render'
actionview (6.1.7) lib/action_view/base.rb:273:in `in_rendering_context'
actionview (6.1.7) lib/action_view/helpers/rendering_helper.rb:34:in `render'
org/jruby/RubyKernel.java:2003:in `public_send'
actionview (6.1.7) lib/action_view/base.rb:247:in `_run'
actionview (6.1.7) lib/action_view/template.rb:154:in `block in render'
activesupport (6.1.7) lib/active_support/notifications.rb:205:in `instrument'
actionview (6.1.7) lib/action_view/template.rb:345:in `instrument_render_template'
actionview (6.1.7) lib/action_view/template.rb:152:in `render'
actionview (6.1.7) lib/action_view/renderer/template_renderer.rb:72:in `block in render_with_layout'
activesupport (6.1.7) lib/active_support/notifications.rb:203:in `block in instrument'
activesupport (6.1.7) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.1.7) lib/active_support/notifications.rb:203:in `instrument'
actionview (6.1.7) lib/action_view/renderer/template_renderer.rb:70:in `render_with_layout'
actionview (6.1.7) lib/action_view/renderer/template_renderer.rb:55:in `render_template'
actionview (6.1.7) lib/action_view/renderer/template_renderer.rb:11:in `render'
actionview (6.1.7) lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
actionview (6.1.7) lib/action_view/renderer/renderer.rb:29:in `render_to_object'
actionview (6.1.7) lib/action_view/rendering.rb:117:in `block in _render_template'
actionview (6.1.7) lib/action_view/base.rb:273:in `in_rendering_context'
actionview (6.1.7) lib/action_view/rendering.rb:116:in `_render_template'
actionpack (6.1.7) lib/action_controller/metal/streaming.rb:218:in `_render_template'
actionview (6.1.7) lib/action_view/rendering.rb:103:in `render_to_body'
actionpack (6.1.7) lib/action_controller/metal/rendering.rb:52:in `render_to_body'
actionpack (6.1.7) lib/action_controller/metal/renderers.rb:142:in `render_to_body'
actionpack (6.1.7) lib/abstract_controller/rendering.rb:25:in `render'
actionpack (6.1.7) lib/action_controller/metal/rendering.rb:36:in `render'
actionpack (6.1.7) lib/action_controller/metal/instrumentation.rb:46:in `block in render'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/benchmark.rb:308:in `realtime'
activesupport (6.1.7) lib/active_support/core_ext/benchmark.rb:14:in `ms'
actionpack (6.1.7) lib/action_controller/metal/instrumentation.rb:46:in `block in render'
actionpack (6.1.7) lib/action_controller/metal/instrumentation.rb:86:in `cleanup_view_runtime'
actionpack (6.1.7) lib/action_controller/metal/instrumentation.rb:45:in `render'
actionpack (6.1.7) lib/action_controller/metal/mime_responds.rb:214:in `respond_to'
actionpack (6.1.7) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
actionpack (6.1.7) lib/abstract_controller/base.rb:228:in `process_action'
actionpack (6.1.7) lib/action_controller/metal/rendering.rb:30:in `process_action'
actionpack (6.1.7) lib/abstract_controller/callbacks.rb:42:in `block in process_action'
activesupport (6.1.7) lib/active_support/callbacks.rb:106:in `run_callbacks'
actionpack (6.1.7) lib/abstract_controller/callbacks.rb:41:in `process_action'
actionpack (6.1.7) lib/action_controller/metal/rescue.rb:22:in `process_action'
actionpack (6.1.7) lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
activesupport (6.1.7) lib/active_support/notifications.rb:203:in `block in instrument'
activesupport (6.1.7) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (6.1.7) lib/active_support/notifications.rb:203:in `instrument'
actionpack (6.1.7) lib/action_controller/metal/instrumentation.rb:33:in `process_action'
actionpack (6.1.7) lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
actionpack (6.1.7) lib/abstract_controller/base.rb:165:in `process'
actionview (6.1.7) lib/action_view/rendering.rb:39:in `process'
actionpack (6.1.7) lib/action_controller/metal.rb:190:in `dispatch'
actionpack (6.1.7) lib/action_controller/metal.rb:254:in `dispatch'
actionpack (6.1.7) lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
actionpack (6.1.7) lib/action_dispatch/routing/route_set.rb:33:in `serve'
actionpack (6.1.7) lib/action_dispatch/journey/router.rb:50:in `block in serve'
org/jruby/RubyArray.java:1865:in `each'
actionpack (6.1.7) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (6.1.7) lib/action_dispatch/routing/route_set.rb:842:in `call'
rack (2.2.4) lib/rack/tempfile_reaper.rb:15:in `call'
rack (2.2.4) lib/rack/etag.rb:27:in `call'
rack (2.2.4) lib/rack/conditional_get.rb:27:in `call'
rack (2.2.4) lib/rack/head.rb:12:in `call'
actionpack (6.1.7) lib/action_dispatch/http/permissions_policy.rb:22:in `call'
actionpack (6.1.7) lib/action_dispatch/http/content_security_policy.rb:19:in `call'
uri:classloader:/jruby/rack/session_store.rb:79:in `context'
rack (2.2.4) lib/rack/session/abstract/id.rb:260:in `call'
actionpack (6.1.7) lib/action_dispatch/middleware/cookies.rb:689:in `call'
actionpack (6.1.7) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
activesupport (6.1.7) lib/active_support/callbacks.rb:98:in `run_callbacks'
actionpack (6.1.7) lib/action_dispatch/middleware/callbacks.rb:26:in `call'

The text was updated successfully, but these errors were encountered:

chadlwilson · 2022-10-18T16:28:26Z

Perhaps related to SCSS source maps. config.sass.inline_source_maps = true|false seems to make no difference, but maybe disabling config.assets.debug entirely or debug for the specific stylesheet tag is the only way to workaround this on Sprockets 4 based on some of the discussion on rails/sprockets#656 (comment)

ntkme · 2022-10-18T16:57:54Z

Stack trace says it is this line: https://github.com/ntkme/sass-embedded-host-ruby/blob/0b4cf29cad898ca647017a0921b4641359eb46c6/lib/sass/embedded/host.rb#L55

So I think it is related to the css input.

chadlwilson · 2022-10-18T17:02:11Z

Argh, sorry, I pasted a stack trace where I had added a debug log line above in compile-request. So the error is actually on host.rb#54 with the real source. Sorry.

https://github.com/ntkme/sass-embedded-host-ruby/blob/0b4cf29cad898ca647017a0921b4641359eb46c6/lib/sass/embedded/host.rb#L54

ntkme · 2022-10-18T19:34:23Z

@chadlwilson Can you please add a debug line to check the encoding of source with source.encoding and see if that is ASCII-8BIT? If so maybe try source.force_encoding('UTF-8') and see if it makes any difference?

chadlwilson · 2022-10-19T15:12:56Z

@chadlwilson Can you please add a debug line to check the encoding of source with source.encoding and see if that is ASCII-8BIT? If so maybe try source.force_encoding('UTF-8') and see if it makes any difference?

Yes, the source.encoding is indeed ASCII-8BIT and source.force_encoding('UTF-8') does fix the issue.

The problem

After a bit of luck, and a lot of hacking around, I discovered the problematic character is U+00D7 aka × aka Unicode Character 'MULTIPLICATION SIGN', encoded in UTF-8 as 0xC3 0x97.

In the file itself, binary encoding seems correct.
Replacing the single character with something else "fixes" the issue even with debug: true
Making it unambiguous by removing the Unicode encoding, e.g use content: "\00d7"; vs content: "×"; also works around it.
Declaring @charset "UTF-8"; seems to make no difference.

So I conclude that, when assets debug is enabled, somewhere in the stack is causing the SCSS to be loaded or parsed differently, with an ASCII-8BIT default, or some bad guesswork about the file's encoding...

The root problem

Dug into Sprockets and found rails/sprockets#669 - similar problem on JS files whose fix made it into Sprockets 4.1.0... also with sourcemaps which are related to debug: true. The above fix didn't touch the mime type for application/css-sourcemap+json which seems to have the same issue?

https://github.com/rails/sprockets/blob/1276b431e2e4c1099dae1b3ff76adc868c863ddd/lib/sprockets.rb#L93-L94

Without a charset, the default method seems to be to detect the charset, but I'm not quite sure how that is ending up as ASCII-8BIT

The fix

Adding the below at start-up seems to fix the issue.

Sprockets.register_mime_type 'application/css-sourcemap+json', extensions: ['.css.map'], charset: :unicode

I guess I could submit a PR to Sprockets? But not entirely sure I understand all the consequences of that charset: :unicode. Seems a sensible default though....

Again, and this is a little embarrassing - another Sprockets problem mis-attributed to poor sass-embeded-host-ruby!

ntkme · 2022-10-19T21:43:44Z

Given the information you provided, I think the issue is:

You source UTF-8 scss file contains 0xC3 0x97, which is perfectly fine.
Somehow the file is read into a ruby string buffer as is but the encoding is set to ASCII-8BIT.
Protobuf constructor tries to convert ASCII-8BIT to UTF-8 but it fails because the source is actually already UTF-8. - The encode method in ruby convert the string byte buffer from one encode to another, the force_encoding method does not change the string byte buffer, just force the existing buffer to be interpreted.

ntkme · 2022-10-20T04:49:02Z

Reading the test case added in rails/sprockets#669, I think what is happening is that in the dev mode, the encoding registered for the mime type of an output file will determine the encoding set on string for input (input[:data].encoding).

UTF-8 is the default encoding for ruby >=2.0 that it has been around for many years. I think it is a good idea to submit a PR to upstream to change the default for application/css-sourcemap+json to :unicode.

chadlwilson · 2022-10-20T05:49:30Z

Great, thank you for your input 🙏

chadlwilson mentioned this issue Oct 19, 2022

Workaround Sprockets issue with unicode chars inside SCSS and config.assets.debug=true gocd/gocd#10932

Merged

ntkme closed this as completed Oct 20, 2022

chadlwilson mentioned this issue Oct 21, 2022

fix: set application/css-sourcemap+json default charset to unicode rails/sprockets#764

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UndefinedConversionError "\xC3" from ASCII-8BIT to UTF-8 within JRuby/Rails/Sprockets dev mode #71

UndefinedConversionError "\xC3" from ASCII-8BIT to UTF-8 within JRuby/Rails/Sprockets dev mode #71

chadlwilson commented Oct 18, 2022 •

edited

Loading

chadlwilson commented Oct 18, 2022

ntkme commented Oct 18, 2022

chadlwilson commented Oct 18, 2022 •

edited

Loading

ntkme commented Oct 18, 2022 •

edited

Loading

chadlwilson commented Oct 19, 2022 •

edited

Loading

ntkme commented Oct 19, 2022 •

edited

Loading

ntkme commented Oct 20, 2022

chadlwilson commented Oct 20, 2022

UndefinedConversionError "\xC3" from ASCII-8BIT to UTF-8 within JRuby/Rails/Sprockets dev mode #71

UndefinedConversionError "\xC3" from ASCII-8BIT to UTF-8 within JRuby/Rails/Sprockets dev mode #71

Comments

chadlwilson commented Oct 18, 2022 • edited Loading

chadlwilson commented Oct 18, 2022

ntkme commented Oct 18, 2022

chadlwilson commented Oct 18, 2022 • edited Loading

ntkme commented Oct 18, 2022 • edited Loading

chadlwilson commented Oct 19, 2022 • edited Loading

The problem

The root problem

The fix

ntkme commented Oct 19, 2022 • edited Loading

ntkme commented Oct 20, 2022

chadlwilson commented Oct 20, 2022

chadlwilson commented Oct 18, 2022 •

edited

Loading

chadlwilson commented Oct 18, 2022 •

edited

Loading

ntkme commented Oct 18, 2022 •

edited

Loading

chadlwilson commented Oct 19, 2022 •

edited

Loading

ntkme commented Oct 19, 2022 •

edited

Loading