Big v0.2 code dump

Author: JakeBecker

CONTRIBUTING.md

@@ -0,0 +1,7 @@
+While you can develop ElixirLS on its own, it's easiest to test out changes if you clone the [vscode-elixir-ls](https://github.com/JakeBecker/vscode-elixir-ls) repository instead. It includes ElixirLS as a Git submodule, so you can make your changes in the submodule directory and launch the extension from the parent directory via the included "Launch Extension" configuration in `launch.json`. See the README on that repo for more details.
+
+### Building and running
+
+To build for release, run the `release.sh` script. It compiles these two apps to escripts in the `release` folder. If doing a public release, compile it with Erlang OTP 19, since OTP 20 builds are not backwards-compatible with earlier versions of Erlang.
+
+Elixir escripts typically embed Elixir, which is not what we want because users will want to run it against their own system Elixir installation. In order to support this, there are scripts in the `bin` folder that look for the system-wide Elixir installation and set the `ERL_LIBS` environment variable prior to running the escripts. These scripts are copied to the `release` folder by `release.sh`.

README.md

@@ -1,20 +1,32 @@
 # Elixir Language Server (ElixirLS)
 
-*This is a very early release. It only works with Elixir 1.4, and it's only been tested on OSX. If all you want is a good, stable Elixir editor, stick with whatever you're using now until ElixirLS is more mature.*
+The Elixir Language Server provides a server that runs in the background, providing IDEs, editors, and other tools with information about Elixir Mix projects. It adheres to the [Language Server Protocol](https://github.com/Microsoft/language-server-protocol), a standard for frontend-independent IDE support. Debugger integration is accomplished through the similar [VS Code Debug Protocol](https://code.visualstudio.com/docs/extensionAPI/api-debugging).
 
-The Elixir Language Server provides a server that runs in the background, providing IDEs, editors, and other tools with information about Elixir Mix projects. It adheres to the [Language Server Protocol](https://github.com/Microsoft/language-server-protocol), a standard supported by Microsoft and Red Hat for frontend-independent IDE support. Debugger integration is accomplished through the similar [VS Code Debug Protocol](https://code.visualstudio.com/docs/extensionAPI/api-debugging).
+## Features
 
-Features include:
-
-- Debugger support!!!
-- Inline reporting of build warnings and errors
+- Debugger support (requires Erlang >= OTP 19)
+- Automatic, incremental Dialyzer analysis (requires Erlang OTP 20)
+- Inline reporting of build warnings and errors (requires Elixir >= 1.6)
 - Documentation lookup on hover
 - Go-to-definition
 - Code completion
+- Code formatter (requries Elixir >= 1.6)
+
+![Screenshot](images/screenshot.png?raw=true)
+
+## Supported versions
+
+Elixir:
+- 1.3 minimum
+- \>= 1.6.0-dev recommended. Required for reporting of build warnings and errors, and for code formatting support.
 
-![Screenshot](screenshot.png?raw=true)
+Erlang:
+- OTP 18 minimum
+- OTP 20 recommended. >= OTP 19 is required for debugger support, and OTP 20 is recommended for automatic incremental Dialyzer integration.
 
-## IDE support
+You may want to install Elixir and Erlang from source, using the [kiex](https://github.com/taylor/kiex) and [kerl](https://github.com/kerl/kerl) tools. This will let you go-to-definition for core Elixir and Erlang modules. Use `kiex install master` to install a pre-release version of Elixir.
+
+## IDE plugins
 
 ElixirLS is intended to be frontend-independent, but at this point has only been tested with VS Code. You can install the VS Code extension by searching ElixirLS from the "extensions" pane. The plugin code resides in the [vscode-elixir-ls](https://github.com/JakeBecker/vscode-elixir-ls) repo.
 
@@ -43,24 +55,18 @@ In order to debug modules in `.exs` files (such as tests), they must be specifie
 
 ## Automatic builds and error reporting
 
-In order to provide features like documentation look-up and code completion, ElixirLS needs to be able to load your project's compiled modules. ElixirLS attempts to compile your project automatically and reports build errors and warnings in the editor.
-
-At the moment, this compilation is performed using a fork of Elixir 1.4's compiler. This is not a good long-term solution and replacing it is a high priority in the near future. See [this blog post](https://medium.com/@JakeBeckerCode/compiler-hacks-in-elixirls-6a6f04834f66) for an explanation.
-
-To avoid interfering with the developer's CLI workflow, ElixirLS creates a folder `.elixir_ls` in the project root and saves its build output there, so add `.elixir_ls` to your gitignore file.
-
-Note that compiling untrusted Elixir files can be dangerous. Elixir can execute arbitrary code at compile time, which is how it can support such extensive metaprogramming. Consequently, compiling untrusted Elixir code is a lot like executing an untrusted script. Since ElixirLS compiles your code automatically, opening a project in an ElixirLS-enabled editor has the same risks as running `mix compile`. It's an unlikely attack vector, but worth being aware of.
+Builds are performed automatically when files are saved. If you want this to happen automatically when you type, you can turn on "autosave" in your IDE.
 
-## Contributing
+Starting in Elixir 1.6, Mix compilers adhere to the [Mix.Task.Compiler](https://hexdocs.pm/mix/master/Mix.Task.Compiler.html) behaviour and return their error and warning diagnostics in a standardized way. If you're using Elixir >= 1.6, errors and warnings will be shown inline in your code as well as in the "Problems" pane in the IDE. If you're using an earlier version of Elixir, you'll need to look at the text log from the extension to see the errors and warnings.
 
-While you can develop ElixirLS on its own, it's easiest to test out changes if you clone the [vscode-elixir-ls](https://github.com/JakeBecker/vscode-elixir-ls) repository instead. It includes ElixirLS as a Git submodule, so you can make your changes in the submodule directory and launch the extension from the parent directory via the included "Launch Extension" configuration in `launch.json`. See the README on that repo for more details.
+## Dialyzer integration
 
-### Building and running
+If you're using Erlang >= OTP 20, ElixirLS will automatically analyze your project with Dialyzer after each successful build. It maintains a "manifest" file in `.elixir_ls/dialyzer_manifest` that stores the results of the analysis. The initial analysis for a project can take a few minutes, but after that's completed, modules are re-analyzed only if necessary, so subsequent analyses are typically very fast -- often less than a second. It also looks at your modules' abstract code to determine whether they reference any modules that haven't been analyzed and includes them automatically.
 
-You can run the apps with `mix run` from the `apps/language_server` or `apps/debugger` directories. To build for release, run the `release.sh` script. It compiles these two apps to escripts in the `release` folder.
+You can control which warnings are shown using the `elixirLS.dialyzerWarnOpts` setting in your project or IDE's `settings.json`. To disable it completely, set `elixirLS.dialyzerEnabled` to false.
 
-Elixir escripts typically embed Elixir, which is not what we want because users will want to run it against their own system Elixir installation. In order to support this, there are scripts in the `bin` folder that look for the system-wide Elixir installation and set the `ERL_LIBS` environment variable prior to running the escripts. These scripts are copied to the `release` folder by `release.sh`.
+ElixirLS's Dialyzer integration uses internal, undocumented Dialyzer APIs, and so it won't be robust against changes to these APIs in future Erlang versions.
 
 ## Acknowledgements and related projects
 
-ElixirLS isn't the first frontend-independent server for Elixir language support. The original was [Alchemist Server](https://github.com/tonini/alchemist-server/), which powers the [Alchemist](https://github.com/tonini/alchemist.el) plugin for Emacs. Another project, [Elixir Sense](https://github.com/msaraiva/elixir_sense), builds upon Alchemist and powers the [Elixir plugin for Atom](https://github.com/msaraiva/atom-elixir) as well as another VS Code plugin, [VSCode Elixir](https://github.com/fr1zle/vscode-elixir). ElixirLS uses Elixir Sense for several code insight features. Credit for those projects goes to their respective authors.
+ElixirLS isn't the first frontend-independent server for Elixir language support. The original was [Alchemist Server](https://github.com/tonini/alchemist-server/), which powers the [Alchemist](https://github.com/tonini/alchemist.el) plugin for Emacs. Another project, [Elixir Sense](https://github.com/msaraiva/elixir_sense), builds upon Alchemist and powers the [Elixir plugin for Atom](https://github.com/msaraiva/atom-elixir) as well as another VS Code plugin, [VSCode Elixir](https://github.com/fr1zle/vscode-elixir). ElixirLS uses Elixir Sense for several code insight features. Credit for those projects goes to their respective authors.

apps/debugger/lib/debugger.ex

@@ -13,13 +13,7 @@ defmodule ElixirLS.Debugger do
     children = [
       # Define workers and child supervisors to be supervised
       worker(ElixirLS.Debugger.Output, [ElixirLS.Debugger.Output]),
-      worker(ElixirLS.Debugger.OutputDevice,
-             [:user, "stdout", [change_all_gls?: change_all_gls?()]],
-             [id: ElixirLS.Debugger.OutputDevice.Stdout]),
-      worker(ElixirLS.Debugger.OutputDevice, [:standard_error, "stderr"],
-             [id: ElixirLS.Debugger.OutputDevice.Stderr]),
       worker(ElixirLS.Debugger.Server, [[name: ElixirLS.Debugger.Server]]),
-      worker(ElixirLS.IOHandler, [ElixirLS.Debugger.Server, [name: ElixirLS.Debugger.IOHandler]]),
     ]
 
     # See http://elixir-lang.org/docs/stable/elixir/Supervisor.html
@@ -29,10 +23,14 @@ defmodule ElixirLS.Debugger do
   end
 
   def stop(_state) do
-    :init.stop
-  end
+    # If IO is being intercepted (meaning we're running in production), allow time to flush errors
+    # then kill the VM
+    if ElixirLS.Utils.WireProtocol.io_intercepted?() do
+      IO.puts("Stopping ElixirLS debugger due to errors.")
+      :timer.sleep(100)
+      :init.stop(1)
+    end
 
-  defp change_all_gls? do
-    !(Enum.any?(Application.started_applications, &match?({:mix, _, _}, &1)) and Mix.env == :test)
+    :ok
   end
 end

apps/debugger/lib/debugger/cli.ex

@@ -1,8 +1,15 @@
 defmodule ElixirLS.Debugger.CLI do
-  
+  alias ElixirLS.Utils.WireProtocol
+  alias ElixirLS.Debugger.{Output, Server}
+
   def main(_args) do
+    WireProtocol.intercept_output(&Output.print/1, &Output.print_err/1)
+
+    Application.ensure_all_started(:elixir_ls_debugger, :permanent)
+
     Mix.Local.append_archives
     Mix.Local.append_paths
-    :timer.sleep(:infinity)
+
+    WireProtocol.stream_packets(&Server.receive_packet/1)
   end
-end
+end

apps/debugger/lib/debugger/output.ex

@@ -2,18 +2,19 @@ defmodule ElixirLS.Debugger.Output do
   @moduledoc """
   Implements the JSON-based request protocol for VS Code debug adapters.
 
-  VS Code debug protocol specifies that a message is either a request, a response, or an event. 
+  VS Code debug protocol specifies that a message is either a request, a response, or an event.
   All messages must include a sequence number. This server keeps a counter to ensure that messages
   are sent with sequence numbers that are unique and sequential, and includes client functions for
   sending these messages.
   """
+  import ElixirLS.Utils.WireProtocol, only: [send: 1]
   use GenServer
   use ElixirLS.Debugger.Protocol
 
   ## Client API
 
   def start_link(name \\ nil) do
-    GenServer.start_link(__MODULE__, 1, name: name)
+    GenServer.start(__MODULE__, 1, name: name)
   end
 
   def send_response(server \\ __MODULE__, request_packet, response_body) do
@@ -24,19 +25,35 @@ defmodule ElixirLS.Debugger.Output do
     GenServer.call(server, {:send_event, event, body})
   end
 
+  def print(server \\ __MODULE__, str) do
+    send_event(
+      server,
+      "output",
+      %{"category" => "stdout", "output" => to_string(str)}
+    )
+  end
+
+  def print_err(server \\ __MODULE__, str) do
+    send_event(
+      server,
+      "output",
+      %{"category" => "stderr", "output" => to_string(str)}
+    )
+  end
+
   ## Server callbacks
 
   def handle_call({:send_response, request_packet, body}, _from, seq) do
-    ElixirLS.IOHandler.send(response(seq, request_packet["seq"], request_packet["command"], body))
+    send(response(seq, request_packet["seq"], request_packet["command"], body))
     {:reply, :ok, seq + 1}
   end
 
   def handle_call({:send_event, event, body}, _from, seq) do
-    ElixirLS.IOHandler.send(event(seq, event, body))
+    send(event(seq, event, body))
     {:reply, :ok, seq + 1}
   end
 
   def handle_call(message, from, s) do
     super(message, from, s)
   end
-end
+end

apps/debugger/lib/debugger/server.ex

@@ -57,6 +57,7 @@ defmodule ElixirLS.Debugger.Server do
   ## Server Callbacks
 
   def init(opts) do
+    Process.flag(:trap_exit, true)
     :int.start
     state = if opts[:output], do: %__MODULE__{output: opts[:output]}, else: %__MODULE__{}
     {:ok, state}
@@ -97,30 +98,23 @@ defmodule ElixirLS.Debugger.Server do
     {:noreply, %{state | task_ref: nil}}
   end
 
-  def handle_info(msg, state) do
-    super(msg, state)
-  end
-
-  def terminate(:normal, state) do
-    super(:normal, state)
+  def handle_info({:EXIT, _, :normal}, state) do
+    {:noreply, state}
   end
 
-  def terminate(reason, state) do
-    IO.puts :standard_error, "Debug server terminated abnormally because " <>
-        Exception.format_exit(reason)
-    super(reason, state)
+  def handle_info(msg, state) do
+    super(msg, state)
   end
 
   ## Helpers
 
   defp handle_request(initialize_req(_, client_info), state) do
     check_erlang_version()
-    Process.group_leader(self(), Process.whereis(:user))
     {capabilities(), %{state | client_info: client_info}}
   end
 
   defp handle_request(launch_req(_, config), state) do
-    Process.spawn(fn -> initialize(config) end, [:link])
+    initialize(config)
     {%{}, %{state | config: config}}
   end
 
@@ -406,23 +400,23 @@ defmodule ElixirLS.Debugger.Server do
     set_stack_trace_mode(config["stackTraceMode"])
 
     File.cd!(project_dir)
-    Application.ensure_started(:mix)
-    Mix.Local.append_archives
-    Mix.Local.append_paths
-
     Code.load_file(System.get_env("MIX_EXS") || "mix.exs")
     task = task || Mix.Project.config[:default_task]
     unless mix_env, do: change_env(task)
 
-    Mix.Tasks.Loadconfig.run([])
-    Mix.Tasks.Deps.Loadpaths.run([])
-    Mix.Tasks.Loadpaths.run([])
+    Mix.Task.run("loadconfig")
     unless is_list(task_args) and "--no-compile" in task_args do
-      Mix.Tasks.Compile.run([])
+      case Mix.Task.run("compile", ["--ignore-module-conflict"]) do
+        {:error, _} ->
+          IO.puts(:standard_error, "Aborting debugger due to compile errors")
+          :init.stop(1)
+        _ ->
+          :ok
+      end
     end
 
-    # Reenable all tasks in case they're run from the debugged code
-    for task <- Mix.Task.all_modules, do: Mix.Task.reenable(task)
+    Mix.Task.run("app.start", [])
+    # Mix.TasksServer.clear()
 
     interpret_modules_in(Mix.Project.build_path)
 
@@ -501,8 +495,7 @@ defmodule ElixirLS.Debugger.Server do
   end
 
   defp launch_task(task, args) do
-    Process.group_leader(self(), Process.whereis(:user))
-    Mix.Task.rerun(task, args)
+    Mix.Task.run(task, args)
   end
 
   # Interpreting modules defined in .exs files requires that we first load the file and save any

apps/debugger/lib/debugger/variables.ex

@@ -34,13 +34,13 @@ defmodule ElixirLS.Debugger.Variables do
   end
 
   def children(var, start, count) when is_map(var) do
-    children = 
+    children =
       var
       |> Map.to_list
       |> Enum.slice(start || 0, count || map_size(var))
 
     for {key, value} <- children do
-      name = 
+      name =
         if is_atom(key) and not String.starts_with?(to_string(key), "Elixir.") do
           to_string(key)
         else
@@ -53,7 +53,7 @@ defmodule ElixirLS.Debugger.Variables do
   def children(_var, _start, _count) do
     []
   end
-  
+
   def num_children(var) when is_list(var) do
     Enum.count(var)
   end
@@ -66,11 +66,11 @@ defmodule ElixirLS.Debugger.Variables do
     if byte_size(var) > 1, do: byte_size(var), else: 0
   end
 
-  def num_children(var) when is_tuple(var) do 
+  def num_children(var) when is_tuple(var) do
     tuple_size(var)
   end
 
-  def num_children(var) when is_map(var) do 
+  def num_children(var) when is_map(var) do
     map_size(var)
   end
 
@@ -96,8 +96,8 @@ defmodule ElixirLS.Debugger.Variables do
   def type(_), do: "term"
 
   defp with_index_as_name(vars, start) do
-    for {var, idx} <- Enum.with_index(vars, start || 0) do
+    for {var, idx} <- Enum.with_index(vars, start) do
       {"#{idx}", var}
     end
   end
-end
+end

apps/debugger/mix.exs

@@ -3,12 +3,12 @@ defmodule ElixirLS.Debugger.Mixfile do
 
   def project do
     [app: :debugger,
-     version: "0.1.0",
+     version: "0.2.0",
      build_path: "../../_build",
      config_path: "config/config.exs",
      deps_path: "../../deps",
      lockfile: "../../mix.lock",
-     elixir: "~> 1.3",
+     elixir: "~> 1.6.0-dev",
      build_embedded: false,
      start_permanent: true,
      build_per_environment: false,
@@ -22,7 +22,7 @@ defmodule ElixirLS.Debugger.Mixfile do
   # Type "mix help compile.app" for more information
   def application do
     # Specify extra applications you'll use from Erlang/Elixir
-    [mod: {ElixirLS.Debugger, []}, applications: []]
+    [mod: {ElixirLS.Debugger, []}, extra_applications: [:mix, :logger]]
   end
 
   # Dependencies can be Hex packages:
@@ -40,14 +40,15 @@ defmodule ElixirLS.Debugger.Mixfile do
   # Type "mix help deps" for more examples and options
   defp deps do
     [{:elixir_sense, github: "msaraiva/elixir_sense"},
-     {:io_handler, in_umbrella: true}]
+     {:elixir_ls_utils, in_umbrella: true}]
   end
 
   defp escript do
-    [main_module: ElixirLS.Debugger.CLI, 
-     embed_elixir: false, 
+    [main_module: ElixirLS.Debugger.CLI,
+     app: nil,
+     embed_elixir: false,
      path: "../../release/debugger",
-     strip_beam: false, 
+     strip_beam: false,
      comment: escript_comment()]
   end