├── .gitignore ├── .travis.yml ├── LICENSE.md ├── README.md ├── VERSION.md ├── config └── config.exs ├── lib ├── process_helper.ex ├── tracer.ex ├── tracer │ ├── agent_cmds.ex │ ├── clause.ex │ ├── collect.ex │ ├── event.ex │ ├── handler_agent.ex │ ├── matcher.ex │ ├── pid_handler.ex │ ├── probe.ex │ ├── probe_list.ex │ ├── tool.ex │ ├── tool_helper.ex │ ├── tool_server.ex │ └── tools │ │ ├── tool_call_seq.ex │ │ ├── tool_call_seq_event.ex │ │ ├── tool_count.ex │ │ ├── tool_count_event.ex │ │ ├── tool_display.ex │ │ ├── tool_duration.ex │ │ ├── tool_duration_event.ex │ │ └── tool_flame_graph.ex ├── tracer_app.ex ├── tracer_macros.ex ├── tracer_server.ex └── tracer_supervisor.ex ├── mix.exs ├── mix.lock ├── scripts ├── flamegraph.pl └── gen_flame_graph.sh └── test ├── test_helper.exs ├── tracer ├── agent_cmds_test.exs ├── clause_test.exs ├── collect_test.exs ├── handler_agent_test.exs ├── matcher_test.exs ├── pid_handler_test.exs ├── probe_list_test.exs ├── probe_test.exs ├── process_helper_test.exs ├── tool_helper_test.exs ├── tool_test.exs └── tools │ ├── tool_call_seq_test.exs │ └── tool_duration_test.exs ├── tracer_server_test.exs └── tracer_test.exs /.gitignore: -------------------------------------------------------------------------------- 1 | # The directory Mix will write compiled artifacts to. 2 | /_build 3 | 4 | # If you run "mix test --cover", coverage assets end up here. 5 | /cover 6 | 7 | # The directory Mix downloads your dependencies sources to. 8 | /deps 9 | 10 | # Where 3rd-party dependencies like ExDoc output generated docs. 11 | /doc 12 | 13 | # Ignore .fetch files in case you like to edit your project deps locally. 14 | /.fetch 15 | 16 | # If the VM crashes, it generates a dump, let's ignore it too. 17 | erl_crash.dump 18 | 19 | # Also ignore archive artifacts (built via "mix archive.build"). 20 | *.ez 21 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: elixir 2 | elixir: 1.4 3 | otp_release: 4 | - 19.0 5 | sudo: false 6 | before_script: 7 | - mix deps.get --only test 8 | script: 9 | - mix test 10 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Copyright (c) 2017, Gabi Zuniga 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Tracer - Elixir Tracing Framework 2 | 3 | [![Build Status](https://api.travis-ci.org/gabiz/tracer.svg)](https://travis-ci.org/gabiz/tracer) 4 | 5 | **Tracer** is a tracing framework for elixir which features an easy to use high level interface, extensibility and safety for using in production. 6 | 7 | ## Installation 8 | 9 | If you need to integrate **Tracer** to your project, then you can install it from 10 | [Hex](https://hex.pm/packages/tracer), by adding `tracer` to your list of dependencies in `mix.exs`: 11 | 12 | ```elixir 13 | def deps do 14 | [{:tracer, "~> 0.1.1"}] 15 | end 16 | ``` 17 | 18 | To use Tracer from the cli, then download it directly from [GitHub](https://github.com/gabiz/tracer). 19 | 20 | When firing `iex` you might want to specify the node name so that you can trace other nodes remotely. Then enter the `use Tracer` command to be able to use its functions as commands without the `Tracer` prefix. 21 | 22 | ```elixir 23 | $ git clone git@github.com:gabiz/tracer.git 24 | ... 25 | $ cd tracer 26 | 27 | $ mix deps.get 28 | ... 29 | $ iex --name tracer@127.0.0.1 -S mix 30 | Erlang/OTP 19 [erts-8.0] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace] 31 | 32 | Interactive Elixir (1.5.1) - press Ctrl+C to exit (type h() ENTER for help) 33 | iex(tracer@127.0.0.1)1> use Tracer 34 | :ok 35 | iex(tracer@127.0.0.1)2> 36 | nil 37 | iex(tracer@127.0.0.1)3> run Count, node: :"phoenix@127.0.0.1", ... 38 | ``` 39 | 40 | ## Tools 41 | 42 | Tools are tracing components that focus on a specific tracing aspect. They are implemented as Elixir modules so you can create your own tools. 43 | 44 | Tracer currently provides the following tools: 45 | * The `Count` tool counts events. 46 | * The `Duration` tool measures how long it takes to execute a function. 47 | * The `CallSeq` - 'Call Sequence' tool displays function call sequences. 48 | * The `FlameGraph` tool which aggregates stack frames over a flame graph. 49 | * The `Display` tool displays standard tracing events. 50 | 51 | ## Count Tool Example 52 | 53 | ```elixir 54 | iex(2)> run Count, process: self(), match: global String.split(string, pattern) 55 | started tracing 56 | :ok 57 | iex(3)> 58 | nil 59 | iex(4)> String.split("Hello World", " ") 60 | ["Hello", "World"] 61 | iex(5)> String.split("Hello World", " ") 62 | ["Hello", "World"] 63 | iex(6)> String.split("Hello World", "o") 64 | ["Hell", " W", "rld"] 65 | iex(7)> String.split("Hello", "o") 66 | ["Hell", ""] 67 | iex(8)> done tracing: tracing_timeout 30000 68 | 1 [string:"Hello World", pattern:"o"] 69 | 1 [string:"Hello" , pattern:"o" ] 70 | 2 [string:"Hello World", pattern:" "] 71 | ``` 72 | 73 | ## Duration Tool Example 74 | 75 | ```elixir 76 | iex(1)> run Duration, match: global Map.new(param) 77 | started tracing 78 | :ok 79 | iex(2)> Map.new(%{a: 1}) 80 | 4 '#PID<0.151.0>' Map.new/1 [param: %{a: 1}] 81 | %{a: 1} 82 | iex(3)> Map.new(%{b: 2}) 83 | 3 '#PID<0.151.0>' Map.new/1 [param: %{b: 2}] 84 | %{b: 2} 85 | iex(4)> Map.new(%{c: [1, 2,3]}) 86 | 6 '#PID<0.151.0>' Map.new/1 [param: %{c: [1, 2, 3]}] 87 | %{c: [1, 2, 3]} 88 | iex(5)> stop 89 | :ok 90 | done tracing: :stop_command 91 | ``` 92 | 93 | Use `aggregation` option to collect all the duration samples and return you a combined result. 94 | `aggregation:` option can be one of `:sum`, `:avg`, `:min`, `:max`, `:dist` 95 | 96 | ## Call Sequence Tool Example 97 | 98 | ```elixir 99 | iex(1)> run CallSeq, show_args: true, show_return: true, start_match: &Map.drop/2, 100 | max_message_count: 10000, max_queue_size: 10000 101 | started tracing 102 | :ok 103 | iex(2)> Map.drop(%{a: 1, b: 2, c: 3}, [:a, :b]) 104 | %{c: 3} 105 | iex(3)> stop 106 | :ok 107 | done tracing: :stop_command 108 | 109 | -> Map.drop/2 [[%{a: 1, b: 2, c: 3}, [:a, :b]]] 110 | -> Enum.to_list/1 [[[:a, :b]]] 111 | <- Enum.to_list/1 [:a, :b] 112 | -> Map.drop_list/2 [[[:a, :b], %{a: 1, b: 2, c: 3}]] 113 | -> :maps.remove/2 [[:a, %{a: 1, b: 2, c: 3}]] 114 | <- :maps.remove/2 %{b: 2, c: 3} 115 | -> Map.drop_list/2 [[[:b], %{b: 2, c: 3}]] 116 | -> :maps.remove/2 [[:b, %{b: 2, c: 3}]] 117 | <- :maps.remove/2 %{c: 3} 118 | -> Map.drop_list/2 [[[], %{c: 3}]] 119 | <- Map.drop_list/2 %{c: 3} 120 | <- Map.drop/2 %{c: 3} 121 | -> :erl_eval.ret_expr/3 [[%{c: 3}, [], :none]] 122 | <- :erl_eval.ret_expr/3 {:value, %{c: 3}, []} 123 | <- :erl_eval.do_apply/6 {:value, %{c: 3}, []} 124 | <- :erl_eval.expr/5 {:value, %{c: 3}, []} 125 | ``` 126 | 127 | ## Flame Graph Tool Example 128 | 129 | ```elixir 130 | iex(17)> run FlameGraph, node: :"phoenix@127.0.0.1", process: SampleApp.Endpoint, 131 | max_message_count: 10000, max_queue_size: 10000, file_name: "phoenix.svg", 132 | ignore: "sleep", resolution: 10, max_depth: 100 133 | started tracing 134 | :ok 135 | iex(18)> stop 136 | :ok 137 | done tracing: :stop_command 138 | ``` 139 | 140 | [Click here (not image) for interactive SVG Flame Graph](https://s3.amazonaws.com/gapix/flame_graph.svg) 141 | 142 | ![Flame Graph](https://s3.amazonaws.com/gapix/flame_graph2.svg?sanitize=true) 143 | 144 | ## Building your own Tool 145 | 146 | Tools have a similar structure like GenServers. 147 | 148 | ```elixir 149 | defmodule MyTool do 150 | alias __MODULE__ 151 | alias Tracer.Probe 152 | use Tracer.Tool 153 | 154 | # store your tool's state 155 | defstruct [] 156 | 157 | def init(opts) do 158 | # init_tool initializes the tool 159 | init_state = init_tool(%MyTool{}, opts, [:match]) 160 | 161 | case Keyword.get(opts, :match) do 162 | nil -> init_state 163 | matcher -> 164 | type = Keyword.get(opts, :type, :call) 165 | probe = Probe.new(type: type, 166 | process: get_process, 167 | match: matcher) 168 | set_probes(init_state, [probe]) 169 | end 170 | end 171 | 172 | # Called when the tool run starts 173 | def handle_start(event, state) do 174 | state 175 | end 176 | 177 | # Called when a trace event triggers 178 | def handle_event(event, state) do 179 | # report event will call to_string(event) to format 180 | # your event, so you can create your own events 181 | report_event(state, event) 182 | state 183 | end 184 | 185 | # Called when the tool run completes 186 | def handle_end(event, state) do 187 | state 188 | end 189 | end 190 | ``` 191 | -------------------------------------------------------------------------------- /VERSION.md: -------------------------------------------------------------------------------- 1 | 0.1.1 2 | -------------------------------------------------------------------------------- /config/config.exs: -------------------------------------------------------------------------------- 1 | # This file is responsible for configuring your application 2 | # and its dependencies with the aid of the Mix.Config module. 3 | use Mix.Config 4 | 5 | # This configuration is loaded before any dependency and is restricted 6 | # to this project. If another project depends on this project, this 7 | # file won't be loaded nor affect the parent project. For this reason, 8 | # if you want to provide default values for your application for 9 | # 3rd-party users, it should be done in your "mix.exs" file. 10 | 11 | # You can configure for your application as: 12 | # 13 | # config :tracer, key: :value 14 | # 15 | # And access this configuration in your application as: 16 | # 17 | # Application.get_env(:tracer, :key) 18 | # 19 | # Or configure a 3rd-party app: 20 | # 21 | # config :logger, level: :info 22 | # 23 | 24 | # It is also possible to import configuration files, relative to this 25 | # directory. For example, you can emulate configuration per environment 26 | # by uncommenting the line below and defining dev.exs, test.exs and such. 27 | # Configuration from the imported file will override the ones defined 28 | # here (which is why it is important to import them last). 29 | # 30 | # import_config "#{Mix.env}.exs" 31 | -------------------------------------------------------------------------------- /lib/process_helper.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.ProcessHelper do 2 | @moduledoc """ 3 | Implements helper functions to find OTP process hierarchy 4 | """ 5 | 6 | @process_keywords [:all, :processes, :ports, :existing, :existing_processes, 7 | :existing_ports, :new, :new_processes] 8 | 9 | # ensure_pid 10 | @process_keywords |> Enum.each(fn keyword -> 11 | def ensure_pid(unquote(keyword)), do: unquote(keyword) 12 | end) 13 | def ensure_pid(pid) when is_pid(pid), do: pid 14 | def ensure_pid(name) when is_atom(name) do 15 | case Process.whereis(name) do 16 | nil -> 17 | raise ArgumentError, 18 | message: "#{inspect name} is not a registered process" 19 | pid when is_pid(pid) -> pid 20 | end 21 | end 22 | 23 | def ensure_pid(pid, nil), do: ensure_pid(pid) 24 | @process_keywords |> Enum.each(fn keyword -> 25 | def ensure_pid(unquote(keyword), _node), do: unquote(keyword) 26 | end) 27 | def ensure_pid(pid, _node) when is_pid(pid), do: pid 28 | def ensure_pid(name, node) when is_atom(name) do 29 | case :rpc.call(node, Process, :whereis, [name]) do 30 | nil -> 31 | raise ArgumentError, 32 | message: "#{inspect name} is not a registered process" 33 | pid when is_pid(pid) -> pid 34 | end 35 | end 36 | 37 | # type 38 | @process_keywords |> Enum.each(fn keyword -> 39 | def type(unquote(keyword)), do: :keyword 40 | end) 41 | def type(pid) do 42 | dict = pid 43 | |> ensure_pid() 44 | |> Process.info() 45 | |> Keyword.get(:dictionary) 46 | 47 | case dict do 48 | [] -> 49 | :regular 50 | _ -> 51 | case Keyword.get(dict, :"$initial_call") do 52 | {:supervisor, _, _} -> :supervisor 53 | {_, :init, _} -> :worker 54 | _ -> :regular 55 | end 56 | end 57 | end 58 | 59 | def type(pid, nil), do: type(pid) 60 | @process_keywords |> Enum.each(fn keyword -> 61 | def type(unquote(keyword), _node), do: :keyword 62 | end) 63 | def type(pid, node) do 64 | dict = pid 65 | |> ensure_pid(node) 66 | |> process_info_on_node(node) 67 | |> Keyword.get(:dictionary) 68 | 69 | case dict do 70 | [] -> 71 | :regular 72 | _ -> 73 | case Keyword.get(dict, :"$initial_call") do 74 | {:supervisor, _, _} -> :supervisor 75 | {_, :init, _} -> :worker 76 | _ -> :regular 77 | end 78 | end 79 | end 80 | 81 | # find_children 82 | @process_keywords |> Enum.each(fn keyword -> 83 | def find_children(unquote(keyword)), do: [] 84 | end) 85 | def find_children(pid) do 86 | pid = ensure_pid(pid) 87 | case type(pid) do 88 | :supervisor -> 89 | child_spec = Supervisor.which_children(pid) 90 | Enum.reduce(child_spec, [], fn 91 | {_mod, pid, _type, _params}, acc when is_pid(pid) -> acc ++ [pid] 92 | _, acc -> acc 93 | end) 94 | _ -> [] 95 | end 96 | end 97 | 98 | def find_children(pid, nil), do: find_children(pid) 99 | @process_keywords |> Enum.each(fn keyword -> 100 | def find_children(unquote(keyword), _node), do: [] 101 | end) 102 | def find_children(pid, node) do 103 | pid = ensure_pid(pid, node) 104 | case type(pid, node) do 105 | :supervisor -> 106 | child_spec = which_children_on_node(pid, node) 107 | Enum.reduce(child_spec, [], fn 108 | {_mod, pid, _type, _params}, acc when is_pid(pid) -> acc ++ [pid] 109 | _, acc -> acc 110 | end) 111 | _ -> [] 112 | end 113 | end 114 | 115 | # find_all_children 116 | @process_keywords |> Enum.each(fn keyword -> 117 | def find_all_children(unquote(keyword)), do: [] 118 | end) 119 | def find_all_children(pid) do 120 | pid = ensure_pid(pid) 121 | case type(pid) do 122 | :supervisor -> 123 | find_all_supervisor_children([pid], []) 124 | _ -> [] 125 | end 126 | end 127 | 128 | def find_all_supervisor_children([], acc), do: acc 129 | def find_all_supervisor_children([sup | sups], pids) do 130 | {s, p} = sup 131 | |> Supervisor.which_children() 132 | |> Enum.reduce({[], []}, fn 133 | {_mod, pid, :supervisor, _params}, {s, p} when is_pid(pid) -> 134 | {s ++ [pid], p ++ [pid]} 135 | {_mod, pid, _type, _params}, {s, p} when is_pid(pid) -> 136 | {s, p ++ [pid]} 137 | _, acc -> acc 138 | end) 139 | find_all_supervisor_children(sups ++ s, pids ++ p) 140 | end 141 | 142 | def find_all_children(pid, nil), do: find_all_children(pid) 143 | @process_keywords |> Enum.each(fn keyword -> 144 | def find_all_children(unquote(keyword), _node), do: [] 145 | end) 146 | def find_all_children(pid, node) do 147 | pid = ensure_pid(pid, node) 148 | case type(pid, node) do 149 | :supervisor -> 150 | find_all_supervisor_children([pid], [], node) 151 | _ -> [] 152 | end 153 | end 154 | 155 | def find_all_supervisor_children([], acc, node), do: acc 156 | def find_all_supervisor_children([sup | sups], pids, node) do 157 | {s, p} = sup 158 | |> which_children_on_node(node) 159 | |> Enum.reduce({[], []}, fn 160 | {_mod, pid, :supervisor, _params}, {s, p} when is_pid(pid) -> 161 | {s ++ [pid], p ++ [pid]} 162 | {_mod, pid, _type, _params}, {s, p} when is_pid(pid) -> 163 | {s, p ++ [pid]} 164 | _, acc -> acc 165 | end) 166 | find_all_supervisor_children(sups ++ s, pids ++ p, node) 167 | end 168 | 169 | def process_info_on_node(pid, node) do 170 | :rpc.call(node, Process, :info, [pid]) 171 | end 172 | 173 | def which_children_on_node(pid, node) do 174 | :rpc.call(node, Supervisor, :which_children, [pid]) 175 | end 176 | end 177 | -------------------------------------------------------------------------------- /lib/tracer.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer do 2 | @moduledoc """ 3 | **Tracer** is a tracing framework for elixir which features an easy to use high level interface, extensibility and safety for using in production. 4 | 5 | To run a tool use the `run` command. Tracing only happens when the tool is running. 6 | All tools accept the following parameters: 7 | * `node: node_name` - Option to run the tool remotely. 8 | * `max_tracing_time: time` - Maximum time to run tool (30sec). 9 | * `max_message_count: count` - Maximum number of events (1000) 10 | * `max_queue_size: size` - Maximum message queue size (1000) 11 | * `process: pid` - Process to trace, also accepts regigered names, 12 | and :all, :existing, :new 13 | * `forward_pid: pid` - Forward results as messages insted of printing 14 | to the display. 15 | ## Examples 16 | ``` 17 | iex> run Count, process: self(), match: global String.split(string, pattern) 18 | :ok 19 | 20 | iex> String.split("Hello World", " ") 21 | ["Hello", "World"] 22 | 23 | iex> String.split("Hello World", " ") 24 | ["Hello", "World"] 25 | 26 | iex String.split("Hello World", "o") 27 | ["Hell", " W", "rld"] 28 | 29 | iex> String.split("Hello", "o") 30 | ["Hell", ""] 31 | 32 | iex> stop 33 | :ok 34 | 35 | 1 [string:"Hello World", pattern:"o"] 36 | 1 [string:"Hello" , pattern:"o" ] 37 | 2 [string:"Hello World", pattern:" "] 38 | ``` 39 | """ 40 | alias Tracer.{Server, Probe, Tool} 41 | import Tracer.Macros 42 | defmacro __using__(_opts) do 43 | quote do 44 | import Tracer 45 | import Tracer.Matcher 46 | alias Tracer.{Tool, Probe, Clause} 47 | alias Tracer.Tool.{Display, Count, CallSeq, Duration, FlameGraph} 48 | :ok 49 | end 50 | end 51 | 52 | delegate :start_server, to: Server, as: :start 53 | delegate :stop_server, to: Server, as: :stop 54 | delegate :stop, to: Server, as: :stop_tool 55 | delegate_1 :set_tool, to: Server, as: :set_tool 56 | 57 | def probe(params) do 58 | Probe.new(params) 59 | end 60 | 61 | def probe(type, params) do 62 | Probe.new([type: type] ++ params) 63 | end 64 | 65 | def tool(type, params) do 66 | Tool.new(type, params) 67 | end 68 | 69 | @doc """ 70 | Runs a tool. Tracing only happens when the tool is running. 71 | * `tool_name` - The name of the tool that want to run. 72 | * `node: node_name` - Option to run the tool remotely. 73 | * `max_tracing_time: time` - Maximum time to run tool (30sec). 74 | * `max_message_count: count` - Maximum number of events (1000) 75 | * `max_queue_size: size` - Maximum message queue size (1000) 76 | * `process: pid` - Process to trace, also accepts regigered names, 77 | and :all, :existing, :new 78 | * `forward_pid: pid` - Forward results as messages insted of printing 79 | to the display. 80 | ## Examples 81 | ``` 82 | iex> run Count, process: self(), match: global String.split(string, pattern) 83 | :ok 84 | 85 | iex> String.split("Hello World", " ") 86 | ["Hello", "World"] 87 | 88 | iex> String.split("Hello World", " ") 89 | ["Hello", "World"] 90 | 91 | iex> String.split("Hello World", "o") 92 | ["Hell", " W", "rld"] 93 | 94 | iex> String.split("Hello", "o") 95 | ["Hell", ""] 96 | 97 | iex> stop 98 | :ok 99 | 100 | 1 [string:"Hello World", pattern:"o"] 101 | 1 [string:"Hello" , pattern:"o" ] 102 | 2 [string:"Hello World", pattern:" "] 103 | ``` 104 | """ 105 | def run(%{"__tool__": _} = tool) do 106 | Server.start_tool(tool) 107 | end 108 | def run(tool_name, params) do 109 | Server.start_tool(tool(tool_name, params)) 110 | end 111 | 112 | end 113 | -------------------------------------------------------------------------------- /lib/tracer/agent_cmds.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.AgentCmds do 2 | @moduledoc """ 3 | Tracer manages a tracing session 4 | """ 5 | alias Tracer.{Probe, ProbeList, HandlerAgent} 6 | 7 | # applies trace directly without handler_agent 8 | def run(_, flags \\ []) 9 | def run(probes, flags) when is_list(probes) do 10 | with :ok <- ProbeList.valid?(probes) do 11 | Enum.map(probes, fn p -> Probe.apply(p, flags) end) 12 | end 13 | end 14 | def run(_, _), do: {:error, :invalid_argument} 15 | 16 | def stop_run do 17 | :erlang.trace(:all, false, [:all]) 18 | end 19 | 20 | def start(nodes, probes, flags) do 21 | tracer_pid = Keyword.get(flags, :forward_pid, self()) 22 | 23 | start_cmds = get_start_cmds(probes) 24 | stop_cmds = get_stop_cmds(probes) 25 | 26 | optional_keys = [:max_tracing_time, 27 | :max_message_count, 28 | :max_queue_size] 29 | agent_flags = [start_trace_cmds: start_cmds, 30 | stop_trace_cmds: stop_cmds, 31 | forward_pid: tracer_pid] ++ 32 | Enum.filter(flags, 33 | fn {key, _} -> Enum.member?(optional_keys, key) end) 34 | 35 | case nodes do 36 | nil -> 37 | [HandlerAgent.start(agent_flags)] 38 | nodes when is_list(nodes) -> 39 | Enum.map(nodes, fn n -> 40 | HandlerAgent.start([node: n] ++ agent_flags) 41 | end) 42 | node -> 43 | [HandlerAgent.start([node: node] ++ agent_flags)] 44 | end 45 | end 46 | 47 | def stop(agent_pids) do 48 | agent_pids 49 | |> Enum.each(fn agent_pid -> 50 | send agent_pid, :stop 51 | end) 52 | :ok 53 | end 54 | 55 | def get_start_cmds(probes, flags \\ []) do 56 | with :ok <- ProbeList.valid?(probes) do 57 | Enum.flat_map(probes, &Probe.get_trace_cmds(&1, flags)) 58 | else 59 | error -> raise RuntimeError, message: "invalid trace #{inspect error}" 60 | end 61 | end 62 | 63 | def get_stop_cmds(_tracer) do 64 | [ 65 | [ 66 | fun: &:erlang.trace/3, 67 | pid_port_spec: :all, 68 | how: :false, 69 | flag_list: [:all] 70 | ], 71 | 72 | [ 73 | fun: &:erlang.trace_pattern/3, 74 | mfa: {:'_', :'_', :'_'}, 75 | match_specs: false, 76 | flag_list: [:local, :call_count, :call_time] 77 | ], 78 | 79 | [ 80 | fun: &:erlang.trace_pattern/3, 81 | mfa: {:'_', :'_', :'_'}, 82 | match_specs: false, 83 | flag_list: [:global] 84 | ] 85 | ] 86 | end 87 | 88 | end 89 | -------------------------------------------------------------------------------- /lib/tracer/clause.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Clause do 2 | @moduledoc """ 3 | Manages a Probe's clause 4 | """ 5 | alias __MODULE__ 6 | require Tracer.Matcher 7 | 8 | @valid_flags [:global, :local, :meta, :call_count, :call_time] 9 | 10 | defstruct type: nil, 11 | mfa: nil, 12 | match_specs: [], 13 | flags: [], 14 | desc: "unavailable", 15 | matches: 0 16 | 17 | def new do 18 | %Clause{} 19 | end 20 | 21 | def get_type(clause) do 22 | clause.type 23 | end 24 | 25 | def set_flags(clause, flags) do 26 | with :ok <- valid_flags?(flags) do 27 | put_in(clause.flags, flags) 28 | end 29 | end 30 | 31 | def get_flags(clause) do 32 | clause.flags 33 | end 34 | 35 | def set_desc(clause, desc) do 36 | put_in(clause.desc, desc) 37 | end 38 | 39 | def get_desc(clause) do 40 | clause.desc 41 | end 42 | 43 | def put_mfa(clause, m \\ :_, f \\ :_, a \\ :_) 44 | def put_mfa(clause, m, f, a) 45 | when is_atom(m) and is_atom(f) and (is_atom(a) or is_integer(a)) do 46 | clause 47 | |> Map.put(:mfa, {m, f, a}) 48 | |> Map.put(:type, :call) 49 | end 50 | def put_mfa(_clause, _, _, _) do 51 | {:error, :invalid_mfa} 52 | end 53 | 54 | def put_fun(clause, fun) when is_function(fun) do 55 | case :erlang.fun_info(fun, :type) do 56 | {:type, :external} -> 57 | with {m, f, a} <- to_mfa(fun) do 58 | put_mfa(clause, m, f, a) 59 | end 60 | _ -> 61 | {:error, "#{inspect(fun)} is not an external fun"} 62 | end 63 | end 64 | 65 | def get_mfa(clause) do 66 | clause.mfa 67 | end 68 | 69 | def add_matcher(clause, matcher) do 70 | put_in(clause.match_specs, matcher ++ clause.match_specs) 71 | end 72 | 73 | def filter(clause, [by: matcher]) do 74 | put_in(clause.match_specs, matcher ++ clause.match_specs) 75 | end 76 | 77 | def matches(clause) do 78 | clause.matches 79 | end 80 | 81 | def valid?(clause) do 82 | with :ok <- validate_mfa(clause) do 83 | :ok 84 | end 85 | end 86 | 87 | def apply(clause, not_remove \\ true) do 88 | with :ok <- valid?(clause) do 89 | res = :erlang.trace_pattern(clause.mfa, 90 | not_remove && clause.match_specs, 91 | clause.flags) 92 | if not_remove == false do 93 | put_in(clause.matches, 0) 94 | else 95 | if is_integer(res), do: put_in(clause.matches, res), else: clause 96 | end 97 | end 98 | end 99 | 100 | def get_trace_cmd(clause, not_remove \\ true) do 101 | with :ok <- valid?(clause) do 102 | [ 103 | fun: &:erlang.trace_pattern/3, 104 | mfa: clause.mfa, 105 | match_spec: not_remove && clause.match_specs, 106 | flag_list: clause.flags 107 | ] 108 | else 109 | error -> raise RuntimeError, message: "invalid clause #{inspect error}" 110 | end 111 | end 112 | 113 | defp validate_mfa(clause) do 114 | case clause.mfa do 115 | nil -> {:error, :missing_mfa} 116 | {m, f, a} 117 | when is_atom(m) and is_atom(f) and (is_atom(a) or is_integer(a)) -> :ok 118 | _ -> {:error, :invalid_mfa} 119 | end 120 | end 121 | 122 | def valid_flags?(flags) when is_list(flags) do 123 | with [] <- Enum.reduce(flags, [], fn f, acc -> 124 | if valid_flag?(f), do: acc, else: [{:invalid_clause_flag, f} | acc] 125 | end) do 126 | :ok 127 | else 128 | error_list -> {:error, error_list} 129 | end 130 | end 131 | def valid_flags?(flag) do 132 | if valid_flag?(flag), do: :ok, else: {:error, :invalid_clause_flag} 133 | end 134 | 135 | defp valid_flag?(flag) do 136 | Enum.member?(@valid_flags, flag) 137 | end 138 | 139 | defp to_mfa(fun) do 140 | with {:module, m} <- :erlang.fun_info(fun, :module), 141 | {:name, f} <- :erlang.fun_info(fun, :name), 142 | {:arity, a} <- :erlang.fun_info(fun, :arity) do 143 | {m, f, a} 144 | else 145 | _ -> {:error, :invalid_mfa} 146 | end 147 | end 148 | 149 | end 150 | -------------------------------------------------------------------------------- /lib/tracer/collect.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Collect do 2 | @moduledoc """ 3 | Collects samples in a map 4 | """ 5 | alias __MODULE__ 6 | 7 | defstruct collections: %{} 8 | 9 | def new do 10 | %Collect{} 11 | end 12 | 13 | def add_sample(state, key, value) do 14 | collection = [value | Map.get(state.collections, key, [])] 15 | put_in(state.collections, Map.put(state.collections, key, collection)) 16 | end 17 | 18 | def get_collections(state) do 19 | Enum.map(state.collections, fn {key, value} -> 20 | {key, Enum.reverse(value)} 21 | end) 22 | end 23 | end 24 | -------------------------------------------------------------------------------- /lib/tracer/event.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Event do 2 | @moduledoc """ 3 | Defines a generic event 4 | """ 5 | 6 | defstruct event: nil 7 | 8 | # def to_string(event) do 9 | # "#{inspect event}" 10 | # end 11 | 12 | defimpl String.Chars, for: Tracer.Event do 13 | def to_string(event) do 14 | "#{inspect event}" 15 | end 16 | end 17 | 18 | def format_ts(ts) do 19 | {{year, month, day}, {hour, minute, second}} = :calendar.now_to_datetime(ts) 20 | "#{inspect month}/#{inspect day}/#{inspect year}-" <> 21 | "#{inspect hour}:#{inspect minute}:#{inspect second}" 22 | end 23 | end 24 | 25 | defmodule Tracer.EventCall do 26 | @moduledoc """ 27 | Defines a call event 28 | """ 29 | alias Tracer.Event 30 | 31 | defstruct mod: nil, fun: nil, arity: nil, 32 | pid: nil, 33 | message: nil, 34 | ts: nil 35 | 36 | def tag, do: :call 37 | 38 | defimpl String.Chars, for: Tracer.EventCall do 39 | def to_string(event) do 40 | "#{Event.format_ts event.ts}: #{inspect event.pid} >> " <> 41 | "#{inspect event.mod}.#{Atom.to_string(event.fun)}/#{inspect event.arity} " <> 42 | if event.message != nil, do: " #{inspect format_message(event.message)}", 43 | else: "" 44 | end 45 | 46 | defp format_message(term) when is_list(term) do 47 | term 48 | |> Enum.map(fn 49 | [key, val] -> {key, val} 50 | other -> other 51 | end) 52 | end 53 | end 54 | end 55 | 56 | defmodule Tracer.EventReturnTo do 57 | @moduledoc """ 58 | Defines an return_to event 59 | """ 60 | alias Tracer.Event 61 | 62 | defstruct mod: nil, fun: nil, arity: nil, 63 | pid: nil, 64 | ts: nil 65 | 66 | def tag, do: :return_to 67 | 68 | defimpl String.Chars, for: Tracer.EventReturnTo do 69 | def to_string(event) do 70 | "#{Event.format_ts event.ts}: #{inspect event.pid} << " <> 71 | "#{inspect event.mod}.#{inspect event.fun}/#{inspect event.arity} " <> 72 | "return_to" 73 | end 74 | end 75 | end 76 | 77 | defmodule Tracer.EventReturnFrom do 78 | @moduledoc """ 79 | Defines a return_from event 80 | """ 81 | alias Tracer.Event 82 | 83 | defstruct mod: nil, fun: nil, arity: nil, 84 | pid: nil, 85 | return_value: nil, 86 | ts: nil 87 | 88 | def tag, do: :return_from 89 | 90 | defimpl String.Chars, for: Tracer.EventReturnFrom do 91 | def to_string(event) do 92 | "#{Event.format_ts event.ts}: #{inspect event.pid} << " <> 93 | "#{inspect event.mod}.#{inspect event.fun}/#{inspect event.arity} " <> 94 | "-> #{inspect event.return_value}" 95 | end 96 | end 97 | end 98 | 99 | defmodule Tracer.EventIn do 100 | @moduledoc """ 101 | Defines an in event 102 | """ 103 | alias Tracer.Event 104 | 105 | defstruct mod: nil, fun: nil, arity: nil, 106 | pid: nil, 107 | ts: nil 108 | 109 | def tag, do: :in 110 | 111 | defimpl String.Chars, for: Tracer.EventIn do 112 | def to_string(event) do 113 | "#{Event.format_ts event.ts}: #{inspect event.pid} In " <> 114 | "#{inspect event.mod}.#{inspect event.fun}/#{inspect event.arity} " 115 | end 116 | end 117 | end 118 | 119 | defmodule Tracer.EventOut do 120 | @moduledoc """ 121 | Defines an out event 122 | """ 123 | alias Tracer.Event 124 | 125 | defstruct mod: nil, fun: nil, arity: nil, 126 | pid: nil, 127 | ts: nil 128 | 129 | def tag, do: :out 130 | 131 | defimpl String.Chars, for: Tracer.EventOut do 132 | def to_string(event) do 133 | "#{Event.format_ts event.ts}: #{inspect event.pid} Out " <> 134 | "#{inspect event.mod}.#{inspect event.fun}/#{inspect event.arity} " 135 | end 136 | end 137 | end 138 | -------------------------------------------------------------------------------- /lib/tracer/handler_agent.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.HandlerAgent do 2 | @moduledoc """ 3 | HandlerAgent takes care of starting and stopping traces in the 4 | NUT (node under test), as well as watching over the event handler 5 | as it processes events. 6 | """ 7 | alias __MODULE__ 8 | alias Tracer.PidHandler 9 | import Tracer.Macros 10 | 11 | @default_max_tracing_time 30_000 12 | 13 | defstruct node: nil, 14 | handler_pid: nil, 15 | timer_ref: nil, 16 | max_tracing_time: @default_max_tracing_time, 17 | pid_handler_opts: [], 18 | start_trace_cmds: [], 19 | stop_trace_cmds: [] 20 | 21 | def start(opts \\ []) do 22 | initial_state = process_opts(%HandlerAgent{}, opts) 23 | pid = spawn_in_target(initial_state) 24 | send pid, :start 25 | pid 26 | end 27 | 28 | def stop(pid) do 29 | send pid, :stop 30 | end 31 | 32 | defp process_opts(state, opts) do 33 | state 34 | |> Map.put(:node, Keyword.get(opts, :node, nil)) 35 | |> Map.put(:start_trace_cmds, Keyword.get(opts, :start_trace_cmds, [])) 36 | |> Map.put(:stop_trace_cmds, Keyword.get(opts, :stop_trace_cmds, [])) 37 | |> assign_to(state) 38 | 39 | state = if Keyword.get(opts, :max_tracing_time) != nil do 40 | put_in(state.max_tracing_time, Keyword.get(opts, :max_tracing_time)) 41 | else 42 | state 43 | end 44 | 45 | state = if Keyword.get(opts, :max_message_count) != nil do 46 | put_in(state.pid_handler_opts, 47 | [{:max_message_count, Keyword.get(opts, :max_message_count)} 48 | | state.pid_handler_opts]) 49 | else 50 | state 51 | end 52 | 53 | state = if Keyword.get(opts, :max_queue_size) != nil do 54 | put_in(state.pid_handler_opts, 55 | [{:max_queue_size, Keyword.get(opts, :max_queue_size)} 56 | | state.pid_handler_opts]) 57 | else 58 | state 59 | end 60 | 61 | event_callback = 62 | if Keyword.get(opts, :forward_pid) != nil do 63 | {:event_callback, {&__MODULE__.forwarding_handler_callback/2, 64 | Keyword.get(opts, :forward_pid)}} 65 | else 66 | {:event_callback, &__MODULE__.discard_handler_callback/1} 67 | end 68 | 69 | if Keyword.get(opts, :event_callback) != nil do 70 | put_in(state.pid_handler_opts, 71 | [{:event_callback, Keyword.get(opts, :event_callback)} 72 | | state.pid_handler_opts]) 73 | else 74 | put_in(state.pid_handler_opts, 75 | [event_callback | state.pid_handler_opts]) 76 | end 77 | end 78 | 79 | defp spawn_in_target(state) do 80 | if state.node != nil do 81 | [__MODULE__, Tracer.PidHandler] |> Enum.each(fn mod -> 82 | ensure_loaded_remote(state.node, mod) 83 | end) 84 | Node.spawn_link(state.node, fn -> process_loop(state) end) 85 | else 86 | spawn_link(fn -> process_loop(state) end) 87 | end 88 | end 89 | 90 | defp process_loop(state) do 91 | receive do 92 | :start -> 93 | Process.flag(:trap_exit, true) 94 | state 95 | |> start_handler() 96 | |> stop_tracing() 97 | |> start_tracing() 98 | |> start_timer() 99 | |> process_loop() 100 | {:timeout, _timeref, _} -> 101 | stop_tracing_and_handler(state) 102 | exit({:done_tracing, :tracing_timeout, state.max_tracing_time}) 103 | :stop -> 104 | stop_tracing_and_handler(state) 105 | exit({:done_tracing, :stop_command}) 106 | {:EXIT, _, :normal} -> # we should be dead by the time this is sent 107 | exit(:normal) 108 | {:EXIT, _, {:message_queue_size, len}} -> 109 | stop_tracing(state) 110 | exit({:done_tracing, :message_queue_size, len}) 111 | {:EXIT, _, {:max_message_count, count}} -> 112 | stop_tracing(state) 113 | exit({:done_tracing, :max_message_count, count}) 114 | :restart_timer -> 115 | state 116 | |> cancel_timer() 117 | |> start_timer() 118 | |> process_loop() 119 | # testing helpers 120 | {:get_handler_pid, sender_pid} -> 121 | send sender_pid, {:handler_pid, state.handler_pid} 122 | process_loop(state) 123 | {:get_pid_handler_opts, sender_pid} -> 124 | send sender_pid, {:pid_handler_opts, state.pid_handler_opts} 125 | process_loop(state) 126 | 127 | _ignore -> process_loop(state) 128 | end 129 | end 130 | 131 | defp start_timer(state) do 132 | ref = :erlang.start_timer(state.max_tracing_time, self(), []) 133 | put_in(state.timer_ref, ref) 134 | end 135 | 136 | defp cancel_timer(state) do 137 | :erlang.cancel_timer(state.timer_ref, []) 138 | put_in(state.timer_ref, nil) 139 | end 140 | 141 | defp stop_tracing_and_handler(state) do 142 | state 143 | |> stop_tracing() 144 | |> stop_handler() 145 | end 146 | 147 | defp start_handler(state) do 148 | handler_pid = PidHandler.start(state.pid_handler_opts) 149 | put_in(state.handler_pid, handler_pid) 150 | end 151 | 152 | defp stop_handler(state) do 153 | state 154 | |> Map.get(:handler_pid) 155 | |> PidHandler.stop() 156 | put_in(state.handler_pid, nil) 157 | end 158 | 159 | def start_tracing(state) do 160 | # TODO store the number of matches, so that it can be send back to admin 161 | # process 162 | trace_fun = &:erlang.trace/3 163 | state.start_trace_cmds 164 | |> Enum.each(fn 165 | [{:fun, ^trace_fun} | args] -> 166 | bare_args = Enum.map(args, fn 167 | # inject tracer option 168 | {:flag_list, flags} -> [{:tracer, state.handler_pid} | flags] 169 | {_other, arg} -> arg 170 | end) 171 | # IO.puts("#{inspect trace_fun} args: #{inspect bare_args}") 172 | _res = apply(trace_fun, bare_args) 173 | # IO.puts("#{inspect trace_fun} args: #{inspect bare_args}" <> 174 | # " = #{inspect res}") 175 | 176 | [{:fun, fun} | args] -> 177 | bare_args = Enum.map(args, &(elem(&1, 1))) 178 | _res = apply(fun, bare_args) 179 | # IO.puts("#{inspect fun} args: #{inspect bare_args} = #{inspect res}") 180 | end) 181 | state 182 | end 183 | 184 | def stop_tracing(state) do 185 | state.stop_trace_cmds 186 | |> Enum.each(fn [{:fun, fun} | args] -> 187 | bare_args = Enum.map(args, &(elem(&1, 1))) 188 | apply(fun, bare_args) 189 | end) 190 | state 191 | end 192 | 193 | # Handler Callbacks 194 | def discard_handler_callback(_event) do 195 | :ok 196 | end 197 | 198 | def forwarding_handler_callback(event, pid) do 199 | send pid, event 200 | {:ok, pid} 201 | end 202 | 203 | # Remote Loading Helpers 204 | # credit: based on redbug https://github.com/massemanet/redbug 205 | defp ensure_loaded_remote(node, mod) do 206 | case :rpc.call(node, mod, :module_info, [:compile]) do 207 | {:badrpc, {:EXIT, {:undef, _}}} -> 208 | # module was not found 209 | load_remote(node, mod) 210 | ensure_loaded_remote(node, mod) 211 | :ok 212 | {:badrpc , _} -> :ok 213 | info when is_list(info) -> 214 | case {get_ts(info), get_ts(mod.module_info(:compile))} do 215 | {:interpreted, _} -> :ok 216 | {target, host} when target < host -> # old code on target 217 | load_remote(node, mod) 218 | ensure_loaded_remote(node, mod) 219 | _ -> :ok 220 | end 221 | end 222 | end 223 | 224 | defp load_remote(node, mod) do 225 | {mod, bin, fun} = :code.get_object_code(mod) 226 | {:module, _mod} = :rpc.call(node, :code, :load_binary, [mod, fun, bin]) 227 | end 228 | 229 | defp get_ts([]), do: :interpreted 230 | defp get_ts([{:time, time} | _]), do: time 231 | defp get_ts([_ | rest]), do: get_ts(rest) 232 | 233 | end 234 | -------------------------------------------------------------------------------- /lib/tracer/matcher.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Matcher do 2 | @moduledoc """ 3 | Matcher translate an elixir expression to a tracing matchspec 4 | 5 | Initial coded based on ericmj's https://github.com/ericmj/ex2ms 6 | module for ets matchspecs. 7 | """ 8 | alias Tracer.Matcher 9 | 10 | defstruct desc: "", 11 | mfa: nil, 12 | ms: [], 13 | flags: [] 14 | 15 | @bool_functions [ 16 | :is_atom, :is_float, :is_integer, :is_list, :is_number, :is_pid, :is_port, 17 | :is_reference, :is_tuple, :is_binary, :is_function, :is_record, :and, :or, 18 | :not, :xor] 19 | 20 | @guard_functions @bool_functions ++ [ 21 | :abs, :element, :hd, :length, :node, :round, :size, :tl, :trunc, :+, :-, :*, 22 | :div, :rem, :band, :bor, :bxor, :bnot, :bsl, :bsr, :>, :>=, :<, :<=, :===, 23 | :==, :!==, :!=, :self] 24 | 25 | @body_custom_functions [ 26 | :count, :gauge, :histogram] 27 | 28 | @body_trace_functions [ 29 | process_dump: 0, caller: 0, return_trace: 0, excpetion_trace: 0, 30 | self: 0, node: 0, disable_trace: [2, 3], enable_trace: [2, 3], 31 | get_tcw: 0, set_tcw: 1, get_seq_token: 0, set_seq_token: 2, 32 | is_seq_token: 0, trace: 2, silent: 1] 33 | 34 | @elixir_erlang [ 35 | ===: :"=:=", !==: :"=/=", !=: :"/=", <=: :"=<", and: :andalso, or: :orelse] 36 | 37 | Enum.map(@guard_functions, fn(atom) -> 38 | defp is_guard_function(unquote(atom)), do: true 39 | end) 40 | defp is_guard_function(_), do: false 41 | 42 | Enum.map(@body_custom_functions, fn(atom) -> 43 | defp is_custom_function(unquote(atom)), do: true 44 | end) 45 | defp is_custom_function(_), do: false 46 | 47 | Enum.map(@body_trace_functions, fn({atom, arity}) -> 48 | defp is_trace_function(unquote(atom)), do: true 49 | if is_list(arity) do 50 | defp trace_function_arity?(unquote(atom), val) do 51 | Enum.member?(unquote(arity), val) 52 | end 53 | else 54 | defp trace_function_arity?(unquote(atom), val) do 55 | unquote(arity) == val 56 | end 57 | end 58 | end) 59 | defp is_trace_function(_), do: false 60 | 61 | Enum.map(@elixir_erlang, fn({elixir, erlang}) -> 62 | defp map_elixir_erlang(unquote(elixir)), do: unquote(erlang) 63 | end) 64 | defp map_elixir_erlang(atom), do: atom 65 | 66 | def base_match(clauses, outer_vars) do 67 | clauses 68 | |> Enum.reduce(%Matcher{}, fn 69 | {:->, _, clause}, acc -> 70 | {head, conds, body, state} = translate_clause(clause, outer_vars) 71 | acc = if Map.get(state, :mod) != nil do 72 | clause_mfa = {state.mod, state.fun, state.arity} 73 | acc_mfa = Map.get(acc, :mfa) 74 | if acc_mfa != nil and acc_mfa != clause_mfa do 75 | raise ArgumentError, message: 76 | "clause mfa #{inspect acc_mfa}" <> 77 | " does not match #{inspect clause_mfa}" 78 | end 79 | Map.put(acc, :mfa, clause_mfa) 80 | else acc end 81 | Map.put(acc, :ms, acc.ms ++ [{head, conds, body}]) 82 | head, acc -> 83 | {head, conds, state} = translate_head([head], outer_vars) 84 | acc = if Map.get(state, :mod) != nil do 85 | clause_mfa = {state.mod, state.fun, state.arity} 86 | acc_mfa = Map.get(acc, :mfa) 87 | if acc_mfa != nil and acc_mfa != clause_mfa do 88 | raise ArgumentError, message: 89 | "clause mfa #{inspect acc_mfa}" <> 90 | " does not match #{inspect clause_mfa}" 91 | end 92 | Map.put(acc, :mfa, clause_mfa) 93 | else acc end 94 | message = state.vars 95 | |> Enum.map(fn {var, key} -> [var, String.to_atom(key)] end) 96 | |> Enum.reverse() 97 | Map.put(acc, :ms, acc.ms ++ [{head, conds, [message: message]}]) 98 | end) 99 | end 100 | 101 | defmacro match([do: clauses]) do 102 | outer_vars = __CALLER__.vars 103 | case base_match(clauses, outer_vars) do 104 | %{mfa: nil} = m -> 105 | m 106 | |> Map.get(:ms) 107 | |> Macro.escape(unquote: true) 108 | _ -> raise ArgumentError, message: "explicit function not allowed" 109 | end 110 | end 111 | defmacro match(_) do 112 | raise ArgumentError, message: "invalid args to matchspec" 113 | end 114 | 115 | [:global, :local] |> Enum.each(fn (flag) -> 116 | defmacro unquote(flag)([do: clauses]) do 117 | outer_vars = __CALLER__.vars 118 | match_desc = clauses 119 | |> Macro.to_string() 120 | |> case do 121 | "(" <> desc -> "#{unquote(flag)} do #{String.slice(desc, 0..-2)} end" 122 | desc -> "#{unquote(flag)} do #{desc} end" 123 | end 124 | clauses 125 | |> case do 126 | clauses when is_list(clauses) -> clauses 127 | {:__block__, _, clauses} -> clauses 128 | clause -> [clause] 129 | end 130 | |> base_match(outer_vars) 131 | |> case do 132 | %{mfa: nil} = bm -> Map.put(bm, :mfa, {:_, :_, :_}) 133 | bm -> bm 134 | end 135 | |> Map.put(:flags, [unquote(flag)]) 136 | |> Map.put(:desc, match_desc) 137 | |> Macro.escape(unquote: true) 138 | end 139 | defmacro unquote(flag)(clause) do 140 | outer_vars = __CALLER__.vars 141 | match_desc = "#{unquote(flag)} " <> Macro.to_string(clause) 142 | [clause] 143 | |> base_match(outer_vars) 144 | |> case do 145 | %{mfa: nil} = bm -> Map.put(bm, :mfa, {:_, :_, :_}) 146 | bm -> bm 147 | end 148 | |> Map.put(:flags, [unquote(flag)]) 149 | |> Map.put(:desc, match_desc) 150 | |> Macro.escape(unquote: true) 151 | end 152 | end) 153 | 154 | defmacrop is_literal(term) do 155 | quote do 156 | is_atom(unquote(term)) or 157 | is_number(unquote(term)) or 158 | is_binary(unquote(term)) 159 | end 160 | end 161 | 162 | defp translate_clause([head, body], outer_vars) do 163 | {head, conds, state} = translate_head(head, outer_vars) 164 | 165 | body = translate_body(body, state) 166 | {head, conds, body, state} 167 | end 168 | 169 | defp set_annotation(state) do 170 | Map.put(state, :annotation, true) 171 | end 172 | 173 | defp annotating?(state) do 174 | Map.get(state, :annotation, false) 175 | end 176 | 177 | defp increase_depth(state) do 178 | Map.put(state, :depth, get_depth(state) + 1) 179 | end 180 | 181 | defp get_depth(state) do 182 | Map.get(state, :depth, 0) 183 | end 184 | 185 | # Translate Body 186 | defp translate_body({:__block__, _, exprs}, state) when is_list(exprs) do 187 | body = Enum.map(exprs, &translate_body_term(&1, state)) 188 | if many_messages?(body) do 189 | raise ArgumentError, message: "multiple messages or incompatible actions" 190 | else 191 | body 192 | end 193 | end 194 | 195 | defp translate_body(nil, _), do: [] 196 | defp translate_body(expr, state) do 197 | [translate_body_term(expr, state)] 198 | end 199 | 200 | defp many_messages?(body) do 201 | Enum.reduce(body, 0, fn 202 | {:message, _}, acc -> acc + 1 203 | _, acc -> acc 204 | end) > 1 205 | end 206 | 207 | defp translate_body_term({var, _, nil}, state) when is_atom(var) do 208 | if match_var = state.vars[var] do 209 | if annotating?(state), do: [var, :"#{match_var}"], else: :"#{match_var}" 210 | else 211 | raise ArgumentError, message: "variable `#{var}` is unbound in matchspec" 212 | end 213 | end 214 | 215 | defp translate_body_term({left, right}, state), 216 | do: translate_body_term({:{}, [], [left, right]}, state) 217 | defp translate_body_term({:{}, _, list}, state) when is_list(list) do 218 | list 219 | |> Enum.map(&translate_body_term(&1, increase_depth(state))) 220 | |> List.to_tuple 221 | end 222 | 223 | defp translate_body_term({:%{}, _, list}, state) do 224 | list 225 | |> Enum.reduce({%{}, state}, fn {key, value}, {map, state} -> 226 | value = translate_body_term(value, increase_depth(state)) 227 | {Map.put(map, key, value), state} 228 | end) 229 | |> elem(0) 230 | end 231 | 232 | defp translate_body_term({:^, _, [var]}, _state) do 233 | {:unquote, [], [var]} 234 | end 235 | 236 | defp translate_body_term({fun, _, args}, state) 237 | when fun === :message or fun === :display and is_list(args) do 238 | if get_depth(state) == 0 do 239 | match_args = Enum.map(args, 240 | &translate_body_term(&1, 241 | state 242 | |> set_annotation() 243 | |> increase_depth() 244 | )) 245 | {fun, match_args} 246 | else 247 | raise ArgumentError, message: "`#{fun}` cannot be nested" 248 | end 249 | end 250 | 251 | defp translate_body_term({fun, _, args}, state) 252 | when is_atom(fun) and is_list(args) do 253 | cond do 254 | is_guard_function(fun) -> 255 | match_args = Enum.map(args, 256 | &translate_body_term(&1, increase_depth(state))) 257 | match_fun = map_elixir_erlang(fun) 258 | [match_fun | match_args] |> List.to_tuple 259 | is_custom_function(fun) -> 260 | if get_depth(state) == 0 do 261 | match_args = Enum.map(args, 262 | &translate_body_term(&1, 263 | state 264 | |> set_annotation() 265 | |> increase_depth() 266 | )) 267 | match_fun = map_elixir_erlang(fun) 268 | {:message, [[:_cmd, match_fun] | match_args]} 269 | else 270 | raise ArgumentError, message: "`#{fun}` cannot be nested" 271 | end 272 | is_trace_function(fun) -> 273 | if trace_function_arity?(fun, length(args)) do 274 | match_args = Enum.map(args, 275 | &translate_body_term(&1, increase_depth(state))) 276 | match_fun = map_elixir_erlang(fun) 277 | [match_fun | match_args] |> List.to_tuple 278 | else 279 | raise ArgumentError, 280 | message: "`#{fun}/#{length(args)}` is not recognized" 281 | end 282 | true -> 283 | raise ArgumentError, message: "`#{fun}` is not recognized" 284 | end 285 | end 286 | 287 | defp translate_body_term(list, state) when is_list(list) do 288 | Enum.map(list, &translate_body_term(&1, state)) 289 | end 290 | 291 | defp translate_body_term(literal, _state) when is_literal(literal) do 292 | literal 293 | end 294 | 295 | defp translate_body_term(_, _state), do: raise_expression_error() 296 | 297 | # Translate Condition 298 | defp translate_cond({var, _, nil}, state) when is_atom(var) do 299 | if match_var = state.vars[var] do 300 | :"#{match_var}" 301 | else 302 | raise ArgumentError, message: "variable `#{var}` is unbound in matchspec" 303 | end 304 | end 305 | 306 | defp translate_cond({left, right}, state), 307 | do: translate_cond({:{}, [], [left, right]}, state) 308 | defp translate_cond({:{}, _, list}, state) when is_list(list) do 309 | list 310 | |> Enum.map(&translate_cond(&1, state)) 311 | |> List.to_tuple 312 | end 313 | 314 | defp translate_cond({:^, _, [var]}, _state) do 315 | {:unquote, [], [var]} 316 | end 317 | 318 | defp translate_cond({fun, _, args}, state) 319 | when is_atom(fun) and is_list(args) do 320 | if is_guard_function(fun) do 321 | match_args = Enum.map(args, &translate_cond(&1, state)) 322 | match_fun = map_elixir_erlang(fun) 323 | [match_fun | match_args] |> List.to_tuple 324 | else 325 | raise ArgumentError, message: "`#{fun}` is not allowed in condition" 326 | end 327 | end 328 | 329 | defp translate_cond(list, state) when is_list(list) do 330 | Enum.map(list, &translate_cond(&1, state)) 331 | end 332 | 333 | defp translate_cond(literal, _state) when is_literal(literal) do 334 | literal 335 | end 336 | 337 | defp translate_cond(_, _state), do: raise_expression_error() 338 | 339 | # Translate Head 340 | defp translate_head([{:when, _, params}], outer_vars) 341 | when is_list(params) and length(params) > 1 do 342 | {param, condition} = Enum.split(params, -1) 343 | 344 | initial_state = %{vars: [], count: 0, outer_vars: outer_vars} 345 | {param, state} = extract_fun(param, initial_state) 346 | {head, state} = do_translate_param(param, state) 347 | 348 | condition = translate_cond(condition, state) 349 | {head, condition, state} 350 | end 351 | 352 | defp translate_head([], outer_vars) do 353 | {[], [], %{vars: [], count: 0, outer_vars: outer_vars}} 354 | end 355 | defp translate_head(param, outer_vars) when is_list(param) do 356 | initial_state = %{vars: [], count: 0, outer_vars: outer_vars} 357 | {param, state} = extract_fun(param, initial_state) 358 | {head, state} = do_translate_param(param, state) 359 | 360 | {head, [], state} 361 | end 362 | 363 | defp translate_head(_, _), do: raise_parameter_error() 364 | 365 | defp extract_fun(head, state) do 366 | case head do 367 | # Case 1: Mod.fun._ 368 | [{{:., _, [{:__aliases__, _, mod}, :_]}, _, []}] -> 369 | {:_, set_mfa(state, Module.concat(mod), :_, :_)} 370 | # Case 2: Mod.fun(args) 371 | [{{:., _, [{:__aliases__, _, mod}, fun]}, _, new_head}] 372 | when is_list(new_head) -> 373 | {new_head, set_mfa(state, Module.concat(mod), fun, length(new_head))} 374 | # Case 3: Mod._._ 375 | [{{:., _, [{{:., _, [{:__aliases__, _, mod}, :_]}, _, []}, :_]}, 376 | _, []}] -> 377 | {:_, set_mfa(state, Module.concat(mod), :_, :_)} 378 | # Case 4: Mod.fun._ 379 | [{{:., _, [{{:., _, [{:__aliases__, _, mod}, fun]}, _, []}, :_]}, 380 | _, []}] -> 381 | {:_, set_mfa(state, Module.concat(mod), fun, :_)} 382 | # Case 5: _._._ 383 | [{{:., _, [{{:., _, [{:_, _, nil}, :_]}, _, []}, :_]}, _, []}] -> 384 | {:_, set_mfa(state, :_, :_, :_)} 385 | # Case 6 _._ 386 | [{{:., _, [{:_, _, nil}, :_]}, _, []}] -> 387 | {:_, set_mfa(state, :_, :_, :_)} 388 | # Case 7 :mod._ 389 | [{{:., _, [mod, fun]}, _, []}] when is_atom(mod) and is_atom(fun) -> 390 | {:_, set_mfa(state, mod, fun, :_)} 391 | [{{:., _, [{{:., _, [mod, fun]}, _, []}, :_]}, _, []}] 392 | when is_atom(mod) and is_atom(fun) -> 393 | {[], set_mfa(state, mod, fun, :_)} 394 | # Case 8 :mod.fun(args) 395 | [{{:., _, [mod, fun]}, _, new_head}] when is_atom(mod) and is_atom(fun) -> 396 | {new_head, set_mfa(state, mod, fun, length(new_head))} 397 | # Case 9 _ 398 | [{:_, _, nil}] -> 399 | {:_, set_mfa(state, :_, :_, :_)} 400 | # Case 10 No function, is (args) 401 | _ -> {head, state} 402 | end 403 | end 404 | 405 | defp set_mfa(state, m, f, a) do 406 | state 407 | |> Map.put(:mod, m) 408 | |> Map.put(:fun, f) 409 | |> Map.put(:arity, a) 410 | end 411 | 412 | defp do_translate_param({:_, _, nil}, state) do 413 | {:_, state} 414 | end 415 | 416 | defp do_translate_param({var, _, nil}, state) when is_atom(var) do 417 | if match_var = state.vars[var] do 418 | {:"#{match_var}", state} 419 | else 420 | match_var = "$#{state.count+1}" 421 | state = state 422 | |> Map.update!(:vars, &[{var, match_var} | &1]) 423 | |> Map.update!(:count, &(&1 + 1)) 424 | {:"#{match_var}", state} 425 | end 426 | end 427 | 428 | defp do_translate_param({left, right}, state) do 429 | do_translate_param({:{}, [], [left, right]}, state) 430 | end 431 | 432 | defp do_translate_param({:{}, _, list}, state) when is_list(list) do 433 | {list, state} = Enum.map_reduce(list, state, &do_translate_param(&1, &2)) 434 | {List.to_tuple(list), state} 435 | end 436 | 437 | defp do_translate_param({:^, _, [var]}, state) do 438 | {{:unquote, [], [var]}, state} 439 | end 440 | 441 | defp do_translate_param(list, state) when is_list(list) do 442 | Enum.map_reduce(list, state, &do_translate_param(&1, &2)) 443 | end 444 | 445 | defp do_translate_param(literal, state) when is_literal(literal) do 446 | {literal, state} 447 | end 448 | 449 | defp do_translate_param({:%{}, _, list}, state) do 450 | Enum.reduce list, {%{}, state}, fn {key, value}, {map, state} -> 451 | {key, key_state} = do_translate_param(key, state) 452 | {value, value_state} = do_translate_param(value, key_state) 453 | {Map.put(map, key, value), value_state} 454 | end 455 | end 456 | 457 | defp do_translate_param(_, _state), do: raise_parameter_error() 458 | 459 | defp raise_expression_error do 460 | raise ArgumentError, message: "illegal expression in matchspec" 461 | end 462 | 463 | defp raise_parameter_error do 464 | raise ArgumentError, message: "invalid parameters" 465 | end 466 | 467 | end 468 | -------------------------------------------------------------------------------- /lib/tracer/pid_handler.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.PidHandler do 2 | @moduledoc """ 3 | PidHandler is the process consumer handler of trace events 4 | It passes them to the event_callback once received 5 | """ 6 | alias __MODULE__ 7 | import Tracer.Macros 8 | 9 | @default_max_message_count 1000 10 | @default_max_queue_size 1000 11 | 12 | defstruct message_count: 0, 13 | max_message_count: @default_max_message_count, 14 | max_queue_size: @default_max_queue_size, 15 | event_callback: nil 16 | 17 | def start(opts) when is_list(opts) do 18 | initial_state = opts 19 | |> Enum.reduce(%PidHandler{}, fn ({keyword, value}, pid_handler) -> 20 | Map.put(pid_handler, keyword, value) 21 | end) 22 | |> tuple_x_and_fx(Map.get(:event_callback)) 23 | |> case do 24 | {_state, nil} -> 25 | raise ArgumentError, message: "missing event_callback configuration" 26 | {state, {cbk_fun, _cbk_state}} when is_function(cbk_fun) -> 27 | state 28 | {state, cbk_fun} when is_function(cbk_fun) -> 29 | case :erlang.fun_info(cbk_fun, :arity) do 30 | {_, 1} -> 31 | # encapsulate into arity 2 32 | put_in(state.event_callback, 33 | {fn(e, []) -> {cbk_fun.(e), []} end, []}) 34 | {_, 2} -> 35 | put_in(state.event_callback, {cbk_fun, []}) 36 | {_, arity} -> 37 | raise ArgumentError, 38 | message: "#invalid arity/#{inspect arity} for: #{inspect cbk_fun}" 39 | end 40 | {_state, invalid_callback} -> 41 | raise ArgumentError, 42 | message: "#invalid event_callback: #{inspect invalid_callback}" 43 | end 44 | 45 | spawn_link(fn -> process_loop(initial_state) end) 46 | end 47 | 48 | def stop(pid) when is_pid(pid) do 49 | send pid, :stop 50 | end 51 | 52 | defp process_loop(state) do 53 | check_message_queue_size(state) 54 | receive do 55 | :stop -> exit(:normal) 56 | trace_event when is_tuple(trace_event) -> 57 | state 58 | |> handle_trace_event(trace_event) 59 | |> process_loop() 60 | _ignored -> process_loop(state) 61 | end 62 | end 63 | 64 | defp handle_trace_event(state, trace_event) do 65 | case Tuple.to_list(trace_event) do 66 | [trace | _] when trace === :trace or trace === :trace_ts -> 67 | state 68 | |> call_event_callback(trace_event) 69 | |> check_message_count() 70 | _unknown -> state # ignore 71 | end 72 | end 73 | 74 | defp call_event_callback(state, trace_event) do 75 | {cbk_fun, cbk_state} = state.event_callback 76 | case cbk_fun.(trace_event, cbk_state) do 77 | {:ok, new_cbk_state} -> 78 | put_in(state.event_callback, {cbk_fun, new_cbk_state}) 79 | error -> exit(error) 80 | end 81 | end 82 | 83 | defp check_message_queue_size(%PidHandler{max_queue_size: max}) do 84 | case :erlang.process_info(self(), :message_queue_len) do 85 | {:message_queue_len, len} when len > max -> 86 | exit({:message_queue_size, len}) 87 | _ -> :ok 88 | end 89 | end 90 | 91 | defp check_message_count(state) do 92 | state 93 | |> Map.put(:message_count, state.message_count + 1) 94 | |> case do 95 | %PidHandler{message_count: count, max_message_count: max} 96 | when count >= max -> exit({:max_message_count, state.max_message_count}) 97 | state -> state 98 | end 99 | end 100 | 101 | end 102 | -------------------------------------------------------------------------------- /lib/tracer/probe.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Probe do 2 | @moduledoc """ 3 | Probe manages a single probe 4 | """ 5 | 6 | alias __MODULE__ 7 | alias Tracer.{Clause, Matcher} 8 | 9 | @valid_types [:call, :procs, :gc, :sched, :send, :receive, 10 | :set_on_spawn, :set_on_first_spawn, 11 | :set_on_link, :set_on_first_link] 12 | @flag_map %{ 13 | call: :call, 14 | procs: :procs, 15 | sched: :running, 16 | send: :send, 17 | receive: :receive, 18 | gc: :garbage_collection, 19 | set_on_spawn: :set_on_spawn, 20 | set_on_first_spawn: :set_on_first_spawn, 21 | set_on_link: :set_on_link, 22 | set_on_first_link: :set_on_first_link 23 | } 24 | 25 | @new_options [ 26 | :type, :process, :match 27 | ] 28 | 29 | defstruct type: nil, 30 | process_list: [], 31 | clauses: [], 32 | enabled?: true, 33 | flags: [] 34 | 35 | def new(opts) when is_list(opts) do 36 | if Keyword.fetch(opts, :type) != :error do 37 | Enum.reduce(opts, 38 | %Probe{flags: default_flags(opts)}, 39 | fn {field, val}, probe -> 40 | cond do 41 | is_tuple(probe) and elem(probe, 0) == :error -> probe 42 | !Enum.member?(@new_options, field) -> 43 | {:error, "#{field} not a valid option"} 44 | true -> 45 | apply(__MODULE__, field, [probe, val]) 46 | end 47 | end) 48 | else 49 | {:error, :missing_type} 50 | end 51 | end 52 | 53 | def new(type, opts \\ []) when is_list(opts) do 54 | if Enum.member?(@valid_types, type) do 55 | new([type: type] ++ opts) 56 | else 57 | {:error, :invalid_probe_type} 58 | end 59 | end 60 | 61 | defp default_flags(opts) do 62 | if Keyword.get(opts, :type) == :call do 63 | [:arity, :timestamp] 64 | else 65 | [:timestamp] 66 | end 67 | end 68 | 69 | # Generate functions Probe.call, Probe.process, ... 70 | # @valid_types |> Enum.each(fn type -> 71 | # def unquote(type)(opts) when is_list(opts) do 72 | # Probe.new([{:type, unquote(type)} | opts]) 73 | # end 74 | # end) 75 | 76 | def get_type(probe) do 77 | probe.type 78 | end 79 | 80 | def process_list(probe) do 81 | probe.process_list 82 | end 83 | 84 | def process_list(probe, procs) do 85 | put_in(probe.process_list, Enum.uniq(procs)) 86 | end 87 | 88 | def add_process(probe, procs) when is_list(procs) do 89 | put_in(probe.process_list, 90 | probe.process_list 91 | |> Enum.concat(procs) 92 | |> Enum.uniq 93 | ) 94 | end 95 | def add_process(probe, proc) do 96 | put_in(probe.process_list, Enum.uniq([proc | probe.process_list])) 97 | end 98 | 99 | def remove_process(probe, procs) when is_list(procs) do 100 | put_in(probe.process_list, probe.process_list -- procs) 101 | end 102 | def remove_process(probe, proc) do 103 | remove_process(probe, [proc]) 104 | end 105 | 106 | def add_clauses(probe, clauses) when is_list(clauses) do 107 | with [] <- valid_clauses?(clauses, Probe.get_type(probe)) do 108 | put_in(probe.clauses, Enum.concat(probe.clauses, clauses)) 109 | else 110 | error_list -> {:error, error_list} 111 | end 112 | end 113 | def add_clauses(probe, clause) do 114 | add_clauses(probe, [clause]) 115 | end 116 | 117 | defp valid_clauses?(clauses, expected_type) do 118 | Enum.reduce(clauses, [], fn 119 | %Clause{type: ^expected_type}, acc -> acc 120 | %Clause{} = c, acc -> [{:invalid_clause_type, c} | acc] 121 | c, acc -> [{:not_a_clause, c} | acc] 122 | end) 123 | end 124 | 125 | def remove_clauses(probe, clauses) when is_list(clauses) do 126 | put_in(probe.clauses, probe.clauses -- clauses) 127 | end 128 | def remove_clauses(probe, clause) do 129 | remove_clauses(probe, [clause]) 130 | end 131 | 132 | def clauses(probe) do 133 | probe.clauses 134 | end 135 | 136 | def enable(probe) do 137 | put_in(probe.enabled?, true) 138 | end 139 | 140 | def disable(probe) do 141 | put_in(probe.enabled?, false) 142 | end 143 | 144 | def enabled?(probe) do 145 | probe.enabled? 146 | end 147 | 148 | def valid?(probe) do 149 | with :ok <- validate_process_list(probe) do 150 | true 151 | end 152 | end 153 | 154 | def apply(probe, flags \\ []) do 155 | with true <- valid?(probe) do 156 | # Apply trace commands 157 | Enum.each(probe.process_list, fn p -> 158 | :erlang.trace(p, probe.enabled?, flags ++ flags(probe)) 159 | end) 160 | # Apply trace_pattern commands 161 | Enum.each(probe.clauses, fn c -> Clause.apply(c, probe.enabled?) end) 162 | probe 163 | end 164 | end 165 | 166 | def get_trace_cmds(probe, flags \\ []) do 167 | if probe.enabled? do 168 | with true <- valid?(probe) do 169 | Enum.map(probe.clauses, &Clause.get_trace_cmd(&1)) 170 | ++ Enum.map(probe.process_list, fn p -> 171 | [ 172 | fun: &:erlang.trace/3, 173 | pid_port_spec: p, 174 | how: true, 175 | flag_list: flags ++ flags(probe) 176 | ] 177 | end) 178 | else 179 | error -> raise RuntimeError, message: "invalid probe #{inspect error}" 180 | end 181 | else 182 | [] 183 | end 184 | end 185 | 186 | [:arity, :timestamp, :return_to] |> Enum.each(fn flag -> 187 | def unquote(flag)(probe, enable) when is_boolean(enable) do 188 | if probe.type == :call do 189 | flag(probe, unquote(flag), enable) 190 | else 191 | {:error, :not_a_call_probe} 192 | end 193 | end 194 | end) 195 | 196 | defp flag(probe, flag, true) do 197 | put_in(probe.flags, Enum.uniq([flag | probe.flags])) 198 | end 199 | defp flag(probe, flag, false) do 200 | put_in(probe.flags, probe.flags -- [flag]) 201 | end 202 | 203 | defp flags(probe) do 204 | [Map.get(@flag_map, probe.type) | probe.flags] 205 | end 206 | 207 | defp valid_type?(type) do 208 | if Enum.member?(@valid_types, type) do 209 | {:ok, type} 210 | else 211 | {:error, :invalid_type} 212 | end 213 | end 214 | 215 | defp validate_process_list(probe) do 216 | if Enum.empty?(probe.process_list) do 217 | {:error, :missing_processes} 218 | else 219 | :ok 220 | end 221 | end 222 | 223 | # Helper Functions 224 | def type(probe, type) do 225 | with {:ok, type} <- valid_type?(type) do 226 | put_in(probe.type, type) 227 | end 228 | end 229 | 230 | def process(probe, process) do 231 | add_process(probe, process) 232 | end 233 | 234 | def match(probe, fun) when is_function(fun) do 235 | case probe.clauses do 236 | [] -> 237 | put_in(probe.clauses, [Clause.new() |> Clause.put_fun(fun)]) 238 | [clause | rest] -> 239 | put_in(probe.clauses, [Clause.put_fun(clause, fun) | rest]) 240 | end 241 | end 242 | 243 | def match(probe, {m}), do: match(probe, {m, :_, :_}) 244 | def match(probe, {m, f}), do: match(probe, {m, f, :_}) 245 | def match(probe, {m, f, a}) do 246 | case probe.clauses do 247 | [] -> 248 | put_in(probe.clauses, [Clause.new() |> Clause.put_mfa(m, f, a)]) 249 | [clause | rest] -> 250 | put_in(probe.clauses, [Clause.put_mfa(clause, m, f, a) | rest]) 251 | end 252 | end 253 | 254 | def match(probe, %Matcher{} = matcher) do 255 | {m, f, a} = Map.get(matcher, :mfa) 256 | clause = Clause.new() 257 | |> Clause.add_matcher(Map.get(matcher, :ms)) 258 | |> Clause.put_mfa(m, f, a) 259 | |> Clause.set_flags(Map.get(matcher, :flags, [])) 260 | |> Clause.set_desc(Map.get(matcher, :desc, "unavailable")) 261 | put_in(probe.clauses, [clause | probe.clauses]) 262 | end 263 | 264 | def match(_, _) do 265 | raise ArgumentError, 266 | message: "Not a valid matcher, check your syntax. Forgot local/global?" 267 | end 268 | 269 | end 270 | -------------------------------------------------------------------------------- /lib/tracer/probe_list.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.ProbeList do 2 | @moduledoc """ 3 | Helper functions to manage and validate a set of probes 4 | """ 5 | 6 | alias Tracer.Probe 7 | 8 | def add_probe(probes, %Probe{} = probe) do 9 | with false <- Enum.any?(probes, fn p -> p.type == probe.type end) do 10 | # keep order 11 | probes ++ [probe] 12 | else 13 | _ -> {:error, :duplicate_probe_type} 14 | end 15 | end 16 | def add_probe(_probes, _) do 17 | {:error, :not_a_probe} 18 | end 19 | 20 | def remove_probe(probes, %Probe{} = probe) do 21 | Enum.filter(probes, fn p -> p != probe end) 22 | end 23 | def remove_probe(_probes, _) do 24 | {:error, :not_a_probe} 25 | end 26 | 27 | def valid?(probes) do 28 | with :ok <- validate_probes(probes) do 29 | :ok 30 | end 31 | end 32 | 33 | defp validate_probes(probes) do 34 | if Enum.empty?(probes) do 35 | {:error, :missing_probes} 36 | else 37 | probes 38 | |> Enum.reduce([], fn p, acc -> 39 | case Probe.valid?(p) do 40 | true -> acc 41 | {:error, error} -> [{:error, error, p} | acc] 42 | end 43 | end) 44 | |> case do 45 | [] -> :ok 46 | errors -> {:error, {:invalid_probe, errors}} 47 | end 48 | end 49 | end 50 | 51 | end 52 | -------------------------------------------------------------------------------- /lib/tracer/tool.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Tool do 2 | @moduledoc """ 3 | Module that is used by all Tool implementations 4 | """ 5 | alias Tracer.{Probe, ProbeList} 6 | 7 | @callback init([any]) :: any 8 | 9 | defmacro __using__(_opts) do 10 | quote do 11 | alias Tracer.Tool 12 | @behaviour Tool 13 | 14 | @__allowed_opts__ [:probe, :probes, :forward_to, :process, 15 | :max_tracing_time, :max_message_count, 16 | :max_queue_size, :node] 17 | 18 | def init_tool(_, _, _allowed_opts \\ nil) 19 | def init_tool(%{"__struct__": mod} = state, opts, allowed_opts) 20 | when is_list(opts) do 21 | if is_list(allowed_opts) do 22 | invalid_opts = get_invalid_options(opts, 23 | allowed_opts ++ @__allowed_opts__) 24 | if not Enum.empty?(invalid_opts) do 25 | raise ArgumentError, message: 26 | "not supported options: #{Enum.join(invalid_opts, ", ")}" 27 | end 28 | end 29 | 30 | tool_state = %Tool{ 31 | forward_to: Keyword.get(opts, :forward_to), 32 | process: Keyword.get(opts, :process, self()), 33 | agent_opts: extract_agent_opts(opts) 34 | } 35 | 36 | state = state 37 | |> Map.put(:"__tool__", tool_state) 38 | |> set_probes(Keyword.get(opts, :probes, [])) 39 | 40 | Enum.reduce(opts, state, fn 41 | {:probe, probe}, state -> 42 | Tool.add_probe(state, probe) 43 | _, state -> state 44 | end) 45 | end 46 | def init_tool(_, _, _) do 47 | raise ArgumentError, 48 | message: "arguments needs to be a map and a keyword list" 49 | end 50 | 51 | defp extract_agent_opts(opts) do 52 | agents_keys = [:max_tracing_time, 53 | :max_message_count, 54 | :max_queue_size, 55 | :node] 56 | Enum.filter(opts, 57 | fn {key, _} -> Enum.member?(agents_keys, key) end) 58 | end 59 | 60 | defp get_invalid_options(opts, allowed_opts) do 61 | Enum.reduce(opts, [], fn {key, val}, acc -> 62 | if Enum.member?(allowed_opts, key), do: acc, 63 | else: acc ++ [Atom.to_string(key)] 64 | end) 65 | end 66 | 67 | defp set_probes(state, probes) do 68 | tool_state = state 69 | |> Map.get(:"__tool__") 70 | |> Map.put(:probes, probes) 71 | Map.put(state, :"__tool__", tool_state) 72 | end 73 | 74 | defp get_probes(state) do 75 | Tool.get_tool_field(state, :probes) 76 | end 77 | 78 | defp get_process(state) do 79 | Tool.get_tool_field(state, :process) 80 | end 81 | 82 | defp report_event(state, event) do 83 | case Tool.get_forward_to(state) do 84 | nil -> 85 | IO.puts to_string(event) 86 | pid when is_pid(pid) -> 87 | send pid, event 88 | end 89 | end 90 | 91 | def handle_start(state), do: state 92 | def handle_event(event, state), do: state 93 | def handle_flush(state), do: state 94 | def handle_stop(state), do: state 95 | def handle_valid?(state), do: :ok 96 | 97 | defoverridable [handle_start: 1, 98 | handle_event: 2, 99 | handle_flush: 1, 100 | handle_stop: 1, 101 | handle_valid?: 1] 102 | 103 | :ok 104 | end 105 | end 106 | 107 | defstruct probes: [], 108 | forward_to: nil, 109 | process: nil, 110 | agent_opts: [] 111 | 112 | def new(tool_module, params) do 113 | tool_module.init(params) 114 | end 115 | 116 | def get_tool_field(state, field) do 117 | state 118 | |> Map.get(:"__tool__") 119 | |> Map.get(field) 120 | end 121 | 122 | def set_tool_field(state, field, value) do 123 | tool_state = state 124 | |> Map.get(:"__tool__") 125 | |> Map.put(field, value) 126 | Map.put(state, :"__tool__", tool_state) 127 | end 128 | 129 | def get_agent_opts(state) do 130 | get_tool_field(state, :agent_opts) 131 | end 132 | 133 | def get_node(state) do 134 | agent_opts = get_tool_field(state, :agent_opts) 135 | Keyword.get(agent_opts, :node, nil) 136 | end 137 | 138 | def get_probes(state) do 139 | get_tool_field(state, :probes) 140 | end 141 | 142 | def get_forward_to(state) do 143 | get_tool_field(state, :forward_to) 144 | end 145 | 146 | def add_probe(state, %Tracer.Probe{} = probe) do 147 | with true <- Probe.valid?(probe) do 148 | probes = get_probes(state) 149 | case ProbeList.add_probe(probes, probe) do 150 | {:error, error} -> 151 | {{:error, error}, state} 152 | probe_list when is_list(probe_list) -> 153 | set_tool_field(state, :probes, probe_list) 154 | end 155 | end 156 | end 157 | def add_probe(_, _) do 158 | {:error, :not_a_probe} 159 | end 160 | 161 | def remove_probe(state, %Tracer.Probe{} = probe) do 162 | probes = get_probes(state) 163 | case ProbeList.remove_probe(probes, probe) do 164 | {:error, error} -> 165 | {{:error, error}, state} 166 | probe_list when is_list(probe_list) -> 167 | set_tool_field(state, :probes, probe_list) 168 | end 169 | end 170 | 171 | def valid?(state) do 172 | with :ok <- ProbeList.valid?(get_probes(state)) do 173 | handle_valid?(state) 174 | else 175 | {:error, :missing_probes} -> 176 | raise ArgumentError, 177 | message: "missing probes, maybe a missing match option?" 178 | other -> 179 | raise ArgumentError, 180 | message: "invalid probe: #{inspect other}" 181 | end 182 | end 183 | 184 | # Routing Helpers 185 | 186 | def handle_start(%{"__struct__": mod} = state) do 187 | mod.handle_start(state) 188 | end 189 | 190 | def handle_event(event, %{"__struct__": mod} = state) do 191 | mod.handle_event(event, state) 192 | end 193 | 194 | def handle_flush(%{"__struct__": mod} = state) do 195 | mod.handle_flush(state) 196 | end 197 | 198 | def handle_stop(%{"__struct__": mod} = state) do 199 | mod.handle_stop(state) 200 | end 201 | 202 | def handle_valid?(%{"__struct__": mod} = state) do 203 | mod.handle_valid?(state) 204 | end 205 | 206 | end 207 | -------------------------------------------------------------------------------- /lib/tracer/tool_helper.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.ToolHelper do 2 | @moduledoc """ 3 | Helper functions for developing tools 4 | """ 5 | 6 | def to_mfa({m, f, a}) when is_atom(m) and is_atom(f) and 7 | (is_integer(a) or a == :_) do 8 | {m, f, a} 9 | end 10 | def to_mfa(m) when is_atom(m) do 11 | {m, :_, :_} 12 | end 13 | def to_mfa(fun) when is_function(fun) do 14 | case :erlang.fun_info(fun, :type) do 15 | {:type, :external} -> 16 | with {:module, m} <- :erlang.fun_info(fun, :module), 17 | {:name, f} <- :erlang.fun_info(fun, :name), 18 | {:arity, a} <- :erlang.fun_info(fun, :arity) do 19 | {m, f, a} 20 | else 21 | _ -> {:error, :invalid_mfa} 22 | end 23 | _ -> 24 | {:error, :not_an_external_function} 25 | end 26 | end 27 | def to_mfa(_), do: {:error, :invalid_mfa} 28 | 29 | end 30 | -------------------------------------------------------------------------------- /lib/tracer/tool_server.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.ToolServer do 2 | @moduledoc """ 3 | The ToolServer module has a process that receives the events from the 4 | pid_handler and reports the events. 5 | """ 6 | 7 | alias __MODULE__ 8 | alias Tracer.{Event, EventCall, EventReturnTo, EventReturnFrom, 9 | EventIn, EventOut, Tool} 10 | 11 | defstruct tool_type: nil, 12 | tool_state: nil 13 | 14 | def start(%{"__tool__": _} = tool) do 15 | initial_state = %ToolServer{} 16 | |> Map.put(:tool_state, Tool.handle_start(tool)) 17 | 18 | spawn_link(fn -> process_loop(initial_state) end) 19 | end 20 | 21 | def stop(nil), do: :ok 22 | def stop(pid) when is_pid(pid) do 23 | send pid, :stop 24 | end 25 | 26 | def flush(nil), do: :ok 27 | def flush(pid) when is_pid(pid) do 28 | send pid, :flush 29 | end 30 | 31 | defp process_loop(%ToolServer{} = state) do 32 | receive do 33 | :stop -> 34 | state 35 | |> Map.put(:tool_state, Tool.handle_stop(state.tool_state)) 36 | exit(:done_reporting) 37 | :flush -> 38 | state 39 | |> Map.put(:tool_state, Tool.handle_flush(state.tool_state)) 40 | |> process_loop() 41 | trace_event when is_tuple(trace_event) -> 42 | state 43 | |> handle_trace_event(trace_event) 44 | |> process_loop() 45 | _ignored -> process_loop(state) 46 | end 47 | end 48 | 49 | defp handle_trace_event(state, trace_event) do 50 | trace_event = case trace_event do 51 | {:trace_ts, pid, :call, {m, f, a}, message, ts} -> 52 | %EventCall{mod: m, fun: f, arity: a, pid: pid, message: message, ts: ts} 53 | {:trace_ts, pid, :call, {m, f, a}, ts} -> 54 | %EventCall{mod: m, fun: f, arity: a, pid: pid, ts: ts} 55 | {:trace_ts, pid, :return_from, {m, f, a}, ret, ts} -> 56 | %EventReturnFrom{mod: m, fun: f, arity: a, pid: pid, 57 | return_value: ret, ts: ts} 58 | {:trace_ts, pid, :return_to, {m, f, a}, ts} -> 59 | %EventReturnTo{mod: m, fun: f, arity: a, pid: pid, ts: ts} 60 | {:trace_ts, pid, :return_to, :undefined, ts} -> 61 | %EventReturnTo{mod: :undefined, fun: :undefined, arity: 0, 62 | pid: pid, ts: ts} 63 | {:trace_ts, pid, :in, {m, f, a}, ts} -> 64 | %EventIn{mod: m, fun: f, arity: a, pid: pid, ts: ts} 65 | {:trace_ts, pid, :out, {m, f, a}, ts} -> 66 | %EventOut{mod: m, fun: f, arity: a, pid: pid, ts: ts} 67 | {:trace, pid, :call, {m, f, a}, [message]} -> 68 | %EventCall{mod: m, fun: f, arity: a, pid: pid, message: message, 69 | ts: now()} 70 | {:trace, pid, :call, {m, f, a}} -> 71 | %EventCall{mod: m, fun: f, arity: a, pid: pid, ts: now()} 72 | {:trace, pid, :return_from, {m, f, a}, ret} -> 73 | %EventReturnFrom{mod: m, fun: f, arity: a, pid: pid, 74 | return_value: ret, ts: now()} 75 | {:trace, pid, :return_to, {m, f, a}} -> 76 | %EventReturnTo{mod: m, fun: f, arity: a, pid: pid, ts: now()} 77 | {:trace, pid, :in, {m, f, a}} -> 78 | %EventIn{mod: m, fun: f, arity: a, pid: pid, ts: now()} 79 | {:trace, pid, :out, {m, f, a}} -> 80 | %EventOut{mod: m, fun: f, arity: a, pid: pid, ts: now()} 81 | _other -> 82 | %Event{event: trace_event} 83 | end 84 | 85 | put_in(state.tool_state, Tool.handle_event(trace_event, state.tool_state)) 86 | end 87 | 88 | defp now do 89 | :erlang.timestamp() 90 | end 91 | 92 | end 93 | -------------------------------------------------------------------------------- /lib/tracer/tools/tool_call_seq.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Tool.CallSeq do 2 | @moduledoc """ 3 | Reports duration type traces 4 | """ 5 | alias __MODULE__ 6 | alias Tracer.{EventCall, EventReturnFrom, Probe, 7 | ToolHelper, Tool.CallSeq.Event, ProcessHelper} 8 | 9 | use Tracer.Tool 10 | import Tracer.Matcher 11 | 12 | defstruct ignore_recursion: nil, 13 | start_mfa: nil, 14 | show_args: nil, 15 | show_return: nil, 16 | max_depth: nil, 17 | started: %{}, 18 | stacks: %{}, 19 | depth: %{} 20 | 21 | @allowed_opts [:ignore_recursion, :max_depth, :show_args, 22 | :show_return, :start_match] 23 | 24 | def init(opts) do 25 | init_state = %CallSeq{} 26 | |> init_tool(opts, @allowed_opts) 27 | |> Map.put(:ignore_recursion, 28 | Keyword.get(opts, :ignore_recursion, true)) 29 | |> Map.put(:max_depth, 30 | Keyword.get(opts, :max_depth, 20)) 31 | |> Map.put(:show_args, 32 | Keyword.get(opts, :show_args, false)) 33 | |> Map.put(:show_return, 34 | Keyword.get(opts, :show_return, false)) 35 | 36 | init_state = case Keyword.get(opts, :start_match) do 37 | nil -> 38 | init_state 39 | fun -> 40 | case ToolHelper.to_mfa(fun) do 41 | {_m, _f, _a} = mfa -> 42 | Map.put(init_state, :start_mfa, mfa) 43 | _error -> 44 | raise ArgumentError, 45 | message: "invalid start_match argument #{inspect fun}" 46 | end 47 | end 48 | 49 | match_spec = if init_state.show_args do 50 | local do _ -> 51 | return_trace() 52 | message(:"$_") end 53 | else 54 | local do _ -> return_trace() end 55 | end 56 | 57 | node = Keyword.get(opts, :node) 58 | process = init_state 59 | |> get_process() 60 | |> ProcessHelper.ensure_pid(node) 61 | 62 | all_child = ProcessHelper.find_all_children(process, node) 63 | probe_call = Probe.new(type: :call, 64 | process: [process | all_child], 65 | match: match_spec) 66 | probe_spawn = Probe.new(type: :set_on_spawn, 67 | process: [process | all_child]) 68 | 69 | set_probes(init_state, [probe_call, probe_spawn]) 70 | end 71 | 72 | def handle_event(event, state) do 73 | case event do 74 | %EventCall{} -> handle_event_call(event, state) 75 | %EventReturnFrom{} -> handle_event_return_from(event, state) 76 | _ -> state 77 | end 78 | end 79 | 80 | def handle_event_call(%EventCall{pid: pid, mod: mod, fun: fun, arity: arity, 81 | ts: ts, message: m}, state) do 82 | enter_ts_ms = ts_to_ms(ts) 83 | key = inspect(pid) 84 | 85 | state = if !Map.get(state.started, key, false) and 86 | match_start_fun(state.start_mfa, {mod, fun, arity}) do 87 | put_in(state.started, Map.put(state.started, key, true)) 88 | else 89 | state 90 | end 91 | 92 | stack_entry = {:enter, {mod, fun, arity, m}, enter_ts_ms} 93 | push_to_stack_if_started(key, stack_entry, state) 94 | end 95 | 96 | defp match_start_fun(nil, _), do: true 97 | defp match_start_fun({mod, fun, arity}, {mod, fun, arity}), do: true 98 | defp match_start_fun({mod, fun, :_}, {mod, fun, _}), do: true 99 | defp match_start_fun({mod, :_, :_}, {mod, _, _}), do: true 100 | defp match_start_fun({:_, :_, :_}, {_, _, _}), do: true 101 | defp match_start_fun(_, _), do: false 102 | 103 | def handle_event_return_from(%EventReturnFrom{pid: pid, mod: mod, fun: fun, 104 | arity: arity, ts: ts, return_value: return_value}, state) do 105 | exit_ts_ms = ts_to_ms(ts) 106 | key = inspect(pid) 107 | 108 | val = if state.show_return, do: return_value, else: nil 109 | stack_entry = {:exit, {mod, fun, arity, val}, exit_ts_ms} 110 | state = push_to_stack_if_started(key, stack_entry, state) 111 | 112 | if Map.get(state.started, key, false) and 113 | Map.get(state.depth, key, 0) == 0 do 114 | put_in(state.started, Map.put(state.started, key, false)) 115 | else 116 | state 117 | end 118 | 119 | end 120 | 121 | defp push_to_stack_if_started(key, {dir, {mod, fun, arity, val}, ts}, state) do 122 | if Map.get(state.started, key, false) do 123 | if state.ignore_recursion do 124 | state.stacks 125 | |> Map.get(key, []) 126 | |> case do 127 | [{^dir, {^mod, ^fun, ^arity, _r}, _ts} | _] -> 128 | state 129 | _ -> 130 | push_to_stack(key, 131 | {dir, {mod, fun, arity, val}, ts}, 132 | state) 133 | end 134 | else 135 | push_to_stack(key, 136 | {dir, {mod, fun, arity, val}, ts}, 137 | state) 138 | end 139 | else 140 | state 141 | end 142 | end 143 | 144 | defp push_to_stack(key, {:enter, _, _} = stack, state) do 145 | state = if Map.get(state.depth, key, 0) < state.max_depth do 146 | new_stack = [stack | 147 | Map.get(state.stacks, key, [])] 148 | put_in(state.stacks, Map.put(state.stacks, key, new_stack)) 149 | else 150 | state 151 | end 152 | increase_depth(state, key) 153 | end 154 | defp push_to_stack(key, {:exit, _, _} = stack, state) do 155 | state = if Map.get(state.depth, key, 0) < state.max_depth + 1 do 156 | new_stack = [stack | 157 | Map.get(state.stacks, key, [])] 158 | put_in(state.stacks, Map.put(state.stacks, key, new_stack)) 159 | else 160 | state 161 | end 162 | increase_depth(state, key, -1) 163 | end 164 | 165 | defp increase_depth(state, key, incr \\ 1) do 166 | put_in(state.depth, 167 | Map.put(state.depth, key, Map.get(state.depth, key, 0) + incr)) 168 | end 169 | 170 | def handle_stop(state) do 171 | # get stack for each process 172 | state.stacks |> Enum.each(fn {pid, stack} -> 173 | stack 174 | |> Enum.reverse() 175 | |> Enum.reduce(0, fn 176 | {:enter, {mod, fun, arity, m}, _enter_ts_ms}, depth -> 177 | report_event(state, %Event{ 178 | type: :enter, 179 | depth: depth, 180 | pid: pid, 181 | mod: mod, 182 | fun: fun, 183 | arity: arity, 184 | message: m 185 | }) 186 | depth + 1 187 | {:exit, {mod, fun, arity, return_value}, _exit_ts_ms}, depth -> 188 | report_event(state, %Event{ 189 | type: :exit, 190 | depth: depth - 1, 191 | pid: pid, 192 | mod: mod, 193 | fun: fun, 194 | arity: arity, 195 | return_value: return_value 196 | }) 197 | depth - 1 198 | end) 199 | end) 200 | state 201 | end 202 | 203 | defp ts_to_ms({mega, seconds, us}) do 204 | (mega * 1_000_000 + seconds) * 1_000_000 + us # round(us/1000) 205 | end 206 | 207 | end 208 | -------------------------------------------------------------------------------- /lib/tracer/tools/tool_call_seq_event.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Tool.CallSeq.Event do 2 | @moduledoc """ 3 | Event generated by the CallSeq tool 4 | """ 5 | alias __MODULE__ 6 | 7 | defstruct type: nil, 8 | depth: 0, 9 | pid: nil, 10 | mod: nil, 11 | fun: nil, 12 | arity: nil, 13 | message: nil, 14 | return_value: nil 15 | 16 | defimpl String.Chars, for: Event do 17 | def to_string(%Event{type: :enter} = event) do 18 | String.duplicate(" ", event.depth) <> 19 | "-> #{inspect event.mod}.#{event.fun}/#{event.arity} " <> 20 | if is_nil(event.message), do: "", 21 | else: List.to_string(safe_inspect(event.message, event.depth + 5)) 22 | end 23 | def to_string(%Event{type: :exit} = event) do 24 | String.duplicate(" ", event.depth) <> 25 | "<- #{inspect event.mod}.#{event.fun}/#{event.arity} " <> 26 | if is_nil(event.return_value), do: "", 27 | else: List.to_string(safe_inspect(event.return_value, event.depth + 5)) 28 | end 29 | 30 | # defp message_to_string(nil, _depth), do: "" 31 | # defp message_to_string(term, depth) when is_list(term) do 32 | # term 33 | # |> Enum.map(fn 34 | # [key, val] -> {key, val} 35 | # other -> "#{inspect other}" 36 | # end) 37 | # |> safe_inspect(depth) 38 | # end 39 | 40 | # borrowed from https://github.com/fishcakez/dbg 41 | def safe_inspect(term, depth) do 42 | options = IEx.configuration() 43 | inspect_options = Keyword.get(options, :inspect, []) 44 | try do 45 | inspect(term, inspect_options) 46 | else 47 | formatted -> 48 | indent(formatted, depth) 49 | catch 50 | _, _ -> 51 | term 52 | |> inspect([records: false, structs: false] ++ inspect_options) 53 | |> indent(depth) 54 | end 55 | end 56 | 57 | defp indent(formatted, depth) do 58 | formatted 59 | |> :binary.split([<>], [:global]) 60 | |> Enum.map(&(["\n" <> String.duplicate(" ", depth) | &1])) 61 | end 62 | end 63 | 64 | end 65 | -------------------------------------------------------------------------------- /lib/tracer/tools/tool_count.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Tool.Count do 2 | @moduledoc """ 3 | Reports count type traces 4 | """ 5 | alias __MODULE__ 6 | alias Tracer.{EventCall, Probe, Tool.Count.Event, ProcessHelper} 7 | use Tracer.Tool 8 | 9 | defstruct counts: %{} 10 | 11 | def init(opts) when is_list(opts) do 12 | init_state = init_tool(%Count{}, opts, [:match]) 13 | 14 | case Keyword.get(opts, :match) do 15 | nil -> init_state 16 | matcher -> 17 | node = Keyword.get(opts, :node) 18 | process = init_state 19 | |> get_process() 20 | |> ProcessHelper.ensure_pid(node) 21 | 22 | all_child = ProcessHelper.find_all_children(process, node) 23 | probe_call = Probe.new(type: :call, 24 | process: [process | all_child], 25 | match: matcher) 26 | probe_spawn = Probe.new(type: :set_on_spawn, 27 | process: [process | all_child]) 28 | 29 | set_probes(init_state, [probe_call, probe_spawn]) 30 | end 31 | end 32 | 33 | def handle_event(event, state) do 34 | case event do 35 | %EventCall{message: message} -> 36 | key = message_to_tuple_list(message) 37 | new_count = Map.get(state.counts, key, 0) + 1 38 | put_in(state.counts, Map.put(state.counts, key, new_count)) 39 | _ -> state 40 | end 41 | end 42 | 43 | def handle_stop(state) do 44 | counts = state.counts 45 | |> Map.to_list() 46 | |> Enum.sort(&(elem(&1, 1) < elem(&2, 1))) 47 | 48 | report_event(state, %Event{ 49 | counts: counts 50 | }) 51 | 52 | state 53 | end 54 | 55 | defp message_to_tuple_list(term) when is_list(term) do 56 | term 57 | |> Enum.map(fn 58 | [key, val] -> {key, val} 59 | # [key, val] -> {key, inspect(val)} 60 | other -> {:_unknown, inspect(other)} 61 | end) 62 | end 63 | 64 | end 65 | -------------------------------------------------------------------------------- /lib/tracer/tools/tool_count_event.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Tool.Count.Event do 2 | @moduledoc """ 3 | Event generated by the Count tool 4 | """ 5 | alias __MODULE__ 6 | 7 | defstruct counts: [] 8 | 9 | defimpl String.Chars, for: Event do 10 | def to_string(event) do 11 | event.counts 12 | |> find_max_lengths() 13 | |> format_count_entries() 14 | |> Enum.map(fn {e, count} -> 15 | "\t#{String.pad_trailing(Integer.to_string(count), 15)}[#{e}]" 16 | end) 17 | |> Enum.join("\n") 18 | end 19 | 20 | defp find_max_lengths(list) do 21 | Enum.reduce(list, {nil, []}, 22 | fn {e, c}, {max, acc} -> 23 | max = max || Enum.map(e, fn _ -> 0 end) 24 | max = e 25 | |> Enum.zip(max) 26 | |> Enum.map(fn 27 | {{:_unknown, other}, m} -> 28 | max(m, String.length(other)) 29 | {{key, val}, m} -> 30 | max(m, String.length("#{Atom.to_string(key)}:#{inspect val}")) 31 | end) 32 | {max, acc ++ [{e, c}]} 33 | end) 34 | end 35 | 36 | defp format_count_entries({max, list}) do 37 | Enum.map(list, fn({e, c}) -> 38 | e_as_string = max 39 | |> Enum.zip(e) 40 | |> Enum.map(fn 41 | {m, {:_unknown, other}} -> 42 | String.pad_trailing(other, m) 43 | {m, {key, val}} -> 44 | String.pad_trailing("#{Atom.to_string(key)}:#{inspect val}", m) 45 | end) 46 | |> Enum.join(", ") 47 | {e_as_string, c} 48 | end) 49 | end 50 | end 51 | 52 | end 53 | -------------------------------------------------------------------------------- /lib/tracer/tools/tool_display.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Tool.Display do 2 | @moduledoc """ 3 | Reports display type tracing 4 | """ 5 | alias __MODULE__ 6 | alias Tracer.{Probe, ProcessHelper} 7 | use Tracer.Tool 8 | 9 | defstruct [] 10 | 11 | def init(opts) do 12 | init_state = init_tool(%Display{}, opts, [:match]) 13 | 14 | case Keyword.get(opts, :match) do 15 | nil -> init_state 16 | matcher -> 17 | node = Keyword.get(opts, :node) 18 | process = init_state 19 | |> get_process() 20 | |> ProcessHelper.ensure_pid(node) 21 | 22 | all_child = ProcessHelper.find_all_children(process, node) 23 | type = Keyword.get(opts, :type, :call) 24 | probe = Probe.new(type: type, 25 | process: [process | all_child], 26 | match: matcher) 27 | set_probes(init_state, [probe]) 28 | end 29 | end 30 | 31 | def handle_event(event, state) do 32 | report_event(state, event) 33 | state 34 | end 35 | 36 | end 37 | -------------------------------------------------------------------------------- /lib/tracer/tools/tool_duration.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Tool.Duration do 2 | @moduledoc """ 3 | Reports duration type traces 4 | """ 5 | alias __MODULE__ 6 | alias Tracer.{EventCall, EventReturnFrom, Matcher, Probe, 7 | Tool.Duration.Event, Collect, ProcessHelper} 8 | 9 | use Tracer.Tool 10 | 11 | defstruct durations: %{}, 12 | aggregation: nil, 13 | stacks: %{}, 14 | collect: nil 15 | 16 | def init(opts) when is_list(opts) do 17 | init_state = %Duration{} 18 | |> init_tool(opts, [:match, :aggregation]) 19 | |> Map.put(:aggregation, 20 | aggreg_fun(Keyword.get(opts, :aggregation, nil))) 21 | |> Map.put(:collect, Collect.new()) 22 | 23 | case Keyword.get(opts, :match) do 24 | nil -> init_state 25 | %Matcher{} = matcher -> 26 | ms_with_return_trace = matcher.ms 27 | |> Enum.map(fn {head, condit, body} -> 28 | {head, condit, [{:return_trace} | body]} 29 | end) 30 | matcher = put_in(matcher.ms, ms_with_return_trace) 31 | 32 | node = Keyword.get(opts, :node) 33 | process = init_state 34 | |> get_process() 35 | |> ProcessHelper.ensure_pid(node) 36 | 37 | all_child = ProcessHelper.find_all_children(process, node) 38 | probe_call = Probe.new(type: :call, 39 | process: [process | all_child], 40 | match: matcher) 41 | probe_spawn = Probe.new(type: :set_on_spawn, 42 | process: [process | all_child]) 43 | 44 | set_probes(init_state, [probe_call, probe_spawn]) 45 | end 46 | end 47 | 48 | def handle_event(event, state) do 49 | case event do 50 | %EventCall{pid: pid, mod: mod, fun: fun, arity: arity, 51 | ts: ts, message: c} -> 52 | ts_ms = ts_to_ms(ts) 53 | key = inspect(pid) 54 | new_stack = [{mod, fun, arity, ts_ms, c} | 55 | Map.get(state.stacks, key, [])] 56 | put_in(state.stacks, Map.put(state.stacks, key, new_stack)) 57 | 58 | %EventReturnFrom{pid: pid, mod: mod, fun: fun, arity: arity, ts: ts} -> 59 | exit_ts = ts_to_ms(ts) 60 | key = inspect(pid) 61 | case Map.get(state.stacks, key, []) do 62 | [] -> 63 | report_event(state, 64 | "stack empty for #{inspect mod}.#{fun}/#{arity}") 65 | state 66 | # ignore recursion calls 67 | [{^mod, ^fun, ^arity, _, _}, 68 | {^mod, ^fun, ^arity, entry_ts, c} | poped_stack] -> 69 | put_in(state.stacks, Map.put(state.stacks, key, 70 | [{mod, fun, arity, entry_ts, c} | poped_stack])) 71 | [{^mod, ^fun, ^arity, entry_ts, c} | poped_stack] -> 72 | duration = exit_ts - entry_ts 73 | 74 | event = %Event{ 75 | duration: duration, 76 | pid: pid, 77 | mod: mod, 78 | fun: fun, 79 | arity: arity, 80 | message: c 81 | } 82 | 83 | state 84 | |> handle_aggregation_if_needed(event) 85 | |> Map.put(:stacks, Map.put(state.stacks, key, poped_stack)) 86 | _ -> 87 | report_event(state, "entry point not found for" <> 88 | " #{inspect mod}.#{fun}/#{arity}") 89 | state 90 | end 91 | _ -> state 92 | end 93 | end 94 | 95 | def handle_stop(%Duration{aggregation: nil} = state), do: state 96 | def handle_stop(state) do 97 | state.collect 98 | |> Collect.get_collections() 99 | |> Enum.each(fn {{mod, fun, arity, message}, value} -> 100 | event = %Event{ 101 | duration: state.aggregation.(value), 102 | mod: mod, 103 | fun: fun, 104 | arity: arity, 105 | message: message 106 | } 107 | 108 | report_event(state, event) 109 | end) 110 | state 111 | end 112 | 113 | defp handle_aggregation_if_needed(%Duration{aggregation: nil} = state, 114 | event) do 115 | report_event(state, event) 116 | state 117 | end 118 | defp handle_aggregation_if_needed(state, 119 | %Event{mod: mod, fun: fun, arity: arity, 120 | message: message, duration: duration}) do 121 | collect = Collect.add_sample( 122 | state.collect, 123 | {mod, fun, arity, message}, 124 | duration) 125 | put_in(state.collect, collect) 126 | end 127 | 128 | defp ts_to_ms({mega, seconds, us}) do 129 | (mega * 1_000_000 + seconds) * 1_000_000 + us # round(us/1000) 130 | end 131 | 132 | defp aggreg_fun(:nil), do: nil 133 | defp aggreg_fun(:max), do: &Enum.max/1 134 | defp aggreg_fun(:mix), do: &Enum.min/1 135 | defp aggreg_fun(:sum), do: &Enum.sum/1 136 | defp aggreg_fun(:avg), do: fn list -> Enum.sum(list) / length(list) end 137 | defp aggreg_fun(:dist), do: fn list -> 138 | Enum.reduce(list, %{}, fn val, buckets -> 139 | index = log_bucket(val) 140 | Map.put(buckets, index, Map.get(buckets, index, 0) + 1) 141 | end) 142 | end 143 | defp aggreg_fun(other) do 144 | raise ArgumentError, message: "unsupported aggregation #{inspect other}" 145 | end 146 | 147 | defp log_bucket(x) do 148 | # + 0.01 avoid 0 to trap 149 | round(:math.pow(2, Float.floor(:math.log(x + 0.01) / :math.log(2)))) 150 | end 151 | end 152 | -------------------------------------------------------------------------------- /lib/tracer/tools/tool_duration_event.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Tool.Duration.Event do 2 | @moduledoc """ 3 | Event generated by the DurationTool 4 | """ 5 | alias __MODULE__ 6 | 7 | defstruct duration: 0, 8 | pid: nil, 9 | mod: nil, 10 | fun: nil, 11 | arity: nil, 12 | message: nil 13 | 14 | defimpl String.Chars, for: Event do 15 | def to_string(%Event{duration: d} = event) when is_integer(d) do 16 | duration_str = String.pad_trailing(Integer.to_string(event.duration), 17 | 20) 18 | "\t#{duration_str} " <> 19 | (if event.pid == nil, do: "", else: inspect(event.pid)) <> 20 | " #{inspect event.mod}.#{event.fun}/#{event.arity}" <> 21 | " #{message_to_string event.message}" 22 | end 23 | def to_string(event) do 24 | header = "#{inspect event.mod}.#{event.fun}/#{event.arity}" <> 25 | " #{message_to_string event.message}\n" 26 | step = (event.duration |> Map.values |> Enum.max) / 41 27 | title = String.pad_leading("value", 15) <> 28 | " ------------- Distribution ------------- count\n" 29 | body = event.duration 30 | |> Enum.map(fn {key, value} -> 31 | String.pad_leading(Integer.to_string(key), 15) <> 32 | " |" <> to_bar(value, step) <> " #{value}\n" 33 | end) |> Enum.join("") 34 | header <> title <> body 35 | end 36 | 37 | defp to_bar(value, step) do 38 | char_num = round(value / step) 39 | String.duplicate("@", char_num) <> String.duplicate(" ", 41 - char_num) 40 | end 41 | 42 | defp message_to_string(nil), do: "" 43 | defp message_to_string(term) when is_list(term) do 44 | term 45 | |> Enum.map(fn 46 | [key, val] -> {key, val} 47 | other -> "#{inspect other}" 48 | end) 49 | |> inspect() 50 | end 51 | end 52 | 53 | end 54 | -------------------------------------------------------------------------------- /lib/tracer/tools/tool_flame_graph.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Tool.FlameGraph do 2 | @moduledoc """ 3 | This tool generates a flame graph 4 | """ 5 | alias __MODULE__ 6 | alias Tracer.{Probe, ProcessHelper, 7 | EventCall, EventReturnTo, EventIn, EventOut} 8 | use Tracer.Tool 9 | import Tracer.Matcher 10 | 11 | @root_dir File.cwd! 12 | @flame_graph_script Path.join(~w(#{@root_dir} scripts gen_flame_graph.sh)) 13 | 14 | defstruct file_name: nil, 15 | ignore: nil, 16 | resolution: nil, 17 | max_depth: nil, 18 | process_state: %{} 19 | 20 | def init(opts) when is_list(opts) do 21 | init_state = %FlameGraph{} 22 | |> init_tool(opts, [:file_name, :resolution, :max_depth, :ignore]) 23 | |> Map.put(:file_name, 24 | Keyword.get(opts, :file_name, "flame_graph.svg")) 25 | |> Map.put(:resolution, 26 | Keyword.get(opts, :resolution, 1_000)) 27 | |> Map.put(:max_depth, 28 | Keyword.get(opts, :max_depth, 50)) 29 | |> Map.put(:ignore, 30 | Keyword.get(opts, :ignore, [])) 31 | 32 | node = Keyword.get(opts, :node) 33 | process = init_state 34 | |> get_process() 35 | |> ProcessHelper.ensure_pid(node) 36 | 37 | process = [process | ProcessHelper.find_all_children(process, node)] 38 | probe_call = Probe.new(type: :call, 39 | process: process, 40 | match: local _) 41 | probe_call = Probe.return_to(probe_call, true) 42 | 43 | probe_spawn = Probe.new(type: :set_on_spawn, 44 | process: process) 45 | 46 | probe_sched = Probe.new(type: :sched, 47 | process: process) 48 | 49 | set_probes(init_state, [probe_call, probe_spawn, probe_sched]) 50 | end 51 | 52 | def handle_event(event, state) do 53 | process_state = Map.get(state.process_state, 54 | event.pid, 55 | %{ 56 | stack: [], 57 | stack_acc: [], 58 | last_ts: 0 59 | }) 60 | 61 | put_in(state.process_state, 62 | Map.put(state.process_state, 63 | event.pid, 64 | handle_event_for_process(event, process_state))) 65 | end 66 | 67 | defp handle_event_for_process(event, state) do 68 | case event do 69 | %EventCall{} -> handle_event_call(event, state) 70 | %EventReturnTo{} -> handle_event_return_to(event, state) 71 | %EventIn{} -> handle_event_in(event, state) 72 | %EventOut{} -> handle_event_out(event, state) 73 | _ -> state 74 | end 75 | end 76 | 77 | defp handle_event_call(%EventCall{mod: m, fun: f, arity: a, ts: ts}, 78 | state) do 79 | ms_ts = ts_to_ms(ts) 80 | state 81 | |> report_stack(ms_ts) 82 | |> push_stack("#{inspect(m)}.#{sanitize_function_name(f)}/#{inspect(a)}") 83 | end 84 | 85 | defp handle_event_return_to(%EventReturnTo{mod: m, fun: f, arity: a, ts: ts}, 86 | state) do 87 | ms_ts = ts_to_ms(ts) 88 | state 89 | |> report_stack(ms_ts) 90 | |> pop_stack_to("#{inspect(m)}.#{sanitize_function_name(f)}/#{inspect(a)}") 91 | end 92 | 93 | defp handle_event_in(%EventIn{ts: ts}, 94 | state) do 95 | ms_ts = ts_to_ms(ts) 96 | state 97 | |> report_stack(ms_ts) 98 | |> pop_sleep() 99 | end 100 | 101 | defp handle_event_out(%EventOut{ts: ts}, 102 | state) do 103 | ms_ts = ts_to_ms(ts) 104 | state 105 | |> report_stack(ms_ts) 106 | |> push_stack("sleep") 107 | end 108 | 109 | def handle_stop(state) do 110 | stack_list = Enum.map(state.process_state, 111 | fn {pid, %{stack_acc: stack_acc}} -> 112 | stack_acc 113 | |> filter_max_depth(state.max_depth) 114 | |> filter_ignored(state.ignore) 115 | |> format_stacks() 116 | |> collapse_stacks() 117 | |> filter_below_resolution(state.resolution) 118 | |> format_with_pid(pid) 119 | |> Enum.sort 120 | end) 121 | 122 | {:ok, file} = File.open("/tmp/flame_graph.txt", [:write]) 123 | IO.write file, stack_list 124 | File.close(file) 125 | System.cmd(@flame_graph_script, ["/tmp/flame_graph.txt", state.file_name]) 126 | state 127 | end 128 | 129 | defp filter_ignored(stack_acc, ignored) when is_list(ignored) do 130 | Enum.filter(stack_acc, fn {stack, _time} -> 131 | !Enum.any?(stack, &Enum.member?(ignored, &1)) 132 | end) 133 | end 134 | defp filter_ignored(stack_acc, ignored), do: filter_ignored(stack_acc, [ignored]) 135 | 136 | defp filter_max_depth(stack_acc, max_depth) do 137 | Enum.filter(stack_acc, fn {stack, _time} -> 138 | length(stack) < max_depth 139 | end) 140 | end 141 | 142 | defp format_stacks(stack_acc) do 143 | Enum.map(stack_acc, fn {stack, ts} -> 144 | {(stack |> Enum.reverse() |> Enum.join(";")), ts} 145 | end) 146 | end 147 | 148 | defp collapse_stacks(stack_acc) do 149 | Enum.reduce(stack_acc, %{}, fn {stack, ts}, acc -> 150 | ts_acc = Map.get(acc, stack, 0) 151 | Map.put(acc, stack, ts_acc + ts) 152 | end) 153 | end 154 | 155 | defp filter_below_resolution(stack_acc, resolution) do 156 | Enum.filter(stack_acc, fn {_stack, time} -> 157 | div(time, resolution) != 0 158 | end) 159 | end 160 | 161 | defp format_with_pid(stack_acc, pid) do 162 | Enum.map(stack_acc, fn 163 | {"", time} -> "#{inspect(pid)} #{time}\n" 164 | {stack, time} -> "#{inspect(pid)};#{stack} #{time}\n" 165 | end) 166 | end 167 | 168 | # collapse recursion 169 | defp push_stack(%{stack: [top | _stack]} = state, top), do: state 170 | defp push_stack(%{stack: stack} = state, entry) do 171 | %{state | stack: [entry | stack]} 172 | end 173 | 174 | defp pop_stack_to(%{stack: stack} = state, entry) do 175 | # when entry != ":undefined.:undefined/0" do 176 | stack 177 | |> Enum.drop_while(fn stack_frame -> stack_frame != entry end) 178 | |> case do 179 | [] -> state 180 | new_stack -> 181 | %{state | stack: new_stack} 182 | end 183 | end 184 | # # undefined and only one entry 185 | # defp pop_stack_to(%{stack: [_entry | stack]} = state, entry) do 186 | # IO.puts "warning entry #{entry} did not match anything " 187 | # %{state | stack: stack} 188 | # end 189 | # defp pop_stack_to(%{stack: []} = state, _entry) do 190 | # # IO.puts "warning: poping empty for entry #{entry} stack: " 191 | # state 192 | # end 193 | # defp pop_stack_to(state, entry) do 194 | # IO.puts "warning entry #{entry} did not match anything " 195 | # state 196 | # end 197 | 198 | defp pop_sleep(%{stack: ["sleep" | stack]} = state) do 199 | %{state | stack: stack} 200 | end 201 | defp pop_sleep(state), do: state 202 | 203 | defp report_stack(%{last_ts: 0} = state, ts) do 204 | %{state | last_ts: ts} 205 | end 206 | defp report_stack(%{stack: stack, 207 | stack_acc: [{stack, stack_time} | stack_acc], 208 | last_ts: last_ts} = state, 209 | ts) do 210 | %{state | stack_acc: [{stack, stack_time + (ts - last_ts)} | stack_acc], 211 | last_ts: ts} 212 | end 213 | defp report_stack(%{stack: stack, stack_acc: stack_acc, 214 | last_ts: last_ts} = state, 215 | ts) do 216 | %{state | stack_acc: [{stack, ts - last_ts} | stack_acc], last_ts: ts} 217 | end 218 | 219 | defp ts_to_ms({mega, seconds, us}) do 220 | (mega * 1_000_000 + seconds) * 1_000_000 + us 221 | end 222 | 223 | defp sanitize_function_name(f) do 224 | to_string(f) 225 | end 226 | 227 | end 228 | -------------------------------------------------------------------------------- /lib/tracer_app.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.App do 2 | @moduledoc """ 3 | Tracer Application 4 | """ 5 | use Application 6 | 7 | def start(_type, _args) do 8 | Tracer.Supervisor.start_link() 9 | end 10 | 11 | end 12 | -------------------------------------------------------------------------------- /lib/tracer_macros.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Macros do 2 | @moduledoc """ 3 | Helper Macros 4 | """ 5 | 6 | defmacro delegate(fun, list) do 7 | quote do 8 | def unquote(fun)() do 9 | apply(Keyword.get(unquote(list), :to), 10 | Keyword.get(unquote(list), :as), []) 11 | end 12 | end 13 | end 14 | 15 | defmacro delegate_1(fun, list) do 16 | quote do 17 | def unquote(fun)(param) do 18 | apply(Keyword.get(unquote(list), :to), 19 | Keyword.get(unquote(list), :as), [param]) 20 | end 21 | end 22 | end 23 | 24 | # Pipe helpers 25 | defmacro tuple_x_and_fx(x, term) do 26 | quote do: {unquote(x), unquote(x) |> unquote(term)} 27 | end 28 | 29 | defmacro assign_to(res, target) do 30 | quote do: unquote(target) = unquote(res) 31 | end 32 | 33 | # defmacro create_match_spec() 34 | end 35 | -------------------------------------------------------------------------------- /lib/tracer_server.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Server do 2 | @moduledoc """ 3 | Orchestrates the tracing session 4 | """ 5 | 6 | use GenServer 7 | alias __MODULE__ 8 | alias Tracer.{AgentCmds, ToolServer, ProbeList, Tool} 9 | 10 | @server_name __MODULE__ 11 | defstruct tool_server_pid: nil, 12 | tracing: false, 13 | forward_pid: nil, 14 | probes: [], 15 | node: nil, 16 | agent_pids: [], 17 | tool: nil 18 | 19 | defmacro ensure_server_up(do: clauses) do 20 | quote do 21 | case Process.whereis(@server_name) do 22 | nil -> 23 | start() 24 | unquote(clauses) 25 | pid -> 26 | unquote(clauses) 27 | end 28 | end 29 | end 30 | 31 | def start_link(params) do 32 | GenServer.start_link(__MODULE__, params, [name: @server_name]) 33 | end 34 | 35 | def start do 36 | Tracer.Supervisor.start_server() 37 | end 38 | 39 | def stop do 40 | case Process.whereis(@server_name) do 41 | nil -> {:error, :not_running} 42 | pid -> 43 | GenServer.call(@server_name, :stop_tool) 44 | Tracer.Supervisor.stop_server(pid) 45 | end 46 | end 47 | 48 | [:stop_tool] 49 | |> Enum.each(fn cmd -> 50 | def unquote(cmd)() do 51 | ensure_server_up do 52 | GenServer.call(@server_name, unquote(cmd)) 53 | end 54 | end 55 | end) 56 | 57 | def set_tool(%{"__tool__": _} = tool) do 58 | with :ok <- Tool.valid?(tool) do 59 | ensure_server_up do 60 | GenServer.call(@server_name, {:set_tool, tool}) 61 | end 62 | end 63 | end 64 | def set_tool(_) do 65 | raise ArgumentError, message: "Argument is not a tool" 66 | end 67 | 68 | def start_tool(%{"__tool__": _} = tool) do 69 | with :ok <- Tool.valid?(tool) do 70 | ensure_server_up do 71 | GenServer.call(@server_name, {:start_tool, tool}) 72 | end 73 | end 74 | end 75 | 76 | def init(_params) do 77 | Process.flag(:trap_exit, true) 78 | {:ok, %Server{}} 79 | end 80 | 81 | def handle_call({:set_tool, tool}, _from, %Server{} = state) do 82 | {:reply, :ok, put_in(state.tool, tool)} 83 | end 84 | 85 | def handle_call({:start_tool, tool}, _from, %Server{} = state) do 86 | with %Server{} = state <- stop_if_tracing(state), 87 | %Server{} = state <- get_running_config(state, tool), 88 | :ok <- ProbeList.valid?(state.probes), 89 | ret when is_pid(ret) <- ToolServer.start(tool) do 90 | state = state 91 | |> Map.put(:tool_server_pid, ret) 92 | 93 | agent_opts = Tool.get_agent_opts(tool) 94 | node = Keyword.get(agent_opts, :node, state.node) 95 | {ret, new_state} = case AgentCmds.start(node, 96 | state.probes, 97 | [forward_pid: state.tool_server_pid] 98 | ++ agent_opts) do 99 | {:error, error} -> 100 | {{:error, error}, state} 101 | agent_pids -> 102 | new_state = state 103 | |> Map.put(:agent_pids, agent_pids) 104 | |> Map.put(:node, node) 105 | |> Map.put(:tracing, true) 106 | # TODO get a notification from agents 107 | :timer.sleep(5) 108 | report_message(state, 109 | :started_tracing, 110 | "started tracing") 111 | {:ok, new_state} 112 | end 113 | 114 | {:reply, ret, new_state} 115 | else 116 | error -> 117 | {:reply, error, state} 118 | end 119 | end 120 | 121 | def handle_call(:stop_tool, _from, %Server{} = state) do 122 | {ret, state} = handle_stop_trace(state) 123 | {:reply, ret, state} 124 | end 125 | 126 | def handle_info({:EXIT, pid, :done_reporting}, 127 | %Server{} = state) do 128 | if state.tool_server_pid == pid do 129 | {:noreply, put_in(state.tool_server_pid, nil)} 130 | else 131 | {:noreply, state} 132 | end 133 | end 134 | def handle_info({:EXIT, pid, {:done_tracing, exit_status}}, 135 | %Server{} = state) do 136 | state = handle_agent_exit(state, pid) 137 | report_message(state, 138 | {:done_tracing, exit_status}, 139 | "done tracing: #{inspect exit_status}") 140 | {:noreply, state} 141 | end 142 | def handle_info({:EXIT, pid, {:done_tracing, key, val}}, 143 | %Server{} = state) do 144 | state = handle_agent_exit(state, pid) 145 | report_message(state, 146 | {:done_tracing, key, val}, 147 | "done tracing: #{to_string(key)} #{val}") 148 | {:noreply, state} 149 | end 150 | def handle_info({:EXIT, pid, exit_code}, 151 | %Server{} = state) do 152 | state = handle_agent_exit(state, pid) 153 | report_message(state, 154 | {:done_tracing, exit_code}, 155 | "done tracing: #{inspect exit_code}") 156 | {:noreply, state} 157 | end 158 | 159 | defp get_running_config(state, tool) do 160 | tool 161 | |> Tool.get_probes() 162 | |> case do 163 | nil -> state 164 | probes when is_list(probes) -> 165 | put_in(state.probes, probes) 166 | probe -> 167 | put_in(state.probes, [probe]) 168 | end 169 | |> Map.put(:forward_pid, Tool.get_forward_to(tool)) 170 | end 171 | 172 | defp report_message(state, event, message) do 173 | if is_pid(state.forward_pid) do 174 | send state.forward_pid, event 175 | else 176 | IO.puts(message) 177 | end 178 | end 179 | 180 | defp handle_stop_trace(state) do 181 | {ret, state} = case AgentCmds.stop(state.agent_pids) do 182 | {:error, error} -> 183 | {{:error, error}, state} 184 | _ -> 185 | new_state = state 186 | |> Map.put(:agent_pids, []) 187 | |> Map.put(:node, nil) 188 | |> Map.put(:tracing, false) 189 | {:ok, new_state} 190 | end 191 | 192 | ToolServer.stop(state.tool_server_pid) 193 | state = put_in(state.tool_server_pid, nil) 194 | {ret, state} 195 | end 196 | 197 | defp handle_agent_exit(state, pid) do 198 | agent_pids = state.agent_pids -- [pid] 199 | state = put_in(state.agent_pids, agent_pids) 200 | if Enum.empty?(agent_pids) do 201 | {_ret, state} = handle_stop_trace(state) 202 | state 203 | else 204 | state 205 | end 206 | end 207 | 208 | defp stop_if_tracing(%Server{tracing: false} = state), do: state 209 | defp stop_if_tracing(state), do: elem(handle_stop_trace(state), 1) 210 | end 211 | -------------------------------------------------------------------------------- /lib/tracer_supervisor.ex: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Supervisor do 2 | @moduledoc """ 3 | Supervises Tracer.Server 4 | """ 5 | use Supervisor 6 | 7 | def start_link do 8 | Supervisor.start_link(__MODULE__, :ok, name: __MODULE__) 9 | end 10 | 11 | def init(:ok) do 12 | children = [ 13 | worker(Tracer.Server, [], restart: :temporary) 14 | ] 15 | 16 | supervise(children, strategy: :simple_one_for_one) 17 | end 18 | 19 | def start_server do 20 | Supervisor.start_child(__MODULE__, [[]]) 21 | end 22 | 23 | def stop_server(pid) do 24 | Supervisor.terminate_child(__MODULE__, pid) 25 | end 26 | 27 | end 28 | -------------------------------------------------------------------------------- /mix.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Mixfile do 2 | use Mix.Project 3 | 4 | @version File.read!("VERSION.md") |> String.trim 5 | 6 | def project do 7 | [app: :tracer, 8 | version: @version, 9 | elixir: "~> 1.4", 10 | build_embedded: Mix.env == :prod, 11 | start_permanent: Mix.env == :prod, 12 | description: description(), 13 | package: package(), 14 | deps: deps(), 15 | test_coverage: [tool: ExCoveralls], 16 | preferred_cli_env: ["coveralls": :test, "coveralls.detail": 17 | :test, "coveralls.post": :test, "coveralls.html": :test], 18 | docs: [extras: ["README.md"]]] 19 | end 20 | 21 | def application do 22 | [mod: {Tracer.App, []}, 23 | extra_applications: [:logger]] 24 | end 25 | 26 | defp deps do 27 | [ 28 | {:credo, "~> 0.8", only: [:dev, :test], runtime: false}, 29 | {:excoveralls, "~> 0.7", only: :test}, 30 | {:ex_doc, "~> 0.16", only: :dev, runtime: false} 31 | ] 32 | end 33 | 34 | defp description do 35 | """ 36 | Elixir Tracing Framework. 37 | """ 38 | end 39 | 40 | defp package do 41 | [files: ~w(lib test mix.exs README.md LICENSE.md VERSION.md), 42 | maintainers: ["Gabi Zuniga"], 43 | licenses: ["MIT"], 44 | links: %{"GitHub" => "https://github.com/gabiz/tracer"}] 45 | end 46 | 47 | end 48 | -------------------------------------------------------------------------------- /mix.lock: -------------------------------------------------------------------------------- 1 | %{"bunt": {:hex, :bunt, "0.2.0", "951c6e801e8b1d2cbe58ebbd3e616a869061ddadcc4863d0a2182541acae9a38", [], [], "hexpm"}, 2 | "certifi": {:hex, :certifi, "2.0.0", "a0c0e475107135f76b8c1d5bc7efb33cd3815cb3cf3dea7aefdd174dabead064", [], [], "hexpm"}, 3 | "credo": {:hex, :credo, "0.8.6", "335f723772d35da499b5ebfdaf6b426bfb73590b6fcbc8908d476b75f8cbca3f", [], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}], "hexpm"}, 4 | "earmark": {:hex, :earmark, "1.2.3", "206eb2e2ac1a794aa5256f3982de7a76bf4579ff91cb28d0e17ea2c9491e46a4", [], [], "hexpm"}, 5 | "ex_doc": {:hex, :ex_doc, "0.16.3", "cd2a4cfe5d26e37502d3ec776702c72efa1adfa24ed9ce723bb565f4c30bd31a", [], [{:earmark, "~> 1.1", [hex: :earmark, repo: "hexpm", optional: false]}], "hexpm"}, 6 | "excoveralls": {:hex, :excoveralls, "0.7.2", "f69ede8c122ccd3b60afc775348a53fc8c39fe4278aee2f538f0d81cc5e7ff3a", [], [{:exjsx, ">= 3.0.0", [hex: :exjsx, repo: "hexpm", optional: false]}, {:hackney, ">= 0.12.0", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm"}, 7 | "exjsx": {:hex, :exjsx, "4.0.0", "60548841e0212df401e38e63c0078ec57b33e7ea49b032c796ccad8cde794b5c", [], [{:jsx, "~> 2.8.0", [hex: :jsx, repo: "hexpm", optional: false]}], "hexpm"}, 8 | "hackney": {:hex, :hackney, "1.9.0", "51c506afc0a365868469dcfc79a9d0b94d896ec741cfd5bd338f49a5ec515bfe", [], [{:certifi, "2.0.0", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "5.1.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "1.0.1", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "1.0.2", [hex: :mimerl, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "1.1.1", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}], "hexpm"}, 9 | "idna": {:hex, :idna, "5.1.0", "d72b4effeb324ad5da3cab1767cb16b17939004e789d8c0ad5b70f3cea20c89a", [], [{:unicode_util_compat, "0.3.1", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm"}, 10 | "jsx": {:hex, :jsx, "2.8.2", "7acc7d785b5abe8a6e9adbde926a24e481f29956dd8b4df49e3e4e7bcc92a018", [], [], "hexpm"}, 11 | "metrics": {:hex, :metrics, "1.0.1", "25f094dea2cda98213cecc3aeff09e940299d950904393b2a29d191c346a8486", [], [], "hexpm"}, 12 | "mimerl": {:hex, :mimerl, "1.0.2", "993f9b0e084083405ed8252b99460c4f0563e41729ab42d9074fd5e52439be88", [], [], "hexpm"}, 13 | "ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.1", "28a4d65b7f59893bc2c7de786dec1e1555bd742d336043fe644ae956c3497fbe", [], [], "hexpm"}, 14 | "unicode_util_compat": {:hex, :unicode_util_compat, "0.3.1", "a1f612a7b512638634a603c8f401892afbf99b8ce93a45041f8aaca99cadb85e", [], [], "hexpm"}} 15 | -------------------------------------------------------------------------------- /scripts/flamegraph.pl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/perl -w 2 | # 3 | # flamegraph.pl flame stack grapher. 4 | # 5 | # This takes stack samples and renders a call graph, allowing hot functions 6 | # and codepaths to be quickly identified. Stack samples can be generated using 7 | # tools such as DTrace, perf, SystemTap, and Instruments. 8 | # 9 | # USAGE: ./flamegraph.pl [options] input.txt > graph.svg 10 | # 11 | # grep funcA input.txt | ./flamegraph.pl [options] > graph.svg 12 | # 13 | # Options are listed in the usage message (--help). 14 | # 15 | # The input is stack frames and sample counts formatted as single lines. Each 16 | # frame in the stack is semicolon separated, with a space and count at the end 17 | # of the line. These can be generated using DTrace with stackcollapse.pl, 18 | # and other tools using the stackcollapse variants. 19 | # 20 | # An optional extra column of counts can be provided to generate a differential 21 | # flame graph of the counts, colored red for more, and blue for less. This 22 | # can be useful when using flame graphs for non-regression testing. 23 | # See the header comment in the difffolded.pl program for instructions. 24 | # 25 | # The output graph shows relative presence of functions in stack samples. The 26 | # ordering on the x-axis has no meaning; since the data is samples, time order 27 | # of events is not known. The order used sorts function names alphabetically. 28 | # 29 | # While intended to process stack samples, this can also process stack traces. 30 | # For example, tracing stacks for memory allocation, or resource usage. You 31 | # can use --title to set the title to reflect the content, and --countname 32 | # to change "samples" to "bytes" etc. 33 | # 34 | # There are a few different palettes, selectable using --color. By default, 35 | # the colors are selected at random (except for differentials). Functions 36 | # called "-" will be printed gray, which can be used for stack separators (eg, 37 | # between user and kernel stacks). 38 | # 39 | # HISTORY 40 | # 41 | # This was inspired by Neelakanth Nadgir's excellent function_call_graph.rb 42 | # program, which visualized function entry and return trace events. As Neel 43 | # wrote: "The output displayed is inspired by Roch's CallStackAnalyzer which 44 | # was in turn inspired by the work on vftrace by Jan Boerhout". See: 45 | # https://blogs.oracle.com/realneel/entry/visualizing_callstacks_via_dtrace_and 46 | # 47 | # Copyright 2011 Joyent, Inc. All rights reserved. 48 | # Copyright 2011 Brendan Gregg. All rights reserved. 49 | # 50 | # CDDL HEADER START 51 | # 52 | # The contents of this file are subject to the terms of the 53 | # Common Development and Distribution License (the "License"). 54 | # You may not use this file except in compliance with the License. 55 | # 56 | # You can obtain a copy of the license at docs/cddl1.txt or 57 | # http://opensource.org/licenses/CDDL-1.0. 58 | # See the License for the specific language governing permissions 59 | # and limitations under the License. 60 | # 61 | # When distributing Covered Code, include this CDDL HEADER in each 62 | # file and include the License file at docs/cddl1.txt. 63 | # If applicable, add the following below this CDDL HEADER, with the 64 | # fields enclosed by brackets "[]" replaced with your own identifying 65 | # information: Portions Copyright [yyyy] [name of copyright owner] 66 | # 67 | # CDDL HEADER END 68 | # 69 | # 21-Nov-2013 Shawn Sterling Added consistent palette file option 70 | # 17-Mar-2013 Tim Bunce Added options and more tunables. 71 | # 15-Dec-2011 Dave Pacheco Support for frames with whitespace. 72 | # 10-Sep-2011 Brendan Gregg Created this. 73 | 74 | use strict; 75 | 76 | use Getopt::Long; 77 | use POSIX; 78 | 79 | # tunables 80 | my $encoding; 81 | my $fonttype = "Verdana"; 82 | my $imagewidth = 1200; # max width, pixels 83 | my $frameheight = 16; # max height is dynamic 84 | my $fontsize = 12; # base text size 85 | my $fontwidth = 0.59; # avg width relative to fontsize 86 | my $minwidth = 0.1; # min function width, pixels 87 | my $nametype = "Function:"; # what are the names in the data? 88 | my $countname = "samples"; # what are the counts in the data? 89 | my $colors = "hot"; # color theme 90 | my $bgcolor1 = "#eeeeee"; # background color gradient start 91 | my $bgcolor2 = "#eeeeb0"; # background color gradient stop 92 | my $nameattrfile; # file holding function attributes 93 | my $timemax; # (override the) sum of the counts 94 | my $factor = 1; # factor to scale counts by 95 | my $hash = 0; # color by function name 96 | my $palette = 0; # if we use consistent palettes (default off) 97 | my %palette_map; # palette map hash 98 | my $pal_file = "palette.map"; # palette map file name 99 | my $stackreverse = 0; # reverse stack order, switching merge end 100 | my $inverted = 0; # icicle graph 101 | my $negate = 0; # switch differential hues 102 | my $titletext = ""; # centered heading 103 | my $titledefault = "Flame Graph"; # overwritten by --title 104 | my $titleinverted = "Icicle Graph"; # " " 105 | 106 | GetOptions( 107 | 'fonttype=s' => \$fonttype, 108 | 'width=f' => \$imagewidth, 109 | 'height=i' => \$frameheight, 110 | 'encoding=s' => \$encoding, 111 | 'fontsize=f' => \$fontsize, 112 | 'fontwidth=f' => \$fontwidth, 113 | 'minwidth=f' => \$minwidth, 114 | 'title=s' => \$titletext, 115 | 'nametype=s' => \$nametype, 116 | 'countname=s' => \$countname, 117 | 'nameattr=s' => \$nameattrfile, 118 | 'total=s' => \$timemax, 119 | 'factor=f' => \$factor, 120 | 'colors=s' => \$colors, 121 | 'hash' => \$hash, 122 | 'cp' => \$palette, 123 | 'reverse' => \$stackreverse, 124 | 'inverted' => \$inverted, 125 | 'negate' => \$negate, 126 | ) or die < outfile.svg\n 128 | --title # change title text 129 | --width # width of image (default 1200) 130 | --height # height of each frame (default 16) 131 | --minwidth # omit smaller functions (default 0.1 pixels) 132 | --fonttype # font type (default "Verdana") 133 | --fontsize # font size (default 12) 134 | --countname # count type label (default "samples") 135 | --nametype # name type label (default "Function:") 136 | --colors # set color palette. choices are: hot (default), mem, io, 137 | # java, js, red, green, blue, yellow, purple, orange 138 | --hash # colors are keyed by function name hash 139 | --cp # use consistent palette (palette.map) 140 | --reverse # generate stack-reversed flame graph 141 | --inverted # icicle graph 142 | --negate # switch differential hues (blue<->red) 143 | 144 | eg, 145 | $0 --title="Flame Graph: malloc()" trace.txt > graph.svg 146 | USAGE_END 147 | 148 | $imagewidth = ceil($imagewidth); 149 | 150 | # internals 151 | my $ypad1 = $fontsize * 4; # pad top, include title 152 | my $ypad2 = $fontsize * 2 + 10; # pad bottom, include labels 153 | my $xpad = 10; # pad lefm and right 154 | my $framepad = 1; # vertical padding for frames 155 | my $depthmax = 0; 156 | my %Events; 157 | my %nameattr; 158 | 159 | if ($titletext eq "") { 160 | unless ($inverted) { 161 | $titletext = $titledefault; 162 | } else { 163 | $titletext = $titleinverted; 164 | } 165 | } 166 | 167 | if ($nameattrfile) { 168 | # The name-attribute file format is a function name followed by a tab then 169 | # a sequence of tab separated name=value pairs. 170 | open my $attrfh, $nameattrfile or die "Can't read $nameattrfile: $!\n"; 171 | while (<$attrfh>) { 172 | chomp; 173 | my ($funcname, $attrstr) = split /\t/, $_, 2; 174 | die "Invalid format in $nameattrfile" unless defined $attrstr; 175 | $nameattr{$funcname} = { map { split /=/, $_, 2 } split /\t/, $attrstr }; 176 | } 177 | } 178 | 179 | if ($colors eq "mem") { $bgcolor1 = "#eeeeee"; $bgcolor2 = "#e0e0ff"; } 180 | if ($colors eq "io") { $bgcolor1 = "#f8f8f8"; $bgcolor2 = "#e8e8e8"; } 181 | 182 | # SVG functions 183 | { package SVG; 184 | sub new { 185 | my $class = shift; 186 | my $self = {}; 187 | bless ($self, $class); 188 | return $self; 189 | } 190 | 191 | sub header { 192 | my ($self, $w, $h) = @_; 193 | my $enc_attr = ''; 194 | if (defined $encoding) { 195 | $enc_attr = qq{ encoding="$encoding"}; 196 | } 197 | $self->{svg} .= < 199 | 200 | 201 | SVG 202 | } 203 | 204 | sub include { 205 | my ($self, $content) = @_; 206 | $self->{svg} .= $content; 207 | } 208 | 209 | sub colorAllocate { 210 | my ($self, $r, $g, $b) = @_; 211 | return "rgb($r,$g,$b)"; 212 | } 213 | 214 | sub group_start { 215 | my ($self, $attr) = @_; 216 | 217 | my @g_attr = map { 218 | exists $attr->{$_} ? sprintf(qq/$_="%s"/, $attr->{$_}) : () 219 | } qw(class style onmouseover onmouseout onclick); 220 | push @g_attr, $attr->{g_extra} if $attr->{g_extra}; 221 | $self->{svg} .= sprintf qq/\n/, join(' ', @g_attr); 222 | 223 | $self->{svg} .= sprintf qq/%s<\/title>/, $attr->{title} 224 | if $attr->{title}; # should be first element within g container 225 | 226 | if ($attr->{href}) { 227 | my @a_attr; 228 | push @a_attr, sprintf qq/xlink:href="%s"/, $attr->{href} if $attr->{href}; 229 | # default target=_top else links will open within SVG 230 | push @a_attr, sprintf qq/target="%s"/, $attr->{target} || "_top"; 231 | push @a_attr, $attr->{a_extra} if $attr->{a_extra}; 232 | $self->{svg} .= sprintf qq//, join(' ', @a_attr); 233 | } 234 | } 235 | 236 | sub group_end { 237 | my ($self, $attr) = @_; 238 | $self->{svg} .= qq/<\/a>\n/ if $attr->{href}; 239 | $self->{svg} .= qq/<\/g>\n/; 240 | } 241 | 242 | sub filledRectangle { 243 | my ($self, $x1, $y1, $x2, $y2, $fill, $extra) = @_; 244 | $x1 = sprintf "%0.1f", $x1; 245 | $x2 = sprintf "%0.1f", $x2; 246 | my $w = sprintf "%0.1f", $x2 - $x1; 247 | my $h = sprintf "%0.1f", $y2 - $y1; 248 | $extra = defined $extra ? $extra : ""; 249 | $self->{svg} .= qq/\n/; 250 | } 251 | 252 | sub stringTTF { 253 | my ($self, $color, $font, $size, $angle, $x, $y, $str, $loc, $extra) = @_; 254 | $x = sprintf "%0.2f", $x; 255 | $loc = defined $loc ? $loc : "left"; 256 | $extra = defined $extra ? $extra : ""; 257 | $self->{svg} .= qq/$str<\/text>\n/; 258 | } 259 | 260 | sub svg { 261 | my $self = shift; 262 | return "$self->{svg}\n"; 263 | } 264 | 1; 265 | } 266 | 267 | sub namehash { 268 | # Generate a vector hash for the name string, weighting early over 269 | # later characters. We want to pick the same colors for function 270 | # names across different flame graphs. 271 | my $name = shift; 272 | my $vector = 0; 273 | my $weight = 1; 274 | my $max = 1; 275 | my $mod = 10; 276 | # if module name present, trunc to 1st char 277 | $name =~ s/.(.*?)`//; 278 | foreach my $c (split //, $name) { 279 | my $i = (ord $c) % $mod; 280 | $vector += ($i / ($mod++ - 1)) * $weight; 281 | $max += 1 * $weight; 282 | $weight *= 0.70; 283 | last if $mod > 12; 284 | } 285 | return (1 - $vector / $max) 286 | } 287 | 288 | sub color { 289 | my ($type, $hash, $name) = @_; 290 | my ($v1, $v2, $v3); 291 | 292 | if ($hash) { 293 | $v1 = namehash($name); 294 | $v2 = $v3 = namehash(scalar reverse $name); 295 | } else { 296 | $v1 = rand(1); 297 | $v2 = rand(1); 298 | $v3 = rand(1); 299 | } 300 | 301 | # theme palettes 302 | if (defined $type and $type eq "hot") { 303 | my $r = 205 + int(50 * $v3); 304 | my $g = 0 + int(230 * $v1); 305 | my $b = 0 + int(55 * $v2); 306 | return "rgb($r,$g,$b)"; 307 | } 308 | if (defined $type and $type eq "mem") { 309 | my $r = 0; 310 | my $g = 190 + int(50 * $v2); 311 | my $b = 0 + int(210 * $v1); 312 | return "rgb($r,$g,$b)"; 313 | } 314 | if (defined $type and $type eq "io") { 315 | my $r = 80 + int(60 * $v1); 316 | my $g = $r; 317 | my $b = 190 + int(55 * $v2); 318 | return "rgb($r,$g,$b)"; 319 | } 320 | 321 | # multi palettes 322 | if (defined $type and $type eq "java") { 323 | if ($name =~ /::/) { # C++ 324 | $type = "yellow"; 325 | } elsif ($name =~ m:/:) { # Java (match "/" in path) 326 | $type = "green" 327 | } else { # system 328 | $type = "red"; 329 | } 330 | # fall-through to color palettes 331 | } 332 | if (defined $type and $type eq "js") { 333 | if ($name =~ /::/) { # C++ 334 | $type = "yellow"; 335 | } elsif ($name =~ m:/:) { # JavaScript (match "/" in path) 336 | $type = "green" 337 | } elsif ($name =~ m/:/) { # JavaScript (match ":" in builtin) 338 | $type = "aqua" 339 | } elsif ($name =~ m/^ $/) { # Missing symbol 340 | $type = "green" 341 | } else { # system 342 | $type = "red"; 343 | } 344 | # fall-through to color palettes 345 | } 346 | 347 | # color palettes 348 | if (defined $type and $type eq "red") { 349 | my $r = 200 + int(55 * $v1); 350 | my $x = 50 + int(80 * $v1); 351 | return "rgb($r,$x,$x)"; 352 | } 353 | if (defined $type and $type eq "green") { 354 | my $g = 200 + int(55 * $v1); 355 | my $x = 50 + int(60 * $v1); 356 | return "rgb($x,$g,$x)"; 357 | } 358 | if (defined $type and $type eq "blue") { 359 | my $b = 205 + int(50 * $v1); 360 | my $x = 80 + int(60 * $v1); 361 | return "rgb($x,$x,$b)"; 362 | } 363 | if (defined $type and $type eq "yellow") { 364 | my $x = 175 + int(55 * $v1); 365 | my $b = 50 + int(20 * $v1); 366 | return "rgb($x,$x,$b)"; 367 | } 368 | if (defined $type and $type eq "purple") { 369 | my $x = 190 + int(65 * $v1); 370 | my $g = 80 + int(60 * $v1); 371 | return "rgb($x,$g,$x)"; 372 | } 373 | if (defined $type and $type eq "aqua") { 374 | my $r = 50 + int(60 * $v1); 375 | my $g = 165 + int(55 * $v1); 376 | my $b = 165 + int(55 * $v1); 377 | return "rgb($r,$g,$b)"; 378 | } 379 | if (defined $type and $type eq "orange") { 380 | my $r = 190 + int(65 * $v1); 381 | my $g = 90 + int(65 * $v1); 382 | return "rgb($r,$g,0)"; 383 | } 384 | 385 | return "rgb(0,0,0)"; 386 | } 387 | 388 | sub color_scale { 389 | my ($value, $max) = @_; 390 | my ($r, $g, $b) = (255, 255, 255); 391 | $value = -$value if $negate; 392 | if ($value > 0) { 393 | $g = $b = int(210 * ($max - $value) / $max); 394 | } elsif ($value < 0) { 395 | $r = $g = int(210 * ($max + $value) / $max); 396 | } 397 | return "rgb($r,$g,$b)"; 398 | } 399 | 400 | sub color_map { 401 | my ($colors, $func) = @_; 402 | if (exists $palette_map{$func}) { 403 | return $palette_map{$func}; 404 | } else { 405 | $palette_map{$func} = color($colors); 406 | return $palette_map{$func}; 407 | } 408 | } 409 | 410 | sub write_palette { 411 | open(FILE, ">$pal_file"); 412 | foreach my $key (sort keys %palette_map) { 413 | print FILE $key."->".$palette_map{$key}."\n"; 414 | } 415 | close(FILE); 416 | } 417 | 418 | sub read_palette { 419 | if (-e $pal_file) { 420 | open(FILE, $pal_file) or die "can't open file $pal_file: $!"; 421 | while ( my $line = ) { 422 | chomp($line); 423 | (my $key, my $value) = split("->",$line); 424 | $palette_map{$key}=$value; 425 | } 426 | close(FILE) 427 | } 428 | } 429 | 430 | my %Node; # Hash of merged frame data 431 | my %Tmp; 432 | 433 | # flow() merges two stacks, storing the merged frames and value data in %Node. 434 | sub flow { 435 | my ($last, $this, $v, $d) = @_; 436 | 437 | my $len_a = @$last - 1; 438 | my $len_b = @$this - 1; 439 | 440 | my $i = 0; 441 | my $len_same; 442 | for (; $i <= $len_a; $i++) { 443 | last if $i > $len_b; 444 | last if $last->[$i] ne $this->[$i]; 445 | } 446 | $len_same = $i; 447 | 448 | for ($i = $len_a; $i >= $len_same; $i--) { 449 | my $k = "$last->[$i];$i"; 450 | # a unique ID is constructed from "func;depth;etime"; 451 | # func-depth isn't unique, it may be repeated later. 452 | $Node{"$k;$v"}->{stime} = delete $Tmp{$k}->{stime}; 453 | if (defined $Tmp{$k}->{delta}) { 454 | $Node{"$k;$v"}->{delta} = delete $Tmp{$k}->{delta}; 455 | } 456 | delete $Tmp{$k}; 457 | } 458 | 459 | for ($i = $len_same; $i <= $len_b; $i++) { 460 | my $k = "$this->[$i];$i"; 461 | $Tmp{$k}->{stime} = $v; 462 | if (defined $d) { 463 | $Tmp{$k}->{delta} += $i == $len_b ? $d : 0; 464 | } 465 | } 466 | 467 | return $this; 468 | } 469 | 470 | # parse input 471 | my @Data; 472 | my $last = []; 473 | my $time = 0; 474 | my $delta = undef; 475 | my $ignored = 0; 476 | my $line; 477 | my $maxdelta = 1; 478 | 479 | # reverse if needed 480 | foreach (<>) { 481 | chomp; 482 | $line = $_; 483 | if ($stackreverse) { 484 | # there may be an extra samples column for differentials 485 | # XXX todo: redo these REs as one. It's repeated below. 486 | my ($stack, $samples) = (/^(.*)\s+?(\d+(?:\.\d*)?)$/); 487 | my $samples2 = undef; 488 | if ($stack =~ /^(.*)\s+?(\d+(?:\.\d*)?)$/) { 489 | $samples2 = $samples; 490 | ($stack, $samples) = $stack =~ (/^(.*)\s+?(\d+(?:\.\d*)?)$/); 491 | unshift @Data, join(";", reverse split(";", $stack)) . " $samples $samples2"; 492 | } else { 493 | unshift @Data, join(";", reverse split(";", $stack)) . " $samples"; 494 | } 495 | } else { 496 | unshift @Data, $line; 497 | } 498 | } 499 | 500 | # process and merge frames 501 | foreach (reverse @Data) { 502 | chomp; 503 | # process: folded_stack count 504 | # eg: func_a;func_b;func_c 31 505 | my ($stack, $samples) = (/^(.*)\s+?(\d+(?:\.\d*)?)$/); 506 | unless (defined $samples and defined $stack) { 507 | ++$ignored; 508 | next; 509 | } 510 | 511 | # there may be an extra samples column for differentials: 512 | my $samples2 = undef; 513 | if ($stack =~ /^(.*)\s+?(\d+(?:\.\d*)?)$/) { 514 | $samples2 = $samples; 515 | ($stack, $samples) = $stack =~ (/^(.*)\s+?(\d+(?:\.\d*)?)$/); 516 | } 517 | $delta = undef; 518 | if (defined $samples2) { 519 | $delta = $samples2 - $samples; 520 | $maxdelta = abs($delta) if abs($delta) > $maxdelta; 521 | } 522 | 523 | $stack =~ tr/<>/()/; 524 | 525 | # merge frames and populate %Node: 526 | $last = flow($last, [ '', split ";", $stack ], $time, $delta); 527 | 528 | if (defined $samples2) { 529 | $time += $samples2; 530 | } else { 531 | $time += $samples; 532 | } 533 | } 534 | flow($last, [], $time, $delta); 535 | 536 | warn "Ignored $ignored lines with invalid format\n" if $ignored; 537 | unless ($time) { 538 | warn "ERROR: No stack counts found\n"; 539 | my $im = SVG->new(); 540 | # emit an error message SVG, for tools automating flamegraph use 541 | my $imageheight = $fontsize * 5; 542 | $im->header($imagewidth, $imageheight); 543 | $im->stringTTF($im->colorAllocate(0, 0, 0), $fonttype, $fontsize + 2, 544 | 0.0, int($imagewidth / 2), $fontsize * 2, 545 | "ERROR: No valid input provided to flamegraph.pl.", "middle"); 546 | print $im->svg; 547 | exit 2; 548 | } 549 | if ($timemax and $timemax < $time) { 550 | warn "Specified --total $timemax is less than actual total $time, so ignored\n" 551 | if $timemax/$time > 0.02; # only warn is significant (e.g., not rounding etc) 552 | undef $timemax; 553 | } 554 | $timemax ||= $time; 555 | 556 | my $widthpertime = ($imagewidth - 2 * $xpad) / $timemax; 557 | my $minwidth_time = $minwidth / $widthpertime; 558 | 559 | # prune blocks that are too narrow and determine max depth 560 | while (my ($id, $node) = each %Node) { 561 | my ($func, $depth, $etime) = split ";", $id; 562 | my $stime = $node->{stime}; 563 | die "missing start for $id" if not defined $stime; 564 | 565 | if (($etime-$stime) < $minwidth_time) { 566 | delete $Node{$id}; 567 | next; 568 | } 569 | $depthmax = $depth if $depth > $depthmax; 570 | } 571 | 572 | # draw canvas, and embed interactive JavaScript program 573 | my $imageheight = ($depthmax * $frameheight) + $ypad1 + $ypad2; 574 | my $im = SVG->new(); 575 | $im->header($imagewidth, $imageheight); 576 | my $inc = < 578 | 579 | 580 | 581 | 582 | 583 | 586 | 750 | INC 751 | $im->include($inc); 752 | $im->filledRectangle(0, 0, $imagewidth, $imageheight, 'url(#background)'); 753 | my ($white, $black, $vvdgrey, $vdgrey) = ( 754 | $im->colorAllocate(255, 255, 255), 755 | $im->colorAllocate(0, 0, 0), 756 | $im->colorAllocate(40, 40, 40), 757 | $im->colorAllocate(160, 160, 160), 758 | ); 759 | $im->stringTTF($black, $fonttype, $fontsize + 5, 0.0, int($imagewidth / 2), $fontsize * 2, $titletext, "middle"); 760 | $im->stringTTF($black, $fonttype, $fontsize, 0.0, $xpad, $imageheight - ($ypad2 / 2), " ", "", 'id="details"'); 761 | $im->stringTTF($black, $fonttype, $fontsize, 0.0, $xpad, $fontsize * 2, 762 | "Reset Zoom", "", 'id="unzoom" onclick="unzoom()" style="opacity:0.0;cursor:pointer"'); 763 | 764 | if ($palette) { 765 | read_palette(); 766 | } 767 | 768 | # draw frames 769 | while (my ($id, $node) = each %Node) { 770 | my ($func, $depth, $etime) = split ";", $id; 771 | my $stime = $node->{stime}; 772 | my $delta = $node->{delta}; 773 | 774 | $etime = $timemax if $func eq "" and $depth == 0; 775 | 776 | my $x1 = $xpad + $stime * $widthpertime; 777 | my $x2 = $xpad + $etime * $widthpertime; 778 | my ($y1, $y2); 779 | unless ($inverted) { 780 | $y1 = $imageheight - $ypad2 - ($depth + 1) * $frameheight + $framepad; 781 | $y2 = $imageheight - $ypad2 - $depth * $frameheight; 782 | } else { 783 | $y1 = $ypad1 + $depth * $frameheight; 784 | $y2 = $ypad1 + ($depth + 1) * $frameheight - $framepad; 785 | } 786 | 787 | my $samples = sprintf "%.0f", ($etime - $stime) * $factor; 788 | (my $samples_txt = $samples) # add commas per perlfaq5 789 | =~ s/(^[-+]?\d+?(?=(?>(?:\d{3})+)(?!\d))|\G\d{3}(?=\d))/$1,/g; 790 | 791 | my $info; 792 | if ($func eq "" and $depth == 0) { 793 | $info = "all ($samples_txt $countname, 100%)"; 794 | } else { 795 | my $pct = sprintf "%.2f", ((100 * $samples) / ($timemax * $factor)); 796 | my $escaped_func = $func; 797 | $escaped_func =~ s/&/&/g; 798 | $escaped_func =~ s//>/g; 800 | unless (defined $delta) { 801 | $info = "$escaped_func ($samples_txt $countname, $pct%)"; 802 | } else { 803 | my $deltapct = sprintf "%.2f", ((100 * $delta) / ($timemax * $factor)); 804 | $deltapct = $delta > 0 ? "+$deltapct" : $deltapct; 805 | $info = "$escaped_func ($samples_txt $countname, $pct%; $deltapct%)"; 806 | } 807 | } 808 | 809 | my $nameattr = { %{ $nameattr{$func}||{} } }; # shallow clone 810 | $nameattr->{class} ||= "func_g"; 811 | $nameattr->{onmouseover} ||= "s('".$info."')"; 812 | $nameattr->{onmouseout} ||= "c()"; 813 | $nameattr->{onclick} ||= "zoom(this)"; 814 | $nameattr->{title} ||= $info; 815 | $im->group_start($nameattr); 816 | 817 | my $color; 818 | if ($func eq "-") { 819 | $color = $vdgrey; 820 | } elsif ($func =~ /sleep/ || $func eq "SLEEP") { 821 | $color = color("blue", $hash, $func); 822 | } elsif (defined $delta) { 823 | $color = color_scale($delta, $maxdelta); 824 | } elsif ($palette) { 825 | $color = color_map($colors, $func); 826 | } else { 827 | $color = color($colors, $hash, $func); 828 | } 829 | $im->filledRectangle($x1, $y1, $x2, $y2, $color, 'rx="2" ry="2"'); 830 | 831 | my $chars = int( ($x2 - $x1) / ($fontsize * $fontwidth)); 832 | my $text = ""; 833 | if ($chars >= 3) { # room for one char plus two dots 834 | $text = substr $func, 0, $chars; 835 | substr($text, -2, 2) = ".." if $chars < length $func; 836 | $text =~ s/&/&/g; 837 | $text =~ s//>/g; 839 | } 840 | $im->stringTTF($black, $fonttype, $fontsize, 0.0, $x1 + 3, 3 + ($y1 + $y2) / 2, $text, ""); 841 | 842 | $im->group_end($nameattr); 843 | } 844 | 845 | print $im->svg; 846 | 847 | if ($palette) { 848 | write_palette(); 849 | } 850 | 851 | # vim: ts=8 sts=8 sw=8 noexpandtab 852 | -------------------------------------------------------------------------------- /scripts/gen_flame_graph.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # cd "$( dirname "${BASH_SOURCE[0]}" )" 3 | $( dirname "${BASH_SOURCE[0]}" )/flamegraph.pl $1 > $2 4 | -------------------------------------------------------------------------------- /test/test_helper.exs: -------------------------------------------------------------------------------- 1 | ExUnit.start(exclude: [:remote_node, :timing]) 2 | -------------------------------------------------------------------------------- /test/tracer/agent_cmds_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.AgentCmds.Test do 2 | use ExUnit.Case 3 | alias Tracer.{AgentCmds, Probe} 4 | import Tracer.Matcher 5 | 6 | # Helper 7 | def test_tracer_proc(opts) do 8 | receive do 9 | event -> 10 | forward_pid = Keyword.get(opts, :forward_to) 11 | # IO.puts ("tracing handler: forward_pid #{inspect forward_pid} event #{inspect event}") 12 | if is_pid(forward_pid) do 13 | send forward_pid, event 14 | end 15 | if Keyword.get(opts, :print, false) do 16 | IO.puts(inspect event) 17 | end 18 | test_tracer_proc(opts) 19 | end 20 | end 21 | 22 | test "run enables probe and starts tracing and stop ends it" do 23 | my_pid = self() 24 | 25 | probe = Probe.new(type: :send) |> Probe.add_process(self()) 26 | 27 | # Run 28 | tracer_pid = spawn fn -> test_tracer_proc(forward_to: my_pid) end 29 | res = AgentCmds.run([probe], tracer: tracer_pid) 30 | assert is_list(res) and length(res) > 0 31 | 32 | send self(), :foo 33 | 34 | assert_receive(:foo) 35 | assert_receive({:trace_ts, ^my_pid, :send, :foo, ^my_pid, _}) 36 | refute_receive({:trace_ts, ^my_pid, :send, :foo, ^my_pid, _}) 37 | 38 | # Stop 39 | res = AgentCmds.stop_run() 40 | assert res > 0 41 | 42 | send self(), :foo_one_more_time 43 | refute_receive({:trace_ts, _, _, _, _, _}) 44 | 45 | end 46 | 47 | test "run full call tracing" do 48 | my_pid = self() 49 | 50 | probe = Probe.new( 51 | type: :call, 52 | process: self(), 53 | match: global do Map.new(%{items: [a, b]}) -> message(a, b) end) 54 | 55 | # Run 56 | tracer_pid = spawn fn -> test_tracer_proc(forward_to: my_pid) end 57 | res = AgentCmds.run([probe], tracer: tracer_pid) 58 | assert is_list(res) and length(res) > 0 59 | 60 | # no match 61 | Map.new(%{other_key: [1, 2]}) 62 | refute_receive({:trace_ts, ^my_pid, :call, 63 | {Map, :new, 1}, _, _}) 64 | 65 | # valid match - ignore timestamps 66 | Map.new(%{items: [1, 2]}) 67 | assert_receive({:trace_ts, ^my_pid, :call, 68 | {Map, :new, 1}, [[:a, 1], [:b, 2]], _}) 69 | 70 | res = AgentCmds.stop_run() 71 | 72 | assert res > 0 73 | 74 | # not expeting more events 75 | Map.new(%{items: [1, 2]}) 76 | refute_receive({:trace_ts, _, _, _, _, _}) 77 | end 78 | 79 | test "get_start_cmds generates trace command list" do 80 | my_pid = self() 81 | 82 | probe = Probe.new( 83 | type: :call, 84 | process: my_pid, 85 | match: global do Map.new(%{items: [a, b]}) -> message(a, b) end) 86 | 87 | tracer_pid = spawn fn -> test_tracer_proc(forward_to: my_pid) end 88 | [trace_pattern_cmd, trace_cmd] = 89 | AgentCmds.get_start_cmds([probe], tracer: tracer_pid) 90 | 91 | assert trace_pattern_cmd == [ 92 | fun: &:erlang.trace_pattern/3, 93 | mfa: {Map, :new, 1}, 94 | match_spec: [{[%{items: [:"$1", :"$2"]}], [], 95 | [message: [[:a, :"$1"], [:b, :"$2"]]]}], 96 | flag_list: [:global]] 97 | 98 | assert trace_cmd == [ 99 | fun: &:erlang.trace/3, 100 | pid_port_spec: my_pid, 101 | how: true, 102 | flag_list: [{:tracer, tracer_pid}, :call, :arity, :timestamp]] 103 | 104 | end 105 | 106 | test "start and stop tracing" do 107 | Process.flag(:trap_exit, true) 108 | my_pid = self() 109 | 110 | probe = Probe.new( 111 | type: :call, 112 | process: self(), 113 | match: global do Map.new(%{items: [a, b]}) -> message(a, b) end) 114 | 115 | tracer_pid = spawn fn -> test_tracer_proc(forward_to: my_pid) end 116 | agent_pids = AgentCmds.start(nil, [probe], forward_pid: tracer_pid) 117 | assert is_list(agent_pids) and length(agent_pids) > 0 118 | 119 | # no match 120 | Map.new(%{other_key: [1, 2]}) 121 | refute_receive({:trace_ts, ^my_pid, :call, 122 | {Map, :new, 1}, _, _}) 123 | 124 | # valid match - ignore timestamps 125 | Map.new(%{items: [1, 2]}) 126 | assert_receive({:trace_ts, ^my_pid, :call, 127 | {Map, :new, 1}, [[:a, 1], [:b, 2]], _}) 128 | 129 | :ok = AgentCmds.stop(agent_pids) 130 | 131 | :timer.sleep(50) 132 | # not expeting more events 133 | Map.new(%{items: [1, 2]}) 134 | refute_receive({:trace_ts, _, _, _, _, _}) 135 | end 136 | 137 | def local_function(a) do 138 | :timer.sleep(1) 139 | a 140 | end 141 | 142 | test "trace local function" do 143 | Process.flag(:trap_exit, true) 144 | my_pid = self() 145 | 146 | probe = Probe.new( 147 | type: :call, 148 | process: self(), 149 | # with_fun: {Tracer.Tracer.Test, :local_function, 1}, 150 | # match: local do (a) -> message(a) end) 151 | match: local do Tracer.AgentCmds.Test.local_function(a) -> message(a) end) 152 | 153 | agent_pids = AgentCmds.start(nil, [probe], forward_pid: self()) 154 | assert is_list(agent_pids) and length(agent_pids) > 0 155 | 156 | :timer.sleep(50) 157 | # call local_function 158 | local_function(1) 159 | 160 | assert_receive({:trace_ts, ^my_pid, :call, 161 | {Tracer.AgentCmds.Test, :local_function, 1}, [[:a, 1]], _}) 162 | 163 | :ok = AgentCmds.stop(agent_pids) 164 | 165 | :timer.sleep(50) 166 | # not expeting more events 167 | local_function(1) 168 | refute_receive({:trace_ts, _, _, _, _, _}) 169 | end 170 | 171 | end 172 | -------------------------------------------------------------------------------- /test/tracer/clause_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Clause.Test do 2 | use ExUnit.Case 3 | alias Tracer.Clause 4 | 5 | test "new returns an empty clause" do 6 | assert Clause.new() == %Clause{} 7 | end 8 | 9 | test "put_mfa returns error when arguments are invalid" do 10 | res = Clause.new() 11 | |> Clause.put_mfa(5, %{}) 12 | 13 | assert res == {:error, :invalid_mfa} 14 | end 15 | 16 | test "put_mfa stores mfa in clause and set type to :call" do 17 | clause = Clause.new() 18 | |> Clause.put_mfa(Map, :get, 2) 19 | 20 | assert Clause.get_mfa(clause) == {Map, :get, 2} 21 | assert Clause.get_type(clause) == :call 22 | end 23 | 24 | test "put_mfa accepts 0, 1, or 2 arguments" do 25 | clause = Clause.new() 26 | |> Clause.put_mfa(Map, :get) 27 | 28 | assert Clause.get_mfa(clause) == {Map, :get, :_} 29 | 30 | clause = Clause.put_mfa(clause, Map) 31 | assert Clause.get_mfa(clause) == {Map, :_, :_} 32 | 33 | clause = Clause.put_mfa(clause) 34 | assert Clause.get_mfa(clause) == {:_, :_, :_} 35 | end 36 | 37 | test "put_fun accepts an external function" do 38 | clause = Clause.new() 39 | |> Clause.put_fun(&Map.get/3) 40 | 41 | assert Clause.get_mfa(clause) == {Map, :get, 3} 42 | end 43 | 44 | test "valid? checks that mfa has been set" do 45 | res = Clause.new() 46 | |> Clause.valid?() 47 | 48 | assert res == {:error, :missing_mfa} 49 | end 50 | 51 | test "apply validates clause before applying clause" do 52 | res = Clause.new() 53 | |> Clause.apply() 54 | 55 | assert res == {:error, :missing_mfa} 56 | end 57 | 58 | test "apply stores the number of matches in matches" do 59 | clause = Clause.new() 60 | |> Clause.put_mfa(Map, :get, 2) 61 | |> Clause.apply() 62 | 63 | assert Clause.matches(clause) == 1 64 | end 65 | 66 | test "apply with not_remove equals to false removes the clause" do 67 | clause = Clause.new() 68 | |> Clause.put_mfa(Map, :get, 2) 69 | |> Clause.apply() 70 | 71 | assert Clause.matches(clause) == 1 72 | 73 | match_spec = :erlang.trace_info({Map, :get, 2}, :match_spec) 74 | assert match_spec == {:match_spec, []} 75 | 76 | clause = Clause.apply(clause, false) 77 | assert Clause.matches(clause) == 0 78 | match_spec = :erlang.trace_info({Map, :get, 2}, :match_spec) 79 | assert match_spec == {:match_spec, false} 80 | end 81 | 82 | test "valid_flags? return error when a flag is invalid" do 83 | assert Clause.valid_flags?([:global,:local]) == :ok 84 | res = Clause.valid_flags?([:global, :foo, :bar]) 85 | assert res == {:error, [invalid_clause_flag: :bar, 86 | invalid_clause_flag: :foo]} 87 | end 88 | 89 | test "set_flags check for valid flags" do 90 | res = Clause.new() 91 | |> Clause.set_flags([:global, :foo, :bar]) 92 | 93 | assert res == {:error, [invalid_clause_flag: :bar, 94 | invalid_clause_flag: :foo]} 95 | end 96 | 97 | test "set_flags sets the flags and get_flags retrieve them" do 98 | flags = Clause.new() 99 | |> Clause.set_flags([:global, :call_count]) 100 | |> Clause.get_flags() 101 | 102 | assert flags == [:global, :call_count] 103 | end 104 | 105 | test "get_trace_cmd includes expected parameters" do 106 | cmd = Clause.new() 107 | |> Clause.put_mfa(Map, :get, 2) 108 | |> Clause.add_matcher([{[:"$1", :"$2"], [is_atom: :"$2"], [message: [[:y, :"$2"]]]}]) 109 | |> Clause.set_flags([:global, :call_count]) 110 | |> Clause.get_trace_cmd() 111 | 112 | assert Keyword.get(cmd, :fun) == &:erlang.trace_pattern/3 113 | assert Keyword.get(cmd, :mfa) == {Map, :get, 2} 114 | assert Keyword.get(cmd, :match_spec) == [{[:"$1", :"$2"], [is_atom: :"$2"], [message: [[:y, :"$2"]]]}] 115 | assert Keyword.get(cmd, :flag_list) == [:global, :call_count] 116 | end 117 | 118 | test "get_trace_cmd raises an exception when the clause is invalid" do 119 | clause = Clause.new() 120 | |> Clause.add_matcher([{[:"$1", :"$2"], [is_atom: :"$2"], [message: [[:y, :"$2"]]]}]) 121 | |> Clause.set_flags([:global, :call_count]) 122 | 123 | assert_raise RuntimeError, "invalid clause {:error, :missing_mfa}", fn -> 124 | Clause.get_trace_cmd(clause) 125 | end 126 | end 127 | 128 | end 129 | -------------------------------------------------------------------------------- /test/tracer/collect_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Collect.Test do 2 | use ExUnit.Case 3 | alias Tracer.Collect 4 | 5 | test "add_sample() collects events" do 6 | collection = Collect.new() 7 | |> Collect.add_sample(:a, :foo) 8 | |> Collect.add_sample(:a, :bar) 9 | assert collection.collections == %{a: [:bar, :foo]} 10 | end 11 | 12 | test "get_collections() returns the collections" do 13 | collection = Collect.new() 14 | |> Collect.add_sample(:a, :foo) 15 | |> Collect.add_sample(:a, :bar) 16 | |> Collect.get_collections() 17 | 18 | assert collection == [{:a, [:foo, :bar]}] 19 | end 20 | 21 | test "get_collections() handles multile keys" do 22 | collection = Collect.new() 23 | |> Collect.add_sample(:a, :foo) 24 | |> Collect.add_sample(:a, :bar) 25 | |> Collect.add_sample(:b, :baz) 26 | |> Collect.get_collections() 27 | 28 | assert collection == [{:a, [:foo, :bar]}, {:b, [:baz]}] 29 | end 30 | 31 | end 32 | -------------------------------------------------------------------------------- /test/tracer/handler_agent_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.HandlerAgent.Test do 2 | use ExUnit.Case 3 | alias Tracer.HandlerAgent 4 | 5 | test "start() creates handler_agent process" do 6 | pid = HandlerAgent.start() 7 | assert is_pid(pid) 8 | assert Process.alive?(pid) 9 | end 10 | 11 | test "start() creates pid_handler process" do 12 | pid = HandlerAgent.start() 13 | send pid, {:get_handler_pid, self()} 14 | assert_receive {:handler_pid, handler_pid} 15 | assert is_pid(handler_pid) 16 | assert Process.alive?(handler_pid) 17 | end 18 | 19 | test "start() stores the handler_pid options" do 20 | pid = HandlerAgent.start(max_message_count: 123, 21 | max_queue_size: 456, 22 | event_callback: &Map.new/1) 23 | 24 | send pid, {:get_pid_handler_opts, self()} 25 | assert_receive {:pid_handler_opts, pid_handler_opts} 26 | assert Keyword.get(pid_handler_opts, :max_message_count) == 123 27 | assert Keyword.get(pid_handler_opts, :max_queue_size) == 456 28 | assert Keyword.get(pid_handler_opts, :event_callback) == &Map.new/1 29 | end 30 | 31 | test "start() sets correctly the forward_pid option" do 32 | pid = HandlerAgent.start(forward_pid: self()) 33 | 34 | send pid, {:get_pid_handler_opts, self()} 35 | assert_receive {:pid_handler_opts, pid_handler_opts} 36 | assert Keyword.get(pid_handler_opts, :event_callback) == 37 | {&HandlerAgent.forwarding_handler_callback/2, self()} 38 | end 39 | 40 | test "agent_handler process finishes after timeout" do 41 | Process.flag(:trap_exit, true) 42 | pid = HandlerAgent.start(max_tracing_time: 50) 43 | assert Process.alive?(pid) 44 | assert_receive({:EXIT, ^pid, {:done_tracing, :tracing_timeout, 50}}) 45 | refute Process.alive?(pid) 46 | end 47 | 48 | test "agent_handler process finishes after message count hits limit" do 49 | Process.flag(:trap_exit, true) 50 | pid = HandlerAgent.start(max_message_count: 1) 51 | assert Process.alive?(pid) 52 | 53 | send pid, {:get_handler_pid, self()} 54 | assert_receive {:handler_pid, handler_pid} 55 | assert Process.alive?(handler_pid) 56 | 57 | send handler_pid, {:trace, :foo} 58 | 59 | assert_receive({:EXIT, ^pid, {:done_tracing, :max_message_count, 1}}) 60 | refute Process.alive?(pid) 61 | refute Process.alive?(handler_pid) 62 | end 63 | 64 | test "agent_handler process finishes after max queue size triggers" do 65 | Process.flag(:trap_exit, true) 66 | pid = HandlerAgent.start(max_queue_size: 1, 67 | event_callback: fn _event -> 68 | :timer.sleep(20); 69 | :ok end) 70 | assert Process.alive?(pid) 71 | 72 | send pid, {:get_handler_pid, self()} 73 | assert_receive {:handler_pid, handler_pid} 74 | assert Process.alive?(handler_pid) 75 | 76 | send handler_pid, {:trace, :foo} 77 | send handler_pid, {:trace, :bar} 78 | send handler_pid, {:trace, :foo_bar} 79 | 80 | assert_receive({:EXIT, ^pid, {:done_tracing, :message_queue_size, _}}) 81 | refute Process.alive?(pid) 82 | refute Process.alive?(handler_pid) 83 | end 84 | 85 | test "stop() aborts the tracing and processes terminate" do 86 | Process.flag(:trap_exit, true) 87 | pid = HandlerAgent.start() 88 | assert Process.alive?(pid) 89 | 90 | send pid, {:get_handler_pid, self()} 91 | assert_receive {:handler_pid, handler_pid} 92 | assert Process.alive?(handler_pid) 93 | 94 | HandlerAgent.stop(pid) 95 | 96 | assert_receive({:EXIT, ^pid, {:done_tracing, :stop_command}}) 97 | refute Process.alive?(pid) 98 | refute Process.alive?(handler_pid) 99 | end 100 | 101 | @tag :remote_node 102 | test "start agent_handler in remote node" do 103 | :net_kernel.start([:"local@127.0.0.1"]) 104 | 105 | remote_node = "remote#{Enum.random(1..100)}@127.0.0.1" 106 | remote_node_a = String.to_atom(remote_node) 107 | spawn(fn -> 108 | System.cmd("elixir", ["--name", remote_node, "-e", ":timer.sleep(5_000)"]) 109 | end) 110 | 111 | :timer.sleep(500) 112 | # check if remote node is up 113 | case :net_adm.ping(remote_node_a) do 114 | :pang -> 115 | assert false 116 | :pong -> :ok 117 | end 118 | 119 | Process.flag(:trap_exit, true) 120 | 121 | pid = HandlerAgent.start(node: remote_node_a, forward_pid: self()) 122 | 123 | assert is_pid(pid) 124 | assert node(pid) == remote_node_a 125 | 126 | send pid, {:get_handler_pid, self()} 127 | assert_receive {:handler_pid, handler_pid} 128 | assert node(handler_pid) == remote_node_a 129 | 130 | send handler_pid, {:trace, :foo} 131 | send handler_pid, {:trace, :bar} 132 | 133 | assert_receive {:trace, :foo} 134 | assert_receive {:trace, :bar} 135 | end 136 | end 137 | -------------------------------------------------------------------------------- /test/tracer/matcher_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Matcher.Test do 2 | use ExUnit.Case 3 | 4 | alias Tracer.Matcher 5 | import Tracer.Matcher 6 | 7 | test "no params" do 8 | assert (match do -> :x end) == 9 | [{ [], [], [:x] }] 10 | end 11 | 12 | test "basic" do 13 | assert (match do x -> x end) == 14 | [{ [:"$1"], [], [:"$1"] }] 15 | end 16 | 17 | test "gproc" do 18 | assert (match do {{:n, :l, {:client, id}}, pid, _} -> {id, pid} end) == 19 | [{[{{:n, :l, {:client, :"$1"}}, :"$2", :_}], [], [{:"$1", :"$2"}]}] 20 | end 21 | 22 | test "match supports bound variables" do 23 | id = 5 24 | assert (match do {{:n, :l, {:client, ^id}}, pid, _} -> pid end) == 25 | [{[{{:n, :l, {:client, 5}}, :"$1", :_}], [], [:"$1"]}] 26 | end 27 | 28 | test "gproc with 3 vars" do 29 | assert (match do {{:n, :l, {:client, id}}, pid, third} -> {id, pid, third} end) == 30 | [{[{{:n, :l, {:client, :"$1"}}, :"$2", :"$3"}], [], [{:"$1", :"$2", :"$3"}]}] 31 | end 32 | 33 | test "gproc with 1 var and 2 bound vars" do 34 | one = 11 35 | two = 22 36 | assert (match do {{:n, :l, {:client, ^one}}, pid, ^two} -> {^one, pid} end) == 37 | [{[{{:n, :l, {:client, 11}}, :"$1", 22}], [], [{11, :"$1"}]}] 38 | end 39 | 40 | test "cond" do 41 | assert (match do x when true -> 0 end) == 42 | [{[:"$1"], [true], [0] }] 43 | 44 | assert (match do x when true and false -> 0 end) == 45 | [{[:"$1"], [{ :andalso, true, false }], [0] }] 46 | end 47 | 48 | test "multiple matchs" do 49 | ms = match do 50 | x -> 0 51 | y -> y 52 | end 53 | assert ms == [{[:"$1"], [], [0] }, {[:"$1"], [], [:"$1"] }] 54 | end 55 | 56 | test "multiple exprs in body" do 57 | ms = match do x -> 58 | x 59 | 0 60 | end 61 | assert ms == [{[:"$1"], [], [:"$1", 0] }] 62 | end 63 | 64 | test "body with message including one literal" do 65 | ms = match do -> message(:a) end 66 | assert ms == [{[], [], [{:message, [:a]}] }] 67 | end 68 | 69 | test "body with message including literals" do 70 | ms = match do -> message(:a, :b, :c) end 71 | assert ms == [{[], [], [{:message, [:a, :b, :c]}] }] 72 | end 73 | 74 | test "body with message including bindings" do 75 | ms = match do (a, b, c) -> message(a, b, c) end 76 | assert ms == [{[:"$1", :"$2", :"$3"], 77 | [], 78 | [{:message, [[:a, :"$1"], [:b, :"$2"], [:c, :"$3"]]}] }] 79 | end 80 | 81 | test "body with count including bindings" do 82 | ms = match do (a, b, c) -> count(a, b, c) end 83 | assert ms == [{[:"$1", :"$2", :"$3"], 84 | [], 85 | [{:message, 86 | [[:_cmd, :count], [:a, :"$1"], [:b, :"$2"], [:c, :"$3"]]}] }] 87 | end 88 | 89 | test "body with :ok statement" do 90 | ms = match do (a, b, c) -> :ok end 91 | assert ms == [{[:"$1", :"$2", :"$3"], 92 | [], 93 | [:ok] }] 94 | end 95 | 96 | test "global with erlang module and any function" do 97 | res = global do :lists._ -> :foo end 98 | assert res == %Matcher{flags: [:global], 99 | mfa: {:lists, :_, :_}, 100 | ms: [{:_, [], [:foo]}], 101 | desc: "global do :lists._() -> :foo end" 102 | } 103 | end 104 | 105 | test "global with erlang module and any arity" do 106 | res = global do :lists.sum -> :foo end 107 | assert res == %Matcher{flags: [:global], 108 | mfa: {:lists, :sum, :_}, 109 | ms: [{:_, [], [:foo]}], 110 | desc: "global do :lists.sum() -> :foo end" 111 | } 112 | end 113 | 114 | test "global with erlang module and function" do 115 | res = global do :lists.max(a) -> :foo end 116 | assert res == %Matcher{flags: [:global], 117 | mfa: {:lists, :max, 1}, 118 | ms: [{[:"$1"], [], [:foo]}], 119 | desc: "global do :lists.max(a) -> :foo end" 120 | } 121 | end 122 | 123 | test "global with Module._ mfa" do 124 | res = global do Map._ -> :foo end 125 | assert res == %Matcher{flags: [:global], 126 | mfa: {Map, :_, :_}, 127 | ms: [{:_, [], [:foo]}], 128 | desc: "global do Map._() -> :foo end" 129 | } 130 | end 131 | 132 | test "global with don't care mfa" do 133 | res = global do _ -> :foo end 134 | assert res == %Matcher{flags: [:global], 135 | mfa: {:_, :_, :_}, 136 | ms: [{:_, [], [:foo]}], 137 | desc: "global do _ -> :foo end" 138 | } 139 | end 140 | 141 | test "global with count including bindings" do 142 | res = global do Map.get(a, b) -> count(a, b) end 143 | assert res == %Matcher{flags: [:global], 144 | mfa: {Map, :get, 2}, 145 | ms: [{[:"$1", :"$2"], [], 146 | [message: [[:_cmd, :count], [:a, :"$1"], [:b, :"$2"]]]}], 147 | desc: "global do Map.get(a, b) -> count(a, b) end" 148 | } 149 | end 150 | 151 | test "local with count including bindings" do 152 | res = local do Map.get(a, b) -> count(a, b) end 153 | assert res == %Matcher{flags: [:local], 154 | mfa: {Map, :get, 2}, 155 | ms: [{[:"$1", :"$2"], [], 156 | [message: [[:_cmd, :count], [:a, :"$1"], [:b, :"$2"]]]}], 157 | desc: "local do Map.get(a, b) -> count(a, b) end" 158 | } 159 | end 160 | 161 | test "local without single clause" do 162 | res = local Map.get(a, b) 163 | assert res == %Matcher{flags: [:local], 164 | mfa: {Map, :get, 2}, 165 | ms: [{[:"$1", :"$2"], [], 166 | [message: [[:a, :"$1"], [:b, :"$2"]]]}], 167 | desc: "local Map.get(a, b)" 168 | } 169 | end 170 | 171 | test "local without single clause no params" do 172 | res = local Map.get(_, _) 173 | assert res == %Matcher{flags: [:local], 174 | mfa: {Map, :get, 2}, 175 | ms: [{[:_, :_], [], 176 | [message: []]}], 177 | desc: "local Map.get(_, _)" 178 | } 179 | end 180 | 181 | test "local without single clause no params no fun" do 182 | res = local Map._ 183 | assert res == %Matcher{flags: [:local], 184 | mfa: {Map, :_, :_}, 185 | ms: [{:_, [], 186 | [message: []]}], 187 | desc: "local Map._()" 188 | } 189 | end 190 | 191 | test "local without single clause match all" do 192 | res = local _ 193 | assert res == %Matcher{flags: [:local], 194 | mfa: {:_, :_, :_}, 195 | ms: [{:_, [], 196 | [message: []]}], 197 | desc: "local _" 198 | } 199 | end 200 | 201 | test "local without body with multiple clauses" do 202 | res = local do Map.get(a, b); Map.get(d, e) end 203 | assert res == %Matcher{flags: [:local], 204 | mfa: {Map, :get, 2}, 205 | ms: [{[:"$1", :"$2"], [], 206 | [message: [[:a, :"$1"], [:b, :"$2"]]]}, 207 | {[:"$1", :"$2"], [], 208 | [message: [[:d, :"$1"], [:e, :"$2"]]]}], 209 | desc: "local do \n Map.get(a, b)\n Map.get(d, e)\n end" 210 | } 211 | end 212 | 213 | test "local without body with one do clause" do 214 | res = local do Map.get(a, b) end 215 | assert res == %Matcher{flags: [:local], 216 | mfa: {Map, :get, 2}, 217 | ms: [{[:"$1", :"$2"], [], 218 | [message: [[:a, :"$1"], [:b, :"$2"]]]}], 219 | desc: "local do Map.get(a, b) end" 220 | } 221 | end 222 | end 223 | -------------------------------------------------------------------------------- /test/tracer/pid_handler_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.PidHandler.Test do 2 | use ExUnit.Case 3 | 4 | alias Tracer.PidHandler 5 | 6 | test "start raises an error when no callback is passed" do 7 | assert_raise ArgumentError, "missing event_callback configuration", fn -> 8 | PidHandler.start(max_message_count: 1) 9 | end 10 | end 11 | 12 | test "start spawn a process and returns its pid" do 13 | pid = PidHandler.start(event_callback: fn _event -> :ok end) 14 | assert is_pid(pid) 15 | assert Process.alive?(pid) 16 | end 17 | 18 | test "stop causes the process to end with a normal status" do 19 | Process.flag(:trap_exit, true) 20 | pid = PidHandler.start(event_callback: fn _event -> :ok end) 21 | assert Process.alive?(pid) 22 | PidHandler.stop(pid) 23 | assert_receive({:EXIT, ^pid, :normal}) 24 | refute Process.alive?(pid) 25 | end 26 | 27 | test "max_message_count triggers when too many events are received" do 28 | Process.flag(:trap_exit, true) 29 | pid = PidHandler.start(max_message_count: 2, 30 | event_callback: fn _event -> :ok end) 31 | assert Process.alive?(pid) 32 | send pid, {:trace, :foo} 33 | send pid, {:trace_ts, :bar} 34 | assert_receive({:EXIT, ^pid, {:max_message_count, 2}}) 35 | refute Process.alive?(pid) 36 | end 37 | 38 | test "unrecognized messages are discarded to avoid queue from filling up" do 39 | Process.flag(:trap_exit, true) 40 | pid = PidHandler.start(max_message_count: 1, 41 | event_callback: fn _event -> :ok end) 42 | assert Process.alive?(pid) 43 | for i <- 1..100, do: send pid, {:not_expeted_message, i} 44 | case Process.info(self(), :message_queue_len) do 45 | {:message_queue_len, len} -> assert len == 0 46 | error -> assert error 47 | end 48 | end 49 | 50 | test "callback is invoked when a trace event is received" do 51 | Process.flag(:trap_exit, true) 52 | test_pid = self() 53 | pid = PidHandler.start(event_callback: fn event -> 54 | send test_pid, event 55 | :ok 56 | end) 57 | assert Process.alive?(pid) 58 | for i <- 1..100, do: send pid, {:trace, i} 59 | for i <- 1..100, do: assert_receive {:trace, ^i} 60 | end 61 | 62 | test "process exits if callback does not return :ok" do 63 | Process.flag(:trap_exit, true) 64 | pid = PidHandler.start(event_callback: fn _event -> :not_ok end) 65 | assert Process.alive?(pid) 66 | send pid, {:trace, :foo} 67 | assert_receive({:EXIT, ^pid, {:not_ok, []}}) 68 | refute Process.alive?(pid) 69 | end 70 | 71 | test "process exits if too many messages wait in mailbox" do 72 | Process.flag(:trap_exit, true) 73 | pid = PidHandler.start(max_queue_size: 10, 74 | event_callback: fn _event -> 75 | :timer.sleep(20); 76 | :ok end) 77 | assert Process.alive?(pid) 78 | for i <- 1..100, do: send pid, {:trace, i} 79 | case Process.info(pid, :message_queue_len) do 80 | {:message_queue_len, len} -> assert len >= 10 81 | _ -> :ok 82 | end 83 | assert_receive({:EXIT, ^pid, {:message_queue_size, _}}) 84 | refute Process.alive?(pid) 85 | end 86 | end 87 | -------------------------------------------------------------------------------- /test/tracer/probe_list_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.ProbeList.Test do 2 | use ExUnit.Case 3 | 4 | alias Tracer.{ProbeList, Probe} 5 | 6 | # test "new returns a tracer" do 7 | # assert Tracer.new() == %Tracer{} 8 | # end 9 | # 10 | # test "new accepts a probe shorthand" do 11 | # res = Tracer.new(probe: Probe.new(type: :call)) 12 | # |> Tracer.probes() 13 | # 14 | # assert res == [Probe.new(type: :call)] 15 | # end 16 | 17 | test "add_probe complains if not passed a probe" do 18 | res = ProbeList.add_probe([], %{}) 19 | assert res == {:error, :not_a_probe} 20 | end 21 | 22 | test "add_probe adds probe to probe list" do 23 | res = ProbeList.add_probe([], Probe.new(type: :call)) 24 | assert res == [Probe.new(type: :call)] 25 | end 26 | 27 | test "add_probe fails if probe_list has a probe of the same type" do 28 | res = [] 29 | |> ProbeList.add_probe(Probe.new(type: :call)) 30 | |> ProbeList.add_probe(Probe.new(type: :call)) 31 | 32 | assert res == {:error, :duplicate_probe_type} 33 | end 34 | 35 | test "remove_probe removes probe from probe list" do 36 | res = [] 37 | |> ProbeList.add_probe(Probe.new(type: :call)) 38 | |> ProbeList.remove_probe(Probe.new(type: :call)) 39 | 40 | assert res == [] 41 | end 42 | 43 | test "valid? returns error if not probes have been configured" do 44 | res = ProbeList.valid?([]) 45 | 46 | assert res == {:error, :missing_probes} 47 | end 48 | 49 | test "valid? return error if probes are invalid" do 50 | res = [] 51 | |> ProbeList.add_probe(Probe.new(type: :call)) 52 | |> ProbeList.add_probe(Probe.new(type: :send)) 53 | |> ProbeList.valid?() 54 | 55 | assert res == {:error, {:invalid_probe, [ 56 | {:error, :missing_processes, Probe.new(type: :send)}, 57 | {:error, :missing_processes, Probe.new(type: :call)} 58 | ]}} 59 | end 60 | 61 | end 62 | -------------------------------------------------------------------------------- /test/tracer/probe_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Probe.Test do 2 | use ExUnit.Case 3 | alias Tracer.Probe 4 | alias Tracer.Clause 5 | import Tracer.Matcher 6 | 7 | test "new returns an error if does not include type argument" do 8 | assert Probe.new(param: :foo) == {:error, :missing_type} 9 | end 10 | 11 | test "new returns an error if receives an invalid type" do 12 | assert Probe.new(type: :foo) == {:error, :invalid_type} 13 | end 14 | 15 | test "new supports explicit type parameter" do 16 | %Probe{} = probe = Probe.new(:call) 17 | assert probe.type == :call 18 | assert probe.process_list == [] 19 | end 20 | 21 | test "new complains if invalid probe type is passed" do 22 | res = Probe.new(:foo) 23 | assert res == {:error, :invalid_probe_type} 24 | end 25 | 26 | test "new returns a new probe with the correct type" do 27 | %Probe{} = probe = Probe.new(type: :call) 28 | assert probe.type == :call 29 | assert probe.process_list == [] 30 | assert probe.enabled? == true 31 | end 32 | 33 | test "process_list stores new processes" do 34 | probe = Probe.new(type: :call) 35 | |> Probe.process_list([:c.pid(0, 1, 0)]) 36 | |> Probe.process_list([self(), self()]) 37 | assert probe.process_list == [self()] 38 | end 39 | 40 | test "add_process adds the new processes" do 41 | probe = Probe.new(type: :call) 42 | |> Probe.add_process([self(), self(), :c.pid(0, 1, 0)]) 43 | assert probe.process_list == [self(), :c.pid(0, 1, 0)] 44 | end 45 | 46 | test "add_process adds the new process" do 47 | probe = Probe.new(type: :call) 48 | |> Probe.add_process(self()) 49 | |> Probe.add_process(self()) 50 | assert probe.process_list == [self()] 51 | end 52 | 53 | test "remove_process removes process" do 54 | probe = Probe.new(type: :call) 55 | |> Probe.add_process(self()) 56 | |> Probe.remove_process(self()) 57 | assert probe.process_list == [] 58 | end 59 | 60 | test "add_clauses return an error when not receiving clauses" do 61 | res = Probe.new(type: :call) 62 | |> Probe.add_clauses(42) 63 | 64 | assert res == {:error, [{:not_a_clause, 42}]} 65 | end 66 | 67 | test "add_clauses return an error if the clause type does not match probe" do 68 | res = Probe.new(type: :procs) 69 | |> Probe.add_clauses(Clause.new() |> Clause.put_mfa()) 70 | 71 | assert res == {:error, 72 | [{:invalid_clause_type, Clause.new() |> Clause.put_mfa()}]} 73 | 74 | end 75 | 76 | test "add_clauses stores clauses" do 77 | probe = Probe.new(type: :call) 78 | |> Probe.add_clauses(Clause.new() |> Clause.put_mfa()) 79 | 80 | %Probe{} = probe 81 | assert Probe.clauses(probe) == [Clause.new() |> Clause.put_mfa()] 82 | end 83 | 84 | test "remove_clause removes the clause" do 85 | probe = Probe.new(type: :call) 86 | |> Probe.add_clauses(Clause.new() |> Clause.put_mfa(Map)) 87 | |> Probe.add_clauses(Clause.new() |> Clause.put_mfa()) 88 | |> Probe.remove_clauses(Clause.new() |> Clause.put_mfa()) 89 | 90 | assert Probe.clauses(probe) == [Clause.new() |> Clause.put_mfa(Map)] 91 | end 92 | 93 | test "enable, disable and enabled? work as expected" do 94 | probe = Probe.new(type: :call) 95 | |> Probe.disable() 96 | assert Probe.enabled?(probe) == false 97 | probe = Probe.enable(probe) 98 | assert Probe.enabled?(probe) == true 99 | end 100 | 101 | test "valid? returns an error if process_list is not configured" do 102 | probe = Probe.new(type: :call) 103 | assert Probe.valid?(probe) == {:error, :missing_processes} 104 | end 105 | 106 | test "arity enables or disables arity flag" do 107 | probe = Probe.new(type: :call) 108 | |> Probe.arity(false) 109 | refute Enum.member?(probe.flags, :arity) 110 | probe = Probe.arity(probe, true) 111 | assert Enum.member?(probe.flags, :arity) 112 | end 113 | 114 | test "probe can be created using with fun option" do 115 | probe = Probe.new( 116 | type: :call, 117 | process: self(), 118 | match: {Map, :get, 2}) 119 | 120 | %Probe{} = probe 121 | assert probe.type == :call 122 | assert probe.process_list == [self()] 123 | assert Enum.count(probe.clauses) == 1 124 | clause = hd(probe.clauses) 125 | assert Clause.get_mfa(clause) == {Map, :get, 2} 126 | end 127 | 128 | test "probe can be created using only match_by option" do 129 | probe = Probe.new( 130 | type: :call, 131 | process: self(), 132 | match: global do Map.get(a, b) -> message(a, b) end) 133 | 134 | %Probe{} = probe 135 | assert probe.type == :call 136 | assert probe.process_list == [self()] 137 | assert probe.flags == [:arity, :timestamp] 138 | assert Enum.count(probe.clauses) == 1 139 | clause = hd(probe.clauses) 140 | assert Clause.get_mfa(clause) == {Map, :get, 2} 141 | assert Clause.get_flags(clause) == [:global] 142 | expected_specs = match do (a, b) -> message(a, b) end 143 | assert clause.match_specs == expected_specs 144 | end 145 | 146 | test "probe can be created using two match_by options" do 147 | probe = Probe.new( 148 | type: :call, 149 | process: self(), 150 | match: global do Map.get(a, b) -> message(a, b) end, 151 | match: local do String.split(a, b) -> message(a, b) end) 152 | 153 | %Probe{} = probe 154 | assert probe.type == :call 155 | assert probe.process_list == [self()] 156 | assert probe.flags == [:arity, :timestamp] 157 | assert Enum.count(probe.clauses) == 2 158 | 159 | clause = hd(probe.clauses) 160 | assert Clause.get_mfa(clause) == {String, :split, 2} 161 | assert Clause.get_flags(clause) == [:local] 162 | expected_specs = match do (a, b) -> message(a, b) end 163 | assert clause.match_specs == expected_specs 164 | 165 | clause = hd(tl(probe.clauses)) 166 | assert Clause.get_mfa(clause) == {Map, :get, 2} 167 | assert Clause.get_flags(clause) == [:global] 168 | expected_specs = match do (a, b) -> message(a, b) end 169 | assert clause.match_specs == expected_specs 170 | end 171 | 172 | test "get_trace_cmds returns the expected command list" do 173 | probe = Probe.new( 174 | type: :call, 175 | process: self(), 176 | match: global do Map.get(a, b) -> message(a, b) end) 177 | 178 | [trace_pattern_cmd, trace_cmd] = Probe.get_trace_cmds(probe) 179 | 180 | assert trace_pattern_cmd == [ 181 | fun: &:erlang.trace_pattern/3, 182 | mfa: {Map, :get, 2}, 183 | match_spec: [{[:"$1", :"$2"], [], [message: [[:a, :"$1"], [:b, :"$2"]]]}], 184 | flag_list: [:global]] 185 | 186 | test_pid = self() 187 | assert trace_cmd == [ 188 | fun: &:erlang.trace/3, 189 | pid_port_spec: test_pid, 190 | how: true, 191 | flag_list: [:call, :arity, :timestamp]] 192 | end 193 | 194 | test "get_trace_cmds raises an exception if the probe is invalid" do 195 | probe = Probe.new(type: :call) 196 | 197 | assert_raise RuntimeError, "invalid probe {:error, :missing_processes}", fn -> 198 | Probe.get_trace_cmds(probe) 199 | end 200 | end 201 | end 202 | -------------------------------------------------------------------------------- /test/tracer/process_helper_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.ProcessHelper.Test do 2 | use ExUnit.Case 3 | alias Tracer.ProcessHelper 4 | 5 | test "ensure_pid() returns pid for a pid" do 6 | assert ProcessHelper.ensure_pid(self()) == self() 7 | end 8 | 9 | test "ensure_pid() traps when for a non registered name" do 10 | assert_raise ArgumentError, "Foo is not a registered process", fn -> 11 | ProcessHelper.ensure_pid(Foo) 12 | end 13 | end 14 | 15 | test "ensure_pid() returns the pid from a registered process" do 16 | Process.register(self(), Foo) 17 | assert ProcessHelper.ensure_pid(Foo) == self() 18 | end 19 | 20 | test "type() handles regular processes" do 21 | res = ProcessHelper.type(self()) 22 | assert res == :regular 23 | end 24 | 25 | test "type() handles supervisor processes" do 26 | res = ProcessHelper.type(:kernel_sup) 27 | assert res == :supervisor 28 | end 29 | 30 | test "type() handles worker processes" do 31 | res = ProcessHelper.type(:file_server_2) 32 | assert res == :worker 33 | end 34 | 35 | test "find_children() for supervisor processes" do 36 | res = ProcessHelper.find_children(Logger.Supervisor) 37 | assert length(res) == 4 38 | end 39 | 40 | test "find_all_children() for workers processes" do 41 | assert ProcessHelper.find_all_children(self()) == [] 42 | end 43 | 44 | test "find_all_children() for supervisor processes" do 45 | res = ProcessHelper.find_all_children(Logger.Supervisor) 46 | assert length(res) == 5 47 | end 48 | 49 | @tag :remote_node 50 | test "ensure_pid() works on other node" do 51 | :net_kernel.start([:"local2@127.0.0.1"]) 52 | 53 | remote_node = "remote#{Enum.random(1..100)}@127.0.0.1" 54 | remote_node_a = String.to_atom(remote_node) 55 | spawn(fn -> 56 | System.cmd("elixir", ["--name", remote_node, 57 | "-e", "Process.register(self(), Foo); :timer.sleep(1000)"]) 58 | end) 59 | 60 | :timer.sleep(500) 61 | # check if remote node is up 62 | case :net_adm.ping(remote_node_a) do 63 | :pang -> 64 | assert false 65 | :pong -> :ok 66 | end 67 | 68 | pid = ProcessHelper.ensure_pid(Foo, remote_node_a) 69 | assert is_pid(pid) 70 | assert node(pid) == remote_node_a 71 | end 72 | 73 | @tag :remote_node 74 | test "type() works on other node" do 75 | :net_kernel.start([:"local2@127.0.0.1"]) 76 | 77 | remote_node = "remote#{Enum.random(1..100)}@127.0.0.1" 78 | remote_node_a = String.to_atom(remote_node) 79 | spawn(fn -> 80 | System.cmd("elixir", ["--name", remote_node, 81 | "-e", ":timer.sleep(1000)"]) 82 | end) 83 | 84 | :timer.sleep(500) 85 | # check if remote node is up 86 | case :net_adm.ping(remote_node_a) do 87 | :pang -> 88 | assert false 89 | :pong -> :ok 90 | end 91 | 92 | pid = ProcessHelper.ensure_pid(Logger.Supervisor, remote_node_a) 93 | assert is_pid(pid) 94 | assert node(pid) == remote_node_a 95 | type = ProcessHelper.type(pid, remote_node_a) 96 | assert type == :supervisor 97 | end 98 | 99 | @tag :remote_node 100 | test "find_children() for supervisor processes on remote node" do 101 | :net_kernel.start([:"local2@127.0.0.1"]) 102 | 103 | remote_node = "remote#{Enum.random(1..100)}@127.0.0.1" 104 | remote_node_a = String.to_atom(remote_node) 105 | spawn(fn -> 106 | System.cmd("elixir", ["--name", remote_node, 107 | "-e", ":timer.sleep(1000)"]) 108 | end) 109 | 110 | :timer.sleep(500) 111 | # check if remote node is up 112 | case :net_adm.ping(remote_node_a) do 113 | :pang -> 114 | assert false 115 | :pong -> :ok 116 | end 117 | 118 | res = ProcessHelper.find_children(Logger.Supervisor, remote_node_a) 119 | assert length(res) == 4 120 | Enum.each(res, fn pid -> 121 | assert node(pid) == remote_node_a 122 | end) 123 | end 124 | 125 | @tag :remote_node 126 | test "find_all_children() for supervisor processes on remote node" do 127 | :net_kernel.start([:"local2@127.0.0.1"]) 128 | 129 | remote_node = "remote#{Enum.random(1..100)}@127.0.0.1" 130 | remote_node_a = String.to_atom(remote_node) 131 | spawn(fn -> 132 | System.cmd("elixir", ["--name", remote_node, 133 | "-e", ":timer.sleep(1000)"]) 134 | end) 135 | 136 | :timer.sleep(500) 137 | # check if remote node is up 138 | case :net_adm.ping(remote_node_a) do 139 | :pang -> 140 | assert false 141 | :pong -> :ok 142 | end 143 | 144 | res = ProcessHelper.find_all_children(Logger.Supervisor, remote_node_a) 145 | assert length(res) == 5 146 | Enum.each(res, fn pid -> 147 | assert node(pid) == remote_node_a 148 | end) 149 | end 150 | end 151 | -------------------------------------------------------------------------------- /test/tracer/tool_helper_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.ToolHelper.Test do 2 | use ExUnit.Case 3 | 4 | alias Tracer.ToolHelper 5 | 6 | test "to_mfa() accepts an mfa tuple" do 7 | assert ToolHelper.to_mfa({Map, :new, 0}) == {Map, :new, 0} 8 | end 9 | 10 | test "to_mfa() accepts a module name" do 11 | assert ToolHelper.to_mfa(Map) == {Map, :_, :_} 12 | assert ToolHelper.to_mfa(:_) == {:_, :_, :_} 13 | end 14 | 15 | test "to_mfa() accepts a function" do 16 | assert ToolHelper.to_mfa(&Map.new/0) == {Map, :new, 0} 17 | end 18 | 19 | test "to_mfa() fails when receiving a fn" do 20 | assert ToolHelper.to_mfa(fn -> :foo end) == 21 | {:error, :not_an_external_function} 22 | end 23 | 24 | test "to_mfa() fails when receiving an invalid mfa" do 25 | assert ToolHelper.to_mfa(%{a: :foo}) == {:error, :invalid_mfa} 26 | end 27 | end 28 | -------------------------------------------------------------------------------- /test/tracer/tool_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Tool.Test do 2 | use ExUnit.Case 3 | alias Tracer.{Tool, Probe} 4 | 5 | defmodule TestTool do 6 | use Tool 7 | alias Tracer.Tool.Test.TestTool 8 | 9 | defstruct dummy: [] 10 | 11 | def init(opts) do 12 | init_tool(%TestTool{}, opts) 13 | end 14 | 15 | def handle_event(event, state) do 16 | report_event(state, {:event, event}) 17 | state 18 | end 19 | 20 | def handle_start(state) do 21 | report_event(state, :start) 22 | state 23 | end 24 | 25 | def handle_flush(state) do 26 | report_event(state, :flush) 27 | state 28 | end 29 | 30 | def handle_stop(state) do 31 | report_event(state, :stop) 32 | state 33 | end 34 | 35 | def handle_valid?(state) do 36 | report_event(state, :valid?) 37 | {:error, :foo} 38 | end 39 | 40 | def trigger_report_event(state, message) do 41 | report_event(state, message) 42 | end 43 | 44 | def call_get_process(state) do 45 | get_process(state) 46 | end 47 | end 48 | 49 | test "init() initializes tool" do 50 | tool = TestTool.init([]) 51 | 52 | assert Map.get(tool, :"__tool__") == %Tool{process: self()} 53 | end 54 | 55 | test "init() raises error if opts is not a list" do 56 | assert_raise ArgumentError, 57 | "arguments needs to be a map and a keyword list", 58 | fn -> 59 | TestTool.init(:foo) 60 | end 61 | end 62 | 63 | test "init() properly stores options" do 64 | tool = TestTool.init(process: :c.pid(0, 42, 0), 65 | forward_to: :c.pid(0, 43, 0), 66 | max_message_count: 777, 67 | max_queue_size: 2000, 68 | max_tracing_time: 10_000, 69 | node: [:"local@127,0.0,1", :"remote@127.0.0.1"], 70 | probes: [Probe.new(type: :call, process: self())], 71 | probe: Probe.new(type: :procs, process: self()), 72 | other_keys: "foo" 73 | ) 74 | 75 | assert Map.get(tool, :"__tool__") == %Tool{ 76 | process: :c.pid(0, 42, 0), 77 | forward_to: :c.pid(0, 43, 0), 78 | agent_opts: [ 79 | max_message_count: 777, 80 | max_queue_size: 2000, 81 | max_tracing_time: 10_000, 82 | node: [:"local@127,0.0,1", :"remote@127.0.0.1"], 83 | ], 84 | probes: [Probe.new(type: :call, process: self()), 85 | Probe.new(type: :procs, process: self())] 86 | } 87 | end 88 | 89 | test "add_probe() adds a probe to the tool" do 90 | tool = TestTool.init([]) 91 | |> Tool.add_probe(Probe.new(type: :call, process: self())) 92 | 93 | assert Map.get(tool, :"__tool__") == %Tool{ 94 | process: self(), 95 | probes: [Probe.new(type: :call, process: self())] 96 | } 97 | end 98 | 99 | test "remove_probe() removes probe from the tool" do 100 | tool = TestTool.init([]) 101 | |> Tool.add_probe(Probe.new(type: :call, process: self())) 102 | |> Tool.remove_probe(Probe.new(type: :call, process: self())) 103 | 104 | assert Map.get(tool, :"__tool__") == %Tool{ 105 | process: self(), 106 | probes: [] 107 | } 108 | end 109 | 110 | test "get_probes() retrieves the tool probes" do 111 | tool = TestTool.init([]) 112 | |> Tool.add_probe(Probe.new(type: :call, process: self())) 113 | 114 | res = Tool.get_probes(tool) 115 | assert res == [Probe.new(type: :call, process: self())] 116 | end 117 | 118 | test "report_event() sends event to forward_to process" do 119 | tool = TestTool.init([forward_to: self()]) 120 | 121 | TestTool.trigger_report_event(tool, :foo) 122 | 123 | assert_receive :foo 124 | refute_receive _ 125 | end 126 | 127 | test "get_process() retrieves configured process" do 128 | tool = TestTool.init([process: :c.pid(0, 42, 0)]) 129 | 130 | assert TestTool.call_get_process(tool) == :c.pid(0, 42, 0) 131 | end 132 | 133 | test "handle_xxx() calls are routed to tool" do 134 | tool = TestTool.init([forward_to: self()]) 135 | 136 | TestTool.handle_valid?(tool) 137 | assert_receive :valid? 138 | 139 | TestTool.handle_start(tool) 140 | assert_receive :start 141 | 142 | TestTool.handle_event(:foo, tool) 143 | assert_receive {:event, :foo} 144 | 145 | TestTool.handle_flush(tool) 146 | assert_receive :flush 147 | 148 | TestTool.handle_stop(tool) 149 | assert_receive :stop 150 | 151 | refute_receive _ 152 | end 153 | 154 | test "valid?() fails if no probes are configured" do 155 | tool = TestTool.init([]) 156 | 157 | assert_raise ArgumentError, 158 | "missing probes, maybe a missing match option?", 159 | fn -> Tool.valid?(tool) end 160 | end 161 | 162 | test "valid?() invokes handle_valid?() for tool to validate its own settings" do 163 | tool = TestTool.init(forward_to: self(), 164 | probes: [Probe.new(type: :call, process: self())]) 165 | 166 | assert Tool.valid?(tool) == {:error, :foo} 167 | assert_receive :valid? 168 | refute_receive _ 169 | end 170 | end 171 | -------------------------------------------------------------------------------- /test/tracer/tools/tool_call_seq_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Tool.CallSeq.Test do 2 | use ExUnit.Case 3 | alias __MODULE__ 4 | alias Tracer.Tool.CallSeq 5 | import Tracer 6 | 7 | setup do 8 | # kill server if alive? for a fresh test 9 | case Process.whereis(Tracer.Server) do 10 | nil -> :ok 11 | pid -> 12 | Process.exit(pid, :kill) 13 | :timer.sleep(10) 14 | end 15 | :ok 16 | end 17 | 18 | test "CallSeq fails when called with invalid option" do 19 | assert_raise ArgumentError, "not supported options: foo, bar", fn -> 20 | run(CallSeq, foo: 4, bar: 5) 21 | end 22 | end 23 | 24 | def recur_len([], acc), do: acc 25 | def recur_len([_h | t], acc), do: recur_len(t, acc + 1) 26 | 27 | @tag :timing 28 | test "CallSeq with start_mach module, show args" do 29 | test_pid = self() 30 | 31 | res = run(CallSeq, 32 | process: test_pid, 33 | show_args: true, 34 | show_return: true, 35 | max_depth: 16, 36 | ignore_recursion: false, 37 | forward_to: test_pid, 38 | start_match: Test) 39 | assert res == :ok 40 | 41 | :timer.sleep(10) 42 | 43 | assert_receive :started_tracing 44 | 45 | recur_len([1, 2, 3, 4, 5], 0) 46 | 47 | :timer.sleep(10) 48 | res = stop() 49 | assert res == :ok 50 | 51 | assert_receive %CallSeq.Event{arity: 2, depth: 0, fun: :recur_len, message: [[[1, 2, 3, 4, 5], 0]], mod: Test, pid: _, return_value: nil, type: :enter} 52 | assert_receive %CallSeq.Event{arity: 2, depth: 1, fun: :recur_len, message: [[[2, 3, 4, 5], 1]], mod: Test, pid: _, return_value: nil, type: :enter} 53 | assert_receive %CallSeq.Event{arity: 2, depth: 2, fun: :recur_len, message: [[[3, 4, 5], 2]], mod: Test, pid: _, return_value: nil, type: :enter} 54 | assert_receive %CallSeq.Event{arity: 2, depth: 3, fun: :recur_len, message: [[[4, 5], 3]], mod: Test, pid: _, return_value: nil, type: :enter} 55 | assert_receive %CallSeq.Event{arity: 2, depth: 4, fun: :recur_len, message: [[[5], 4]], mod: Test, pid: _, return_value: nil, type: :enter} 56 | assert_receive %CallSeq.Event{arity: 2, depth: 5, fun: :recur_len, message: [[[], 5]], mod: Test, pid: _, return_value: nil, type: :enter} 57 | assert_receive %CallSeq.Event{arity: 2, depth: 5, fun: :recur_len, message: nil, mod: Test, pid: _, return_value: 5, type: :exit} 58 | assert_receive %CallSeq.Event{arity: 2, depth: 4, fun: :recur_len, message: nil, mod: Test, pid: _, return_value: 5, type: :exit} 59 | assert_receive %CallSeq.Event{arity: 2, depth: 3, fun: :recur_len, message: nil, mod: Test, pid: _, return_value: 5, type: :exit} 60 | assert_receive %CallSeq.Event{arity: 2, depth: 2, fun: :recur_len, message: nil, mod: Test, pid: _, return_value: 5, type: :exit} 61 | assert_receive %CallSeq.Event{arity: 2, depth: 1, fun: :recur_len, message: nil, mod: Test, pid: _, return_value: 5, type: :exit} 62 | assert_receive %CallSeq.Event{arity: 2, depth: 0, fun: :recur_len, message: nil, mod: Test, pid: _, return_value: 5, type: :exit} 63 | 64 | assert_receive {:done_tracing, :stop_command} 65 | # not expeting more events 66 | refute_receive(_) 67 | end 68 | 69 | @tag :timing 70 | test "CallSeq with start_mach fun, ignore_recursion" do 71 | test_pid = self() 72 | 73 | res = run(CallSeq, 74 | process: test_pid, 75 | forward_to: test_pid, 76 | start_match: &Test.recur_len/2) 77 | assert res == :ok 78 | 79 | :timer.sleep(10) 80 | 81 | assert_receive :started_tracing 82 | 83 | recur_len([1, 2, 3, 4, 5], 0) 84 | 85 | :timer.sleep(10) 86 | res = stop() 87 | assert res == :ok 88 | 89 | assert_receive %CallSeq.Event{arity: 2, depth: 0, fun: :recur_len, message: nil, mod: Test, pid: _, return_value: nil, type: :enter} 90 | assert_receive %CallSeq.Event{arity: 2, depth: 0, fun: :recur_len, message: nil, mod: Test, pid: _, return_value: nil, type: :exit} 91 | 92 | assert_receive {:done_tracing, :stop_command} 93 | # not expeting more events 94 | refute_receive(_) 95 | end 96 | 97 | end 98 | -------------------------------------------------------------------------------- /test/tracer/tools/tool_duration_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Duration.Test do 2 | use ExUnit.Case 3 | 4 | import Tracer 5 | import Tracer.Matcher 6 | alias Tracer.Tool.Duration 7 | 8 | setup do 9 | # kill server if alive? for a fresh test 10 | case Process.whereis(Tracer.Server) do 11 | nil -> :ok 12 | pid -> 13 | Process.exit(pid, :kill) 14 | :timer.sleep(10) 15 | end 16 | :ok 17 | end 18 | 19 | test "duration tool without aggregaton" do 20 | test_pid = self() 21 | 22 | res = run(Duration, 23 | process: test_pid, 24 | forward_to: test_pid, 25 | match: local Map.new(val)) 26 | assert res == :ok 27 | 28 | :timer.sleep(50) 29 | 30 | %{tracing: true} = :sys.get_state(Tracer.Server, 100) 31 | 32 | Map.new(%{}) 33 | Map.new(%{a: :foo}) 34 | 35 | assert_receive :started_tracing 36 | assert_receive(%{pid: ^test_pid, mod: Map, fun: :new, 37 | arity: 1, duration: _, message: [[:val, %{}]]}) 38 | assert_receive(%{pid: ^test_pid, mod: Map, fun: :new, 39 | arity: 1, duration: _, message: [[:val, %{a: :foo}]]}) 40 | 41 | res = stop() 42 | assert res == :ok 43 | 44 | assert_receive {:done_tracing, :stop_command} 45 | # not expeting more events 46 | refute_receive(_) 47 | end 48 | 49 | test "duration tool with aggregaton" do 50 | test_pid = self() 51 | 52 | res = run(Duration, 53 | process: test_pid, 54 | aggregation: :dist, 55 | forward_to: test_pid, 56 | match: local Map.new(val)) 57 | assert res == :ok 58 | :timer.sleep(50) 59 | 60 | Map.new(%{}) 61 | Map.new(%{a: :foo}) 62 | 63 | assert_receive :started_tracing 64 | :timer.sleep(50) 65 | 66 | res = stop() 67 | assert res == :ok 68 | 69 | assert_receive %Duration.Event{arity: 1, duration: %{}, fun: :new, message: [[:val, %{}]], mod: Map, pid: nil} 70 | assert_receive %Duration.Event{arity: 1, duration: %{}, fun: :new, message: [[:val, %{a: :foo}]], mod: Map, pid: nil} 71 | assert_receive {:done_tracing, :stop_command} 72 | # not expeting more events 73 | refute_receive(_) 74 | end 75 | end 76 | -------------------------------------------------------------------------------- /test/tracer_server_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Server.Test do 2 | use ExUnit.Case 3 | alias Tracer.{Server, Probe, Tool, 4 | Tool.Count, Tool.Duration, Tool.Display} 5 | import Tracer.Matcher 6 | 7 | setup do 8 | # kill server if alive? for a fresh test 9 | case Process.whereis(Tracer.Server) do 10 | nil -> :ok 11 | pid -> 12 | Process.exit(pid, :kill) 13 | :timer.sleep(10) 14 | end 15 | :ok 16 | end 17 | 18 | test "start() creates server" do 19 | {:ok, pid} = Server.start() 20 | assert is_pid(pid) 21 | registered_pid = Process.whereis(Tracer.Server) 22 | assert registered_pid != nil 23 | assert registered_pid == pid 24 | end 25 | 26 | test "start() fails if server already started" do 27 | {:ok, pid} = Server.start() 28 | {:error, {:already_started, server_pid}} = Server.start() 29 | assert pid == server_pid 30 | end 31 | 32 | test "stop() stops a running server" do 33 | {:ok, pid} = Server.start() 34 | assert Process.alive?(pid) 35 | Server.stop() 36 | refute Process.alive?(pid) 37 | end 38 | 39 | test "start_tool() starts a trace" do 40 | test_pid = self() 41 | {:ok, _} = Server.start() 42 | probe = Probe.new(type: :call, 43 | process: self(), 44 | match: local do Map.new(a) -> message(a) end) 45 | 46 | tool = Tool.new(Display, forward_to: test_pid, probe: probe) 47 | :ok = Server.start_tool(tool) 48 | 49 | state = :sys.get_state(Tracer.Server, 100) 50 | %{tracing: tracing, 51 | tool_server_pid: tool_server_pid, 52 | agent_pids: [agent_pid], 53 | probes: [%{process_list: [^test_pid]}]} = state 54 | assert tracing 55 | assert is_pid(tool_server_pid) 56 | assert Process.alive?(tool_server_pid) 57 | assert is_pid(agent_pid) 58 | assert Process.alive?(agent_pid) 59 | 60 | :timer.sleep(50) # avoid the test from bailing too quickly 61 | res = :erlang.trace_info(test_pid, :flags) 62 | assert res == {:flags, [:arity, :call, :timestamp]} 63 | res = :erlang.trace_info({Map, :new, 1}, :all) 64 | assert res == {:all, 65 | [traced: :local, 66 | match_spec: [{[:"$1"], [], [message: [[:a, :"$1"]]]}], 67 | meta: false, 68 | meta_match_spec: false, 69 | call_time: false, 70 | call_count: false]} 71 | 72 | # test a trace event 73 | Map.new(%{}) 74 | assert_receive %Tracer.EventCall{mod: Map, fun: :new, arity: 1, 75 | message: [[:a, %{}]], pid: ^test_pid, ts: _} 76 | end 77 | 78 | test "stop_tool() stops tracing" do 79 | test_pid = self() 80 | {:ok, _} = Server.start() 81 | probe = Probe.new(type: :call, 82 | process: self(), 83 | match: local do Map.new(a) -> message(a) end) 84 | tool = Tool.new(Display, forward_to: test_pid, probe: probe) 85 | :ok = Server.start_tool(tool) 86 | 87 | # check tracing is enabled 88 | :timer.sleep(50) # avoid the test from bailing too quickly 89 | res = :erlang.trace_info(test_pid, :flags) 90 | assert res == {:flags, [:arity, :call, :timestamp]} 91 | res = :erlang.trace_info({Map, :new, 1}, :all) 92 | assert res == {:all, 93 | [traced: :local, 94 | match_spec: [{[:"$1"], [], [message: [[:a, :"$1"]]]}], 95 | meta: false, 96 | meta_match_spec: false, 97 | call_time: false, 98 | call_count: false]} 99 | 100 | assert_receive :started_tracing 101 | res = Server.stop_tool() 102 | assert res == :ok 103 | 104 | :timer.sleep(50) # avoid the test from bailing too quickly 105 | res = :erlang.trace_info(test_pid, :flags) 106 | assert res == {:flags, []} 107 | res = :erlang.trace_info({Map, :new, 1}, :all) 108 | assert res == {:all, false} 109 | 110 | assert_receive {:done_tracing, :stop_command} 111 | # no trace events should be received 112 | Map.new(%{}) 113 | refute_receive(_) 114 | end 115 | 116 | test "start_tool() allows to override tracing limits" do 117 | test_pid = self() 118 | {:ok, _} = Server.start() 119 | probe = Probe.new(type: :call, 120 | process: self(), 121 | match: local do Map.new(a) -> message(a) end) 122 | 123 | tool = Tool.new(Display, forward_to: test_pid, probe: probe, 124 | max_message_count: 1) 125 | :ok = Server.start_tool(tool) 126 | 127 | :timer.sleep(50) 128 | Map.new(%{}) 129 | assert_receive %Tracer.EventCall{mod: Map, fun: :new, arity: 1, 130 | message: [[:a, %{}]], pid: ^test_pid, ts: _} 131 | 132 | assert_receive {:done_tracing, :max_message_count, 1} 133 | end 134 | 135 | @tag :remote_node 136 | test "start_trace() allows to start on a remote node" do 137 | :net_kernel.start([:"local2@127.0.0.1"]) 138 | 139 | remote_node = "remote#{Enum.random(1..100)}@127.0.0.1" 140 | remote_node_a = String.to_atom(remote_node) 141 | spawn(fn -> 142 | System.cmd("elixir", ["--name", remote_node, 143 | "-e", "for _ <- 1..200 do Map.new(%{}); :timer.sleep(25) end"]) 144 | end) 145 | 146 | :timer.sleep(500) 147 | # check if remote node is up 148 | case :net_adm.ping(remote_node_a) do 149 | :pang -> 150 | assert false 151 | :pong -> :ok 152 | end 153 | 154 | test_pid = self() 155 | {:ok, _} = Server.start() 156 | probe = Probe.new(type: :call, 157 | process: :all, 158 | match: local do Map.new(a) -> message(a) end) 159 | 160 | tool = Tool.new(Display, node: [remote_node_a], forward_to: test_pid, probe: probe) 161 | :ok = Server.start_tool(tool) 162 | 163 | :timer.sleep(500) 164 | assert_receive %Tracer.EventCall{mod: Map, fun: :new, arity: 1, 165 | message: [[:a, %{}]], pid: _, ts: _} 166 | end 167 | 168 | @tag :timing 169 | test "trace with a count tool" do 170 | test_pid = self() 171 | 172 | {:ok, _} = Server.start() 173 | probe = Probe.new( 174 | type: :call, 175 | process: test_pid, 176 | match: local do String.split(a, b) -> message(a, b) end) 177 | 178 | tool = Tool.new(Count, forward_to: test_pid, probe: probe) 179 | :ok = Server.start_tool(tool) 180 | 181 | :timer.sleep(50) 182 | 183 | String.split("hello world", ",") 184 | String.split("x,y", ",") 185 | String.split("z,y", ",") 186 | String.split("x,y", ",") 187 | String.split("z,y", ",") 188 | String.split("x,y", ",") 189 | 190 | :timer.sleep(50) 191 | assert_receive :started_tracing 192 | res = Server.stop_tool() 193 | assert res == :ok 194 | 195 | assert_receive %Count.Event{counts: 196 | [{[a: "hello world", b: ","], 1}, 197 | {[a: "z,y", b: ","], 2}, 198 | {[a: "x,y", b: ","], 3}]} 199 | 200 | 201 | assert_receive {:done_tracing, :stop_command} 202 | # not expeting more events 203 | refute_receive(_) 204 | end 205 | 206 | def recur_len([], acc), do: acc 207 | def recur_len([_h | t], acc), do: recur_len(t, acc + 1) 208 | 209 | @tag :timing 210 | test "trace with a duration tool" do 211 | test_pid = self() 212 | 213 | {:ok, _} = Server.start() 214 | probe = Probe.new( 215 | type: :call, 216 | process: test_pid, 217 | match: local do Tracer.Server.Test.recur_len(list, val) -> return_trace(); message(list, val) end) 218 | 219 | tool = Tool.new(Duration, forward_to: test_pid, probe: probe) 220 | :ok = Server.start_tool(tool) 221 | 222 | assert_receive :started_tracing 223 | 224 | recur_len([1, 2, 3, 4, 5], 0) 225 | recur_len([1, 2, 3, 5], 2) 226 | 227 | assert_receive(%{pid: ^test_pid, mod: Tracer.Server.Test, fun: :recur_len, 228 | arity: 2, duration: _, message: [[:list, [1, 2, 3, 4, 5]], [:val, 0]]}) 229 | assert_receive(%{pid: ^test_pid, mod: Tracer.Server.Test, fun: :recur_len, 230 | arity: 2, duration: _, message: [[:list, [1, 2, 3, 5]], [:val, 2]]}) 231 | 232 | res = Server.stop_tool() 233 | assert res == :ok 234 | 235 | assert_receive {:done_tracing, :stop_command} 236 | # not expeting more events 237 | refute_receive(_) 238 | end 239 | 240 | @tag :timing 241 | test "trace with a display tool" do 242 | test_pid = self() 243 | 244 | {:ok, _} = Server.start() 245 | probe = Probe.new( 246 | type: :call, 247 | process: test_pid, 248 | match: local do String.split(string, pattern) -> return_trace(); message(string, pattern) end) 249 | 250 | tool = Tool.new(Display, forward_to: test_pid, probe: probe) 251 | :ok = Server.start_tool(tool) 252 | 253 | assert_receive :started_tracing 254 | 255 | String.split("a, add", " ") 256 | String.split("a,b", ",") 257 | String.split("c,b", ",") 258 | String.split("a,b", ",") 259 | String.split("c,b", ",") 260 | String.split("a,b", ",") 261 | 262 | 263 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 264 | message: [[:string, "a, add"], [:pattern, " "]], ts: _}) 265 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 266 | return_value: ["a,", "add"], ts: _}) 267 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 268 | message: [[:string, "a,b"], [:pattern, ","]], ts: _}) 269 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 270 | return_value: ["a", "b"], ts: _}) 271 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 272 | message: [[:string, "c,b"], [:pattern, ","]], ts: _}) 273 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 274 | return_value: ["c", "b"], ts: _}) 275 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 276 | message: [[:string, "a,b"], [:pattern, ","]], ts: _}) 277 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 278 | return_value: ["a", "b"], ts: _}) 279 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 280 | message: [[:string, "c,b"], [:pattern, ","]], ts: _}) 281 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 282 | return_value: ["c", "b"], ts: _}) 283 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 284 | message: [[:string, "a,b"], [:pattern, ","]], ts: _}) 285 | assert_receive(%{pid: ^test_pid, mod: String, fun: :split, arity: 2, 286 | return_value: ["a", "b"], ts: _}) 287 | 288 | res = Server.stop_tool() 289 | assert res == :ok 290 | 291 | assert_receive {:done_tracing, :stop_command} 292 | # not expeting more events 293 | refute_receive(_) 294 | end 295 | 296 | @tag :timing 297 | test "child servers are killed after trace finishes" do 298 | test_pid = self() 299 | {:ok, _} = Server.start() 300 | probe = Probe.new(type: :call, 301 | process: self(), 302 | match: local do Map.new(a) -> message(a) end) 303 | 304 | tool = Tool.new(Display, forward_to: test_pid, probe: probe) 305 | :ok = Server.start_tool(tool) 306 | 307 | state = :sys.get_state(Tracer.Server, 100) 308 | %{tracing: _tracing, 309 | tool_server_pid: tool_server_pid, 310 | agent_pids: [agent_pid], 311 | probes: [%{process_list: [^test_pid]}]} = state 312 | 313 | assert Process.alive?(agent_pid) 314 | assert Process.alive?(tool_server_pid) 315 | 316 | :ok = Server.stop_tool() 317 | 318 | :timer.sleep(20) 319 | refute Process.alive?(agent_pid) 320 | refute Process.alive?(tool_server_pid) 321 | end 322 | 323 | @tag :timing 324 | test "child servers are killed after trace restartes" do 325 | test_pid = self() 326 | {:ok, _} = Server.start() 327 | probe = Probe.new(type: :call, 328 | process: self(), 329 | match: local do Map.new(a) -> message(a) end) 330 | 331 | tool = Tool.new(Display, forward_to: test_pid, probe: probe) 332 | :ok = Server.start_tool(tool) # 1 333 | 334 | :timer.sleep(10) 335 | state = :sys.get_state(Tracer.Server, 100) 336 | %{tracing: _tracing, 337 | tool_server_pid: tool_server_pid, 338 | agent_pids: [agent_pid], 339 | probes: [%{process_list: [^test_pid]}]} = state 340 | 341 | :ok = Server.start_tool(tool) # 2 342 | 343 | :timer.sleep(20) 344 | refute Process.alive?(agent_pid) 345 | refute Process.alive?(tool_server_pid) 346 | 347 | state = :sys.get_state(Tracer.Server, 10) 348 | %{tracing: _tracing, 349 | tool_server_pid: tool_server_pid, 350 | agent_pids: [agent_pid], 351 | probes: [%{process_list: [^test_pid]}]} = state 352 | 353 | assert Process.alive?(agent_pid) 354 | assert Process.alive?(tool_server_pid) 355 | 356 | :ok = Server.start_tool(tool) # 3 357 | 358 | :timer.sleep(20) 359 | refute Process.alive?(agent_pid) 360 | refute Process.alive?(tool_server_pid) 361 | end 362 | 363 | end 364 | -------------------------------------------------------------------------------- /test/tracer_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Tracer.Test do 2 | use ExUnit.Case 3 | 4 | import Tracer 5 | import Tracer.Matcher 6 | alias Tracer.{Tool.Count, Tool.Duration, Tool.CallSeq, Tool.Display} 7 | 8 | setup do 9 | # kill server if alive? for a fresh test 10 | case Process.whereis(Tracer.Server) do 11 | nil -> :ok 12 | pid -> 13 | Process.exit(pid, :kill) 14 | :timer.sleep(10) 15 | end 16 | :ok 17 | end 18 | 19 | @tag :timing 20 | test "can add multiple probes" do 21 | {:ok, pid} = Tracer.start_server() 22 | assert Process.alive?(pid) 23 | test_pid = self() 24 | 25 | tool = Tracer.tool(Display, forward_to: test_pid) 26 | |> Tracer.Tool.add_probe(Tracer.probe(type: :call, process: :all, 27 | match: local do Map.new() -> :ok end)) 28 | |> Tracer.Tool.add_probe(Tracer.probe(type: :gc, process: self())) 29 | |> Tracer.Tool.add_probe(Tracer.probe(type: :set_on_link, process: [self()])) 30 | |> Tracer.Tool.add_probe(Tracer.probe(type: :procs, process: [self()])) 31 | |> Tracer.Tool.add_probe(Tracer.probe(type: :receive, process: [self()])) 32 | |> Tracer.Tool.add_probe(Tracer.probe(type: :send, process: [self()])) 33 | |> Tracer.Tool.add_probe(Tracer.probe(type: :sched, process: [self()])) 34 | 35 | probes = Tracer.Tool.get_probes(tool) 36 | assert probes == 37 | [ 38 | %Tracer.Probe{enabled?: true, flags: [:arity, :timestamp], 39 | process_list: [:all], type: :call, 40 | clauses: [%Tracer.Clause{matches: 0, type: :call, 41 | desc: "local do Map.new() -> :ok end", 42 | flags: [:local], 43 | match_specs: [{[], [], [:ok]}], 44 | mfa: {Map, :new, 0}}]}, 45 | %Tracer.Probe{clauses: [], enabled?: true, 46 | flags: [:timestamp], process_list: [test_pid], 47 | type: :gc}, 48 | %Tracer.Probe{clauses: [], enabled?: true, 49 | flags: [:timestamp], 50 | process_list: [test_pid], type: :set_on_link}, 51 | %Tracer.Probe{clauses: [], enabled?: true, 52 | flags: [:timestamp], 53 | process_list: [test_pid], type: :procs}, 54 | %Tracer.Probe{clauses: [], enabled?: true, 55 | flags: [:timestamp], 56 | process_list: [test_pid], type: :receive}, 57 | %Tracer.Probe{clauses: [], enabled?: true, 58 | flags: [:timestamp], 59 | process_list: [test_pid], type: :send}, 60 | %Tracer.Probe{clauses: [], enabled?: true, 61 | flags: [:timestamp], 62 | process_list: [test_pid], type: :sched} 63 | ] 64 | 65 | Tracer.run(tool) 66 | 67 | %{tracing: true} = :sys.get_state(Tracer.Server, 100) 68 | 69 | assert_receive :started_tracing 70 | 71 | res = :erlang.trace_info(test_pid, :flags) 72 | assert res == {:flags, [:arity, :garbage_collection, :running, 73 | :set_on_link, :procs, 74 | :call, :receive, :send, :timestamp]} 75 | res = :erlang.trace_info({Map, :new, 0}, :all) 76 | assert res == {:all, 77 | [traced: :local, 78 | match_spec: [{[], [], [:ok]}], 79 | meta: false, 80 | meta_match_spec: false, 81 | call_time: false, 82 | call_count: false]} 83 | 84 | end 85 | 86 | test "display tool" do 87 | test_pid = self() 88 | 89 | res = run(Display, 90 | forward_to: test_pid, 91 | process: test_pid, 92 | match: local do Map.new() -> :ok end) 93 | assert res == :ok 94 | 95 | :timer.sleep(50) 96 | %{tracing: true} = :sys.get_state(Tracer.Server, 100) 97 | 98 | Map.new() 99 | 100 | assert_receive :started_tracing 101 | res = stop() 102 | assert res == :ok 103 | 104 | assert_receive %Tracer.EventCall{arity: 0, fun: :new, message: nil, 105 | mod: Map, pid: ^test_pid, ts: _} 106 | 107 | assert_receive {:done_tracing, :stop_command} 108 | # not expeting more events 109 | refute_receive(_) 110 | end 111 | 112 | test "count tool" do 113 | test_pid = self() 114 | 115 | res = run(Count, 116 | process: test_pid, 117 | forward_to: test_pid, 118 | match: global do Map.new(%{a: a}); Map.new(%{b: b}) end) 119 | assert res == :ok 120 | 121 | :timer.sleep(50) 122 | 123 | %{tracing: true} = :sys.get_state(Tracer.Server, 100) 124 | 125 | Map.new(%{}) 126 | Map.new(%{}) 127 | Map.new(%{}) 128 | Map.new(%{}) 129 | Map.new(%{a: :foo}) 130 | Map.new(%{a: :foo}) 131 | Map.new(%{b: :bar}) 132 | Map.new(%{b: :bar}) 133 | Map.new(%{b: :bar}) 134 | Map.new(%{b: :bar}) 135 | Map.new(%{b: :bar}) 136 | Map.new(%{b: :bar}) 137 | 138 | assert_receive :started_tracing 139 | res = stop() 140 | assert res == :ok 141 | 142 | assert_receive %Count.Event{counts: 143 | [{[a: :foo], 2}, 144 | {[b: :bar], 6}]} 145 | 146 | assert_receive {:done_tracing, :stop_command} 147 | # not expeting more events 148 | refute_receive(_) 149 | end 150 | 151 | def recur_len([], acc), do: acc 152 | def recur_len([_h | t], acc), do: recur_len(t, acc + 1) 153 | 154 | test "duration tool" do 155 | test_pid = self() 156 | 157 | res = run(Duration, 158 | process: test_pid, 159 | forward_to: test_pid, 160 | match: local Tracer.Test.recur_len(list, val)) 161 | assert res == :ok 162 | 163 | :timer.sleep(50) 164 | 165 | %{tracing: true} = :sys.get_state(Tracer.Server, 100) 166 | 167 | recur_len([1, 2, 3, 4, 5], 0) 168 | recur_len([1, 2, 3, 5], 2) 169 | 170 | assert_receive :started_tracing 171 | assert_receive(%{pid: ^test_pid, mod: Tracer.Test, fun: :recur_len, 172 | arity: 2, duration: _, message: [[:list, [1, 2, 3, 4, 5]], [:val, 0]]}) 173 | assert_receive(%{pid: ^test_pid, mod: Tracer.Test, fun: :recur_len, 174 | arity: 2, duration: _, message: [[:list, [1, 2, 3, 5]], [:val, 2]]}) 175 | 176 | res = stop() 177 | assert res == :ok 178 | 179 | assert_receive {:done_tracing, :stop_command} 180 | # not expeting more events 181 | refute_receive(_) 182 | end 183 | 184 | @tag :timing 185 | test "call_seq tool" do 186 | test_pid = self() 187 | 188 | res = run(CallSeq, 189 | process: test_pid, 190 | forward_to: test_pid, 191 | show_args: true, 192 | show_return: true, 193 | ignore_recursion: false, 194 | start_match: Tracer.Test) 195 | assert res == :ok 196 | 197 | :timer.sleep(10) 198 | 199 | assert_receive :started_tracing 200 | 201 | recur_len([1, 2, 3, 4, 5], 0) 202 | 203 | :timer.sleep(10) 204 | res = stop() 205 | assert res == :ok 206 | 207 | assert_receive %CallSeq.Event{arity: 2, depth: 0, fun: :recur_len, message: [[[1, 2, 3, 4, 5], 0]], mod: Tracer.Test, pid: _, return_value: nil, type: :enter} 208 | assert_receive %CallSeq.Event{arity: 2, depth: 1, fun: :recur_len, message: [[[2, 3, 4, 5], 1]], mod: Tracer.Test, pid: _, return_value: nil, type: :enter} 209 | assert_receive %CallSeq.Event{arity: 2, depth: 2, fun: :recur_len, message: [[[3, 4, 5], 2]], mod: Tracer.Test, pid: _, return_value: nil, type: :enter} 210 | assert_receive %CallSeq.Event{arity: 2, depth: 3, fun: :recur_len, message: [[[4, 5], 3]], mod: Tracer.Test, pid: _, return_value: nil, type: :enter} 211 | assert_receive %CallSeq.Event{arity: 2, depth: 4, fun: :recur_len, message: [[[5], 4]], mod: Tracer.Test, pid: _, return_value: nil, type: :enter} 212 | assert_receive %CallSeq.Event{arity: 2, depth: 5, fun: :recur_len, message: [[[], 5]], mod: Tracer.Test, pid: _, return_value: nil, type: :enter} 213 | assert_receive %CallSeq.Event{arity: 2, depth: 5, fun: :recur_len, message: nil, mod: Tracer.Test, pid: _, return_value: 5, type: :exit} 214 | assert_receive %CallSeq.Event{arity: 2, depth: 4, fun: :recur_len, message: nil, mod: Tracer.Test, pid: _, return_value: 5, type: :exit} 215 | assert_receive %CallSeq.Event{arity: 2, depth: 3, fun: :recur_len, message: nil, mod: Tracer.Test, pid: _, return_value: 5, type: :exit} 216 | assert_receive %CallSeq.Event{arity: 2, depth: 2, fun: :recur_len, message: nil, mod: Tracer.Test, pid: _, return_value: 5, type: :exit} 217 | assert_receive %CallSeq.Event{arity: 2, depth: 1, fun: :recur_len, message: nil, mod: Tracer.Test, pid: _, return_value: 5, type: :exit} 218 | assert_receive %CallSeq.Event{arity: 2, depth: 0, fun: :recur_len, message: nil, mod: Tracer.Test, pid: _, return_value: 5, type: :exit} 219 | 220 | assert_receive {:done_tracing, :stop_command} 221 | # not expeting more events 222 | refute_receive(_) 223 | end 224 | 225 | end 226 | --------------------------------------------------------------------------------