Reduce invalid/noisy mutations in Phoenix component and DSL-heavy modules

[anagrius]

Summary

I tried Muex 0.6.1 against a Phoenix 1.8 / LiveView application. It is promising on plain business-logic modules, but Phoenix/component/DSL-heavy modules produce a high volume of invalid mutations that are difficult to act on.

The main pattern: the literal mutator mutates compile-time metadata / DSL atoms / aliases and reports many of them at line: 0. This makes the report noisy and can hide the useful mutations.

Version

muex 0.6.1
Elixir 1.20.1-otp-27
Phoenix 1.8.x / LiveView 1.2.x

Reproduction Shape

Run Muex on a Phoenix component module with use Phoenix.Component, imports, attr, slot, docs, and HEEx functions:

mix muex \
  --files lib/my_app_web/components/ui/card.ex \
  --mutators literal,return_value \
  --fail-at 0 \
  --max-mutations 20 \
  --timeout 60000 \
  --optimize \
  --optimize-level conservative \
  --format json \
  --verbose

A representative component has this shape:

defmodule MyAppWeb.Components.UI.Card do
  use Phoenix.Component

  import MyAppWeb.ClassMerger, only: [cn: 1]

  attr :class, :string, default: nil
  slot :inner_block, required: true

  def card(assigns) do
    ~H"""
    <div class={cn(["rounded-lg border bg-card", @class])}>
      <%= render_slot(@inner_block) %>
    </div>
    """
  end
end

Observed Output

In one 20-mutant run against a Phoenix component:

{
  "summary": {
    "timeout": 0,
    "invalid": 16,
    "killed": 4,
    "total": 20,
    "survived": 0,
    "mutation_score_high": 100.0,
    "mutation_score_low": 100.0
  }
}

Most invalid mutations were compile-time or DSL metadata literals, for example:

{
  "status": "invalid",
  "description": "Literal: :Phoenix to :mutated_atom",
  "location": {
    "line": 0,
    "file": "lib/my_app_web/components/ui/card.ex"
  },
  "mutator": "Muex.Mutator.Literal"
}

{
  "status": "invalid",
  "description": "Literal: :Component to :mutated_atom",
  "location": {
    "line": 0,
    "file": "lib/my_app_web/components/ui/card.ex"
  },
  "mutator": "Muex.Mutator.Literal"
}

{
  "status": "invalid",
  "description": "Literal: :class to :mutated_atom",
  "location": {
    "line": 0,
    "file": "lib/my_app_web/components/ui/card.ex"
  },
  "mutator": "Muex.Mutator.Literal"
}

We saw similar noise in Phoenix router and Ash resource modules. A router sample produced 10/10 invalid mutations, mostly FunctionCall: swap arguments in pipeline() / plug() / aliases.

By contrast, focused runs on plain logic modules worked well. For example:

mix muex \
  --files lib/my_app/query/value.ex \
  --test-paths test/my_app/query/value_test.exs \
  --mutators return_value,conditional,comparison,boolean \
  --fail-at 0 \
  --timeout 60000 \
  --format json

That produced 9/9 killed, 0 invalid, 0 timeout.

Why This Hurts

For Phoenix projects, a lot of source files contain:

• use Phoenix.Component
• attr / slot macros
• HEEx functions
• routers with pipeline, scope, plug, live, etc.
• Ash / Ecto DSL declarations
• large @doc / @moduledoc strings

Mutating aliases, DSL option keys, docs, and generated compile-time metadata tends to produce invalid mutations rather than useful survived/killed results. The line: 0 location also makes it hard to know whether a mutation is actionable.

Suggestions

Potential improvements:

1. Treat module aliases and compile-time metadata literals as low-value / invalid-prone and skip them by default.
2. Avoid mutating @doc, @moduledoc, attr, slot, use, import, alias, Phoenix router DSL calls, and Ash/Ecto schema/resource DSL unless explicitly requested.
3. If a mutation has no useful source location (line: 0), either skip it by default or label it as compile-time metadata.
4. Add a Phoenix-oriented preset, for example --preset phoenix, that disables the noisy mutators/locations and focuses on function bodies / conditionals / return values.
5. Longer term, a report that includes the actual patch/diff for each mutation would make survived or suspicious mutations much easier to reproduce.

Muex is already useful on focused business-logic files. This issue is specifically about reducing invalid/noisy mutations in common Phoenix/LiveView/Ash-style source files so the reports stay actionable.

[am-kantox]

Hi Thomas!

Thanks for this! I could not agree more with all the above and despite it all can kinda be handled by specifying the explicit list of files, everything of the above would be a great addition to the library.

Unfortunately, I currently have next to zero bandwidth, and I might have this implemented by the beginning of July at best.

On the other hand, I could gladly accept a PR (as the problem is already stated very clean and descriptive, I believe even an AI-generated code would do.)

[am-kantox]

On the other hand, one might use ExPanda to un-macro everything before running muex, but I am not sure if it’s worth the hassle.

[am-kantox]

Hi @anagrius, please try v0.7.0 and tell if I covered all your requests.