Add DateTime add/4 and shift_zone/3 UTC fast path

[reisub]

Assisted-by: Claude Fable:5

I ran into an issue where I was trying to call DateTime.add() inside a mix task without the app started, which means without :tzdata started, and was surprised to see it fail on a UTC DateTime.

The proposed change adds a fast path for UTC DateTimes which avoids consulting the TZ database. It fixes this (admittedly niche!) issue.

Etc/UTC datetime, Apple M1 Max, Benchee averages (warmup: 1, time: 2, memory_time: 1)

Benchmark	Time (before → after)	Memory (before → after)
add/4 with Tzdata.TimeZoneDatabase	1.24 μs → 0.20 μs (6.3x faster)	2 KB → 0.49 KB (4.1x less)
add/4 with Calendar.UTCOnlyTimeZoneDatabase	0.22 μs → 0.20 μs (parity¹)	504 B → 504 B (unchanged)
shift_zone/3 to "Etc/UTC" with Tzdata.TimeZoneDatabase	1.24 μs → 0.21 μs (6.0x faster)	2 KB → 0.49 KB (4.1x less)
shift_zone/3 to "Etc/UTC" with Calendar.UTCOnlyTimeZoneDatabase	0.22 μs → 0.21 μs (parity¹)	504 B → 504 B (unchanged)
shift/3²	0.25 μs → 0.36 μs (1.4x slower)	0.83 KB → 1.15 KB (+0.32 KB)

¹ Identical 167 ns medians; the average-level difference is outlier noise.
² Before = the removed dedicated Etc/UTC clause, which never consulted the database — hence no per-database split and the small regression from the added iso_days round trip.

Benchmark code

Mix.install([:benchee, :tzdata], config: [tzdata: [autoupdate: :disabled]])

defmodule EmptyTimeZoneDatabase do
  @behaviour Calendar.TimeZoneDatabase
  @impl true
  def time_zone_period_from_utc_iso_days(_, _), do: {:error, :time_zone_not_found}
  @impl true
  def time_zone_periods_from_wall_datetime(_, _), do: {:error, :time_zone_not_found}
end

defmodule BenchDateTimeUTC do
  # ---------------------------------------------------------------------------
  # "before": DateTime.add/4 general clause on main — resolves the zone by
  # consulting the time zone database, also for "Etc/UTC".
  # ---------------------------------------------------------------------------
  def add_before(%{calendar: calendar} = datetime, amount_to_add, unit, time_zone_database)
      when is_integer(amount_to_add) do
    %{
      microsecond: {_, precision},
      time_zone: time_zone,
      utc_offset: utc_offset,
      std_offset: std_offset
    } = datetime

    if not is_integer(unit) and unit not in ~w(second millisecond microsecond nanosecond)a do
      raise ArgumentError,
            "unsupported time unit. Expected :day, :hour, :minute, :second, :millisecond, :microsecond, :nanosecond, or a positive integer, got #{inspect(unit)}"
    end

    precision = max(Calendar.ISO.time_unit_to_precision(unit), precision)

    result =
      datetime
      |> to_iso_days()
      |> Calendar.ISO.shift_time_unit(amount_to_add, unit)
      |> apply_tz_offset(utc_offset + std_offset)
      |> shift_zone_before(calendar, precision, time_zone, time_zone_database)

    case result do
      {:ok, result_datetime} -> result_datetime
      {:error, error} -> raise ArgumentError, "cannot add, reason: #{inspect(error)}"
    end
  end

  # ---------------------------------------------------------------------------
  # "after": same general clause, but shift_zone_for_iso_days_utc/5 gained an
  # "Etc/UTC" head that skips the database.
  # ---------------------------------------------------------------------------
  def add_after(%{calendar: calendar} = datetime, amount_to_add, unit, time_zone_database)
      when is_integer(amount_to_add) do
    %{
      microsecond: {_, precision},
      time_zone: time_zone,
      utc_offset: utc_offset,
      std_offset: std_offset
    } = datetime

    if not is_integer(unit) and unit not in ~w(second millisecond microsecond nanosecond)a do
      raise ArgumentError,
            "unsupported time unit. Expected :day, :hour, :minute, :second, :millisecond, :microsecond, :nanosecond, or a positive integer, got #{inspect(unit)}"
    end

    precision = max(Calendar.ISO.time_unit_to_precision(unit), precision)

    result =
      datetime
      |> to_iso_days()
      |> Calendar.ISO.shift_time_unit(amount_to_add, unit)
      |> apply_tz_offset(utc_offset + std_offset)
      |> shift_zone_after(calendar, precision, time_zone, time_zone_database)

    case result do
      {:ok, result_datetime} -> result_datetime
      {:error, error} -> raise ArgumentError, "cannot add, reason: #{inspect(error)}"
    end
  end

  # ---------------------------------------------------------------------------
  # "before": DateTime.shift/3 dedicated "Etc/UTC" whole-function clause on
  # main (shifts wall time directly, no iso_days round trip, no database).
  # ---------------------------------------------------------------------------
  def shift_before(%{calendar: calendar, time_zone: "Etc/UTC"} = datetime, duration) do
    %{
      year: year,
      month: month,
      day: day,
      hour: hour,
      minute: minute,
      second: second,
      microsecond: microsecond
    } = datetime

    {year, month, day, hour, minute, second, microsecond} =
      calendar.shift_naive_datetime(
        year,
        month,
        day,
        hour,
        minute,
        second,
        microsecond,
        Duration.new!(duration)
      )

    %DateTime{
      year: year,
      month: month,
      day: day,
      hour: hour,
      minute: minute,
      second: second,
      microsecond: microsecond,
      time_zone: "Etc/UTC",
      zone_abbr: "UTC",
      std_offset: 0,
      utc_offset: 0
    }
  end

  # ---------------------------------------------------------------------------
  # "after": DateTime.shift/3 general clause with the "Etc/UTC" head on
  # shift_zone_for_iso_days_utc/5 (the dedicated clause is removed).
  # ---------------------------------------------------------------------------
  def shift_after(%{calendar: calendar} = datetime, duration, time_zone_database) do
    %{
      year: year,
      month: month,
      day: day,
      hour: hour,
      minute: minute,
      second: second,
      microsecond: microsecond,
      std_offset: std_offset,
      utc_offset: utc_offset,
      time_zone: time_zone
    } = datetime

    {year, month, day, hour, minute, second, {_, precision} = microsecond} =
      calendar.shift_naive_datetime(
        year,
        month,
        day,
        hour,
        minute,
        second,
        microsecond,
        Duration.new!(duration)
      )

    result =
      calendar.naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond)
      |> apply_tz_offset(utc_offset + std_offset)
      |> shift_zone_after(calendar, precision, time_zone, time_zone_database)

    case result do
      {:ok, result_datetime} -> result_datetime
      {:error, error} -> raise ArgumentError, "cannot shift, reason: #{inspect(error)}"
    end
  end

  # ---------------------------------------------------------------------------
  # "before": DateTime.shift_zone/3 general clause on main — resolves the
  # target zone by consulting the time zone database, also for "Etc/UTC".
  # (UTC -> UTC is not benchmarked: it hits the same-zone no-op clause and
  # never reaches the database, before or after.)
  # ---------------------------------------------------------------------------
  def shift_zone_before(datetime, time_zone, time_zone_database) do
    %{
      std_offset: std_offset,
      utc_offset: utc_offset,
      calendar: calendar,
      microsecond: {_, precision}
    } = datetime

    datetime
    |> to_iso_days()
    |> apply_tz_offset(utc_offset + std_offset)
    |> shift_zone_before(calendar, precision, time_zone, time_zone_database)
  end

  # ---------------------------------------------------------------------------
  # "after": same general clause, but the helper gained the "Etc/UTC" head.
  # ---------------------------------------------------------------------------
  def shift_zone_after(datetime, time_zone, time_zone_database) do
    %{
      std_offset: std_offset,
      utc_offset: utc_offset,
      calendar: calendar,
      microsecond: {_, precision}
    } = datetime

    datetime
    |> to_iso_days()
    |> apply_tz_offset(utc_offset + std_offset)
    |> shift_zone_after(calendar, precision, time_zone, time_zone_database)
  end

  # ---------------------------------------------------------------------------
  # Zone resolution helpers replicated from DateTime
  # ---------------------------------------------------------------------------

  defp shift_zone_before(iso_days_utc, calendar, precision, time_zone, time_zone_db) do
    case time_zone_db.time_zone_period_from_utc_iso_days(iso_days_utc, time_zone) do
      {:ok, %{std_offset: std_offset, utc_offset: utc_offset, zone_abbr: zone_abbr}} ->
        {year, month, day, hour, minute, second, {microsecond_without_precision, _}} =
          iso_days_utc
          |> apply_tz_offset(-(utc_offset + std_offset))
          |> calendar.naive_datetime_from_iso_days()

        datetime = %DateTime{
          calendar: calendar,
          year: year,
          month: month,
          day: day,
          hour: hour,
          minute: minute,
          second: second,
          microsecond: {microsecond_without_precision, precision},
          std_offset: std_offset,
          utc_offset: utc_offset,
          zone_abbr: zone_abbr,
          time_zone: time_zone
        }

        {:ok, datetime}

      {:error, _} = error ->
        error
    end
  end

  defp shift_zone_after(iso_days_utc, calendar, precision, "Etc/UTC", _time_zone_db) do
    {year, month, day, hour, minute, second, {microsecond, _}} =
      calendar.naive_datetime_from_iso_days(iso_days_utc)

    datetime = %DateTime{
      calendar: calendar,
      year: year,
      month: month,
      day: day,
      hour: hour,
      minute: minute,
      second: second,
      microsecond: {microsecond, precision},
      std_offset: 0,
      utc_offset: 0,
      zone_abbr: "UTC",
      time_zone: "Etc/UTC"
    }

    {:ok, datetime}
  end

  defp shift_zone_after(iso_days_utc, calendar, precision, time_zone, time_zone_db) do
    shift_zone_before(iso_days_utc, calendar, precision, time_zone, time_zone_db)
  end

  defp to_iso_days(%{
         calendar: calendar,
         year: year,
         month: month,
         day: day,
         hour: hour,
         minute: minute,
         second: second,
         microsecond: microsecond
       }) do
    calendar.naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond)
  end

  defp apply_tz_offset(iso_days, 0), do: iso_days

  defp apply_tz_offset(iso_days, offset) do
    Calendar.ISO.add_day_fraction_to_iso_days(iso_days, -offset, 86400)
  end
end

dt = ~U[2018-11-15 10:00:00Z]

# Sanity: all copies must agree with DateTime.add/4 under the UTC-only database
for unit <- [:second, :millisecond, :microsecond, :nanosecond, 7],
    amount <- [0, 1, -1, 3600, -3600, 86_400_000] do
  reference = DateTime.add(dt, amount, unit, Calendar.UTCOnlyTimeZoneDatabase)
  ^reference = BenchDateTimeUTC.add_before(dt, amount, unit, Calendar.UTCOnlyTimeZoneDatabase)
  ^reference = BenchDateTimeUTC.add_before(dt, amount, unit, Tzdata.TimeZoneDatabase)
  ^reference = BenchDateTimeUTC.add_after(dt, amount, unit, EmptyTimeZoneDatabase)
end

for duration <- [[hour: 1], [month: 1], [year: -1, day: 2], [microsecond: {4000, 4}]] do
  reference = DateTime.shift(dt, duration, Calendar.UTCOnlyTimeZoneDatabase)
  ^reference = BenchDateTimeUTC.shift_before(dt, duration)
  ^reference = BenchDateTimeUTC.shift_after(dt, duration, EmptyTimeZoneDatabase)
end

# Sanity: shift_zone copies must agree with DateTime.shift_zone/3 for a
# non-UTC datetime shifted to "Etc/UTC"
cph = DateTime.shift_zone!(dt, "Europe/Copenhagen", Tzdata.TimeZoneDatabase)
reference = DateTime.shift_zone(cph, "Etc/UTC", Tzdata.TimeZoneDatabase)
^reference = BenchDateTimeUTC.shift_zone_before(cph, "Etc/UTC", Tzdata.TimeZoneDatabase)
^reference = BenchDateTimeUTC.shift_zone_after(cph, "Etc/UTC", EmptyTimeZoneDatabase)

Benchee.run(
  %{
    "add before" => fn db -> BenchDateTimeUTC.add_before(dt, 3600, :second, db) end,
    "add after" => fn db -> BenchDateTimeUTC.add_after(dt, 3600, :second, db) end
  },
  inputs: %{
    "Tzdata.TimeZoneDatabase" => Tzdata.TimeZoneDatabase,
    "Calendar.UTCOnlyTimeZoneDatabase" => Calendar.UTCOnlyTimeZoneDatabase
  },
  memory_time: 1,
  time: 2,
  warmup: 1
)

Benchee.run(
  %{
    "shift_zone before" => fn db -> BenchDateTimeUTC.shift_zone_before(cph, "Etc/UTC", db) end,
    "shift_zone after" => fn db -> BenchDateTimeUTC.shift_zone_after(cph, "Etc/UTC", db) end
  },
  inputs: %{
    "Tzdata.TimeZoneDatabase" => Tzdata.TimeZoneDatabase,
    "Calendar.UTCOnlyTimeZoneDatabase" => Calendar.UTCOnlyTimeZoneDatabase
  },
  memory_time: 1,
  time: 2,
  warmup: 1
)

Benchee.run(
  %{
    "shift before (dedicated Etc/UTC clause)" => fn ->
      BenchDateTimeUTC.shift_before(dt, hour: 1)
    end,
    "shift after (general clause + helper fast path)" => fn ->
      BenchDateTimeUTC.shift_after(dt, [hour: 1], Calendar.UTCOnlyTimeZoneDatabase)
    end
  },
  memory_time: 1,
  time: 2,
  warmup: 1
)

Raw results

Operating System: macOS
CPU Information: Apple M1 Max
Number of Available Cores: 10
Available memory: 32 GB
Elixir 1.20.3
Erlang 29.0.5
JIT enabled: true

##### With input Calendar.UTCOnlyTimeZoneDatabase #####
Name                 ips        average  deviation         median         99th %
add after         5.08 M      196.84 ns  ±3827.11%         167 ns         292 ns
add before        4.56 M      219.47 ns  ±3883.94%         167 ns         333 ns

Comparison: 
add after         5.08 M
add before        4.56 M - 1.11x slower +22.63 ns

Memory usage statistics:

Name          Memory usage
add after            504 B
add before           504 B - 1.00x memory usage +0 B

**All measurements for memory usage were the same**

##### With input Tzdata.TimeZoneDatabase #####
Name                 ips        average  deviation         median         99th %
add after         5.09 M       0.196 μs  ±3897.87%       0.167 μs        0.29 μs
add before        0.81 M        1.24 μs   ±684.87%        1.17 μs        1.42 μs

Comparison: 
add after         5.09 M
add before        0.81 M - 6.32x slower +1.04 μs

Memory usage statistics:

Name          Memory usage
add after          0.49 KB
add before            2 KB - 4.06x memory usage +1.51 KB

**All measurements for memory usage were the same**

##### With input Calendar.UTCOnlyTimeZoneDatabase #####
Name                        ips        average  deviation         median         99th %
shift_zone after         4.82 M      207.38 ns  ±3522.91%         167 ns         333 ns
shift_zone before        4.51 M      221.59 ns  ±3010.93%         208 ns         333 ns

Comparison: 
shift_zone after         4.82 M
shift_zone before        4.51 M - 1.07x slower +14.21 ns

Memory usage statistics:

Name                 Memory usage
shift_zone after            504 B
shift_zone before           504 B - 1.00x memory usage +0 B

**All measurements for memory usage were the same**

##### With input Tzdata.TimeZoneDatabase #####
Name                        ips        average  deviation         median         99th %
shift_zone after         4.82 M        0.21 μs  ±3679.13%       0.167 μs        0.33 μs
shift_zone before        0.81 M        1.24 μs   ±684.66%        1.17 μs        1.42 μs

Comparison: 
shift_zone after         4.82 M
shift_zone before        0.81 M - 5.98x slower +1.03 μs

Memory usage statistics:

Name                 Memory usage
shift_zone after          0.49 KB
shift_zone before            2 KB - 4.06x memory usage +1.51 KB

**All measurements for memory usage were the same**

Name                                                      ips        average  deviation         median         99th %
shift before (dedicated Etc/UTC clause)                3.94 M      253.62 ns  ±2409.75%         209 ns         375 ns
shift after (general clause + helper fast path)        2.79 M      358.79 ns  ±2435.38%         292 ns         459 ns

Comparison: 
shift before (dedicated Etc/UTC clause)                3.94 M
shift after (general clause + helper fast path)        2.79 M - 1.41x slower +105.17 ns

Memory usage statistics:

Name                                               Memory usage
shift before (dedicated Etc/UTC clause)                 0.83 KB
shift after (general clause + helper fast path)         1.15 KB - 1.39x memory usage +0.32 KB

**All measurements for memory usage were the same**

[josevalim]

I agree with this but then we should:

1. Change only around the call to the calendar module, instead of the whole function
2. Apply similar inlining to add/diff/shift

[reisub]

I agree with this but then we should:

1. Change only around the call to the calendar module, instead of the whole function
2. Apply similar inlining to add/diff/shift

Thanks, that makes sense.

I've pushed a commit to add a new function head for shift_zone_for_iso_days_utc which is then used by add/4, shift/3, shift_zone/3. That way we don't need separate function heads for the three public facing functions.

This does marginally slow down shift/3, but the absolute value is very small so I think it's acceptable tradeoff.

You also mentioned DateTime.diff/3, but it doesn't need to consult the TZ database so no changes there.

[josevalim]

💚 💙 💜 💛 ❤️