Skip to content

Commit 3cf76d3

Browse files
vicentereigclaude
andcommitted
feat: add instrumentation system for API cost tracking
- Add EventRegistry with thread-safe wildcard pattern subscriptions - Add BaseSubscriber base class for custom event subscribers - Add CostTracker built-in subscriber for tracking API costs - Emit exa.request.start, exa.request.complete, exa.request.error events - Add Endpoint T::Enum for typed endpoint identification - Unify CostDollars types into shared lib/exa/responses/cost.rb - Update all response types to use fully-typed CostDollars struct - Add comprehensive tests for instrumentation system - Update README with instrumentation documentation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent fe60134 commit 3cf76d3

File tree

16 files changed

+1294
-62
lines changed

16 files changed

+1294
-62
lines changed

README.md

Lines changed: 146 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,8 @@ This README is intentionally exhaustive—LLM agents and humans alike should be
2323
- [Events, Imports, Webhooks](#events-imports-webhooks)
2424
6. [Structured Output via Sorbet + dspy-schema](#structured-output-via-sorbet--dspy-schema)
2525
7. [Streaming & Transport Helpers](#streaming--transport-helpers)
26-
8. [Testing & TDD Plan](#testing--tdd-plan)
26+
8. [Instrumentation & Cost Tracking](#instrumentation--cost-tracking)
27+
9. [Testing & TDD Plan](#testing--tdd-plan)
2728

2829
---
2930

@@ -367,6 +368,150 @@ See `test/transport/stream_test.rb` for examples.
367368

368369
---
369370

371+
## Instrumentation & Cost Tracking
372+
373+
The gem includes a built-in instrumentation system for tracking API usage and costs. Events are emitted around every API request, and you can subscribe to them for logging, monitoring, or cost management.
374+
375+
### Basic Cost Tracking
376+
377+
```ruby
378+
require "exa"
379+
380+
client = Exa::Client.new(api_key: ENV.fetch("EXA_API_KEY"))
381+
382+
# Create and subscribe a cost tracker
383+
tracker = Exa::Instrumentation::CostTracker.new
384+
tracker.subscribe
385+
386+
# Make API calls - costs are tracked automatically
387+
response = client.search.search(query: "AI papers", num_results: 10)
388+
puts response.cost_dollars&.total # => 0.005
389+
390+
client.search.contents(urls: ["https://example.com"], text: true)
391+
392+
# Check accumulated costs
393+
puts tracker.total_cost # => 0.006
394+
puts tracker.request_count # => 2
395+
puts tracker.average_cost # => 0.003
396+
397+
# Get breakdown by endpoint
398+
tracker.summary.each do |endpoint, cost|
399+
puts "#{endpoint.serialize}: $#{cost}"
400+
end
401+
# => search: $0.005
402+
# => contents: $0.001
403+
404+
# Print a formatted report
405+
puts tracker.report
406+
407+
# Reset tracking
408+
tracker.reset!
409+
410+
# Unsubscribe when done
411+
tracker.unsubscribe
412+
```
413+
414+
### Custom Event Subscribers
415+
416+
Subscribe to specific events using wildcard patterns:
417+
418+
```ruby
419+
# Subscribe to all request events
420+
Exa.instrumentation.subscribe("exa.request.*") do |event_name, payload|
421+
case event_name
422+
when "exa.request.start"
423+
puts "Starting #{payload.endpoint.serialize} request..."
424+
when "exa.request.complete"
425+
puts "Completed in #{payload.duration_ms.round(2)}ms, cost: $#{payload.cost_dollars}"
426+
when "exa.request.error"
427+
puts "Error: #{payload.error_class} - #{payload.error_message}"
428+
end
429+
end
430+
431+
# Subscribe to specific events
432+
Exa.instrumentation.subscribe("exa.request.error") do |_name, payload|
433+
ErrorTracker.notify(payload.error_class, payload.error_message)
434+
end
435+
```
436+
437+
### Building Custom Subscribers
438+
439+
Extend `BaseSubscriber` for reusable instrumentation:
440+
441+
```ruby
442+
class BudgetGuard < Exa::Instrumentation::BaseSubscriber
443+
def initialize(budget_limit)
444+
@budget = budget_limit
445+
@spent = 0.0
446+
@mutex = Mutex.new
447+
super()
448+
end
449+
450+
def subscribe
451+
add_subscription("exa.request.complete") do |_name, payload|
452+
next unless payload.cost_dollars
453+
454+
@mutex.synchronize do
455+
@spent += payload.cost_dollars
456+
raise "Budget exceeded! Spent $#{@spent} of $#{@budget}" if @spent > @budget
457+
end
458+
end
459+
end
460+
461+
attr_reader :spent
462+
end
463+
464+
guard = BudgetGuard.new(1.00) # $1 budget
465+
guard.subscribe
466+
# ... make API calls ...
467+
guard.unsubscribe
468+
```
469+
470+
### Event Types
471+
472+
| Event | Payload | Description |
473+
|-------|---------|-------------|
474+
| `exa.request.start` | `RequestStart` | Emitted when a request begins |
475+
| `exa.request.complete` | `RequestComplete` | Emitted on successful completion |
476+
| `exa.request.error` | `RequestError` | Emitted when a request fails |
477+
478+
**RequestStart** fields: `request_id`, `endpoint`, `http_method`, `path`, `timestamp`
479+
480+
**RequestComplete** fields: `request_id`, `endpoint`, `duration_ms`, `status`, `cost_dollars`, `timestamp`
481+
482+
**RequestError** fields: `request_id`, `endpoint`, `duration_ms`, `error_class`, `error_message`, `timestamp`
483+
484+
### Async Compatibility
485+
486+
The instrumentation system is thread-safe and works inside `Async` blocks:
487+
488+
```ruby
489+
require "async"
490+
require "exa/internal/transport/async_requester"
491+
492+
tracker = Exa::Instrumentation::CostTracker.new
493+
tracker.subscribe
494+
495+
Async do
496+
requester = Exa::Internal::Transport::AsyncRequester.new
497+
client = Exa::Client.new(api_key: ENV.fetch("EXA_API_KEY"), requester: requester)
498+
499+
# Concurrent requests - all tracked safely
500+
tasks = 5.times.map do |i|
501+
Async { client.search.search(query: "query #{i}", num_results: 3) }
502+
end
503+
tasks.each(&:wait)
504+
505+
puts "Total cost for #{tracker.request_count} requests: $#{tracker.total_cost}"
506+
ensure
507+
requester.close
508+
end
509+
510+
tracker.unsubscribe
511+
```
512+
513+
---
514+
370515
## Testing & TDD Plan
371516

372517
Run the suite:

lib/exa.rb

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,30 @@
1212
require_relative "exa/types"
1313
require_relative "exa/responses"
1414
require_relative "exa/resources"
15+
require_relative "exa/instrumentation"
16+
17+
module Exa
18+
class << self
19+
# Returns the global instrumentation event registry.
20+
# Use this to subscribe to API events for cost tracking, logging, etc.
21+
#
22+
# @example Subscribe to all request events
23+
# Exa.instrumentation.subscribe('exa.request.*') do |event_name, payload|
24+
# puts "#{event_name}: #{payload.inspect}"
25+
# end
26+
#
27+
# @return [Exa::Instrumentation::EventRegistry]
28+
def instrumentation
29+
@instrumentation ||= Instrumentation::EventRegistry.new
30+
end
31+
32+
# Emit an event to all subscribers.
33+
# Primarily used internally by the transport layer.
34+
#
35+
# @param event_name [String] The event name (e.g., 'exa.request.complete')
36+
# @param payload [Object] The event payload
37+
def emit(event_name, payload)
38+
instrumentation.notify(event_name, payload)
39+
end
40+
end
41+
end

lib/exa/instrumentation.rb

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# frozen_string_literal: true
2+
3+
require "securerandom"
4+
5+
module Exa
6+
module Instrumentation
7+
# Thread-safe event registry for subscribing to and emitting API events.
8+
# Supports wildcard pattern matching (e.g., 'exa.request.*').
9+
class EventRegistry
10+
def initialize
11+
@listeners = {}
12+
@mutex = Mutex.new
13+
end
14+
15+
# Subscribe to events matching a pattern.
16+
# @param pattern [String] Event pattern (supports '*' wildcard)
17+
# @yield [event_name, payload] Block called when matching events are emitted
18+
# @return [String] Subscription ID for later unsubscription
19+
def subscribe(pattern, &block)
20+
return unless block_given?
21+
22+
subscription_id = SecureRandom.uuid
23+
@mutex.synchronize do
24+
@listeners[subscription_id] = {
25+
pattern: pattern,
26+
block: block
27+
}
28+
end
29+
30+
subscription_id
31+
end
32+
33+
# Unsubscribe from events.
34+
# @param subscription_id [String] The ID returned from subscribe
35+
def unsubscribe(subscription_id)
36+
@mutex.synchronize do
37+
@listeners.delete(subscription_id)
38+
end
39+
end
40+
41+
# Clear all listeners.
42+
def clear_listeners
43+
@mutex.synchronize do
44+
@listeners.clear
45+
end
46+
end
47+
48+
# Emit an event to all matching subscribers.
49+
# @param event_name [String] The event name (e.g., 'exa.request.complete')
50+
# @param payload [Object] The event payload (usually a typed struct)
51+
def notify(event_name, payload)
52+
# Take a snapshot of current listeners to avoid holding the mutex during execution
53+
matching_listeners = @mutex.synchronize do
54+
@listeners.select do |_id, listener|
55+
pattern_matches?(listener[:pattern], event_name)
56+
end.dup
57+
end
58+
59+
matching_listeners.each do |_id, listener|
60+
listener[:block].call(event_name, payload)
61+
rescue StandardError
62+
# Silently ignore listener errors to avoid breaking the main flow
63+
end
64+
end
65+
66+
# Returns the count of registered listeners.
67+
def listener_count
68+
@mutex.synchronize { @listeners.size }
69+
end
70+
71+
private
72+
73+
def pattern_matches?(pattern, event_name)
74+
if pattern.include?("*")
75+
# Convert wildcard pattern to regex
76+
# exa.request.* becomes ^exa\.request\..*$
77+
regex_pattern = "^#{Regexp.escape(pattern).gsub('\\*', '.*')}$"
78+
Regexp.new(regex_pattern).match?(event_name)
79+
else
80+
# Exact match
81+
pattern == event_name
82+
end
83+
end
84+
end
85+
86+
# Base class for creating event subscribers.
87+
# Subclasses should implement #subscribe to add subscriptions.
88+
#
89+
# @example
90+
# class MyCostTracker < Exa::Instrumentation::BaseSubscriber
91+
# def initialize
92+
# @total = 0.0
93+
# super()
94+
# end
95+
#
96+
# def subscribe
97+
# add_subscription('exa.request.complete') do |_, payload|
98+
# @total += payload.cost_dollars || 0
99+
# end
100+
# end
101+
# end
102+
class BaseSubscriber
103+
def initialize
104+
@subscriptions = []
105+
end
106+
107+
# Override to add subscriptions. Called automatically after initialize.
108+
def subscribe
109+
raise NotImplementedError, "Subclasses must implement #subscribe"
110+
end
111+
112+
# Unsubscribe from all registered subscriptions.
113+
def unsubscribe
114+
@subscriptions.each { |id| Exa.instrumentation.unsubscribe(id) }
115+
@subscriptions.clear
116+
end
117+
118+
protected
119+
120+
# Add a subscription to the global event registry.
121+
# @param pattern [String] Event pattern to match
122+
# @yield [event_name, payload] Block called for matching events
123+
# @return [String] Subscription ID
124+
def add_subscription(pattern, &block)
125+
subscription_id = Exa.instrumentation.subscribe(pattern, &block)
126+
@subscriptions << subscription_id
127+
subscription_id
128+
end
129+
end
130+
end
131+
end
132+
133+
require_relative "instrumentation/events"
134+
require_relative "instrumentation/cost_tracker"

0 commit comments

Comments
 (0)