Merge pull request #9 from lordixir/main

v2.4.0: fix critical bugs, optimize cache, clean dead code

Author: lordixir

CHANGELOG.md

@@ -2,12 +2,70 @@
 
 All notable changes to this project will be documented in this file.
 
+## [2.4.0] - 2026-02-22
+
+### Added
+- **Go Embed Version**: VERSION file embedded at compile time via `go:embed`
+- **Domain Validation**: `-d` flag now validates domain format before processing
+- **Backward-Compatible JSON**: `found_websites` + `founded_websites` dual output
+- **Config File Support**: `-config config.yaml` flag for YAML configuration
+- **Resume Support**: `-resume cache.json` flag to resume interrupted scans
+- **Output Directory**: `-output-dir ./exports` flag for export file location
+
+### Fixed
+- **Race Condition**: `PrintResult` protected with `sync.Once` (prevents double-call)
+- **DNS Key Mismatch**: Protocol prefix stripped before DNS result lookup
+- **CLI Flag Override**: `flag.Visit` ensures only explicitly set flags override config
+- **Windows Path Separator**: `filepath.Join` replaces hardcoded `/`
+- **Flaky DNS Test**: `minExpected` set to 0 for network-dependent test
+- **Chrome 131 Comment**: Updated stale comment to Chrome 135
+
+### Improved
+- **Cache Performance**: `IsScanned()` O(n) → O(1) with persistent `map[string]struct{}`
+- **JSON Field Name**: `founded_websites` → `found_websites` (grammar fix)
+
+### Removed
+- Unused `GetTransport()` method (conflicting transport settings)
+- Unused `AddSmartJitter()`, `SaveConfigFile()`, `ParseDNSServers()` functions
+- Unused `BatchReverseDNSWithResults()` function and `DNSResult` struct
+- Unused `PoolMetrics` struct and related atomic operations
+
+---
+
+## [2.3.0] - 2025-12-23
+
+### Added
+- **Batch DNS Lookup**: Parallel DNS resolution at end of scan (20 concurrent queries)
+- **Connection Pool Optimization**: Pre-warming, larger buffers, TLS session cache
+
+### Changed
+- **Chrome 135 User-Agent**: Updated from Chrome 131 to Chrome 135 for January 2026
+- **macOS 15 Sequoia**: Added new macOS version strings
+- **Windows 11 24H2**: Updated Windows version strings
+- Increased connection pool sizes: MaxIdle 200-1000, MaxPerHost 50-200
+- Extended idle timeout: 60s → 90s
+- Larger I/O buffers: 64KB read/write
+
+### Fixed
+- Batch DNS now correctly extracts IP from URL (removes https:// prefix)
+
+---
+
+## [2.2.2] - 2025-12-12
+
+### Fixed
+- **uTLS Transport Activated**: Chrome 131 TLS fingerprint now fully integrated into HTTP client
+- Fixed HTTP/2 compatibility issue by setting `ForceAttemptHTTP2: false`
+- uTLS transport now properly used for all HTTPS connections
+
+---
+
 ## [2.2.1] - 2025-12-11
 
 ### Added
 - **Referer Header Rotation**: Random referer from Google, Bing, DuckDuckGo for more realistic requests
-- **Smart Jitter Function**: `config.AddSmartJitter()` with occasional long pauses (1-3s) for natural patterns
-- **uTLS Transport**: Chrome 131 TLS fingerprint support (ready for integration)
+- **Smart Jitter Function**: `config.AddSmartJitter()` *(removed in v2.4.0)*
+- **uTLS Transport**: Chrome 131 TLS fingerprint support
 - New config constants: `DialTimeout`, `MaxJitterMs`
 - Unified RNG functions in config: `GetRandomInt`, `GetRandomString`, `ShuffleStrings`
 

CONTRIBUTING.md

@@ -56,21 +56,28 @@ go vet ./...
 
 ```
 ipmap/
-├── main.go              # Entry point, CLI flags
+├── main.go              # Entry point, CLI flags, go:embed VERSION
+├── VERSION              # Embedded version string
 ├── config/
-│   └── config.go        # Global config, RNG, jitter functions
+│   ├── config.go        # Global config, RNG, jitter, logging
+│   └── loader.go        # YAML config file loading
 ├── modules/
-│   ├── scanner.go       # Chrome 131 headers, uTLS transport
-│   ├── request.go       # HTTP client with retry
-│   ├── resolve_site.go  # Worker pool, IP scanning
+│   ├── scanner.go       # Chrome 135 headers, uTLS transport
+│   ├── request.go       # HTTP client with retry, connection pool
+│   ├── resolve_site.go  # Worker pool, IP scanning, batch DNS
 │   ├── get_site.go      # Site discovery per IP
+│   ├── get_domain_title.go # Domain title fetching
+│   ├── helpers.go       # Shared utilities (ExtractTitle)
+│   ├── cache.go         # Scan state persistence for resume
 │   ├── validators.go    # Input validation
 │   ├── rate_limiter.go  # Token bucket rate limiter
-│   └── ...
+│   ├── dns_resolver.go  # Batch reverse DNS lookups
+│   ├── result_print.go  # Result formatting and export
+│   ├── interrupt_handler.go # Ctrl+C handling
+│   └── calc_ip_address.go   # CIDR to IP calculation
 ├── tools/
 │   ├── find_asn.go      # ASN scanning
 │   └── find_ip.go       # IP block scanning
-├── bin/                 # Cross-platform builds
 └── README.md
 ```
 
@@ -113,10 +120,10 @@ When modifying the scanner module:
 
 1. **TLS Fingerprint**: Use `utls.HelloChrome_Auto` for latest Chrome fingerprint
 2. **Header Order**: Maintain exact Chrome header order (not alphabetical)
-3. **Accept-Encoding**: Include `zstd` for Chrome 131+
-4. **Jitter**: Use `config.AddJitter()` (0-200ms) or `config.AddSmartJitter()` (with occasional long pauses)
-5. **User-Agent**: Use Chrome 130+ versions only
-6. **Referer**: Rotate between Google, Bing, DuckDuckGo URLs
+3. **Accept-Encoding**: Include `zstd` for Chrome 135+
+4. **Jitter**: Use `config.AddJitter()` (0-200ms random delay)
+5. **User-Agent**: Use Chrome 133+ versions only
+6. **Referer**: Rotate between Google, Bing, DuckDuckGo, Yahoo URLs
 
 ## License
 

README.md

@@ -7,7 +7,7 @@ An open-source, cross-platform powerful network analysis tool for discovering we
 - ASN scanning (Autonomous System Number) with IPv4/IPv6 support
 - IP block scanning (CIDR format)
 - HTTPS/HTTP automatic fallback
-- **Chrome 131 TLS Fingerprint** (JA3/JA4 spoofing via uTLS)
+- **Chrome 135 TLS Fingerprint** (JA3/JA4 spoofing via uTLS)
 - **Real Chrome Header Order** (WAF bypass optimized)
 - **Referer Header Rotation** (Google, Bing, DuckDuckGo)
 - Firewall bypass techniques (IP shuffling, header randomization, smart jitter)
@@ -57,6 +57,11 @@ go build -o ipmap .
 -proxy http://127.0.0.1:8080         # Proxy URL (HTTP/HTTPS/SOCKS5)
 -rate 50                             # Rate limit (requests/sec, 0 = unlimited)
 -dns 8.8.8.8,1.1.1.1                 # Custom DNS servers
+-ipv6                                # Enable IPv6 scanning
+-config config.yaml                  # Load config from YAML file
+-resume cache.json                   # Resume interrupted scan from cache
+-output-dir ./exports                # Directory for export files
+-insecure=false                      # Enable TLS certificate verification
 ```
 
 ### Examples

VERSION

@@ -1 +1 @@
-2.2.1
+2.4.0

build.ps1

@@ -1,7 +1,7 @@
 # ipmap Multi-Platform Build Script
 # Builds for macOS (ARM64 + AMD64) and Linux (AMD64)
 
-$VERSION = "2.2.1"
+$VERSION = "2.2.2"
 $APP_NAME = "ipmap"
 $BUILD_DIR = "bin"
 
@@ -52,6 +52,19 @@ if ($LASTEXITCODE -eq 0) {
 }
 Write-Host ""
 
+# Build for Windows AMD64
+Write-Host "Building for Windows AMD64..." -ForegroundColor Yellow
+$env:GOOS = "windows"
+$env:GOARCH = "amd64"
+go build -o "$BUILD_DIR/${APP_NAME}_windows_amd64.exe" .
+if ($LASTEXITCODE -eq 0) {
+    Write-Host "SUCCESS: Windows AMD64 build completed" -ForegroundColor Green
+} else {
+    Write-Host "ERROR: Windows AMD64 build failed" -ForegroundColor Red
+    exit 1
+}
+Write-Host ""
+
 # Show file sizes
 Write-Host "Build Summary:" -ForegroundColor Cyan
 Write-Host "================================================" -ForegroundColor Gray

config/config.go

@@ -42,7 +42,7 @@ var (
 	Format string
 
 	// MaxRetries is the number of retry attempts for failed requests
-	MaxRetries int = 2
+	MaxRetries int = 0
 
 	// Workers is the number of concurrent scanning goroutines
 	Workers int = 100
@@ -55,6 +55,15 @@ var (
 
 	// DNSServers is the list of custom DNS servers
 	DNSServers []string
+
+	// EnableIPv6 enables IPv6 address scanning (default: false)
+	EnableIPv6 bool = false
+
+	// OutputDir is the directory for export files (default: current directory)
+	OutputDir string = ""
+
+	// InsecureSkipVerify skips TLS certificate verification (default: true for backward compatibility)
+	InsecureSkipVerify bool = true
 )
 
 // ====================================================================
@@ -104,19 +113,6 @@ func AddJitter() {
 	}
 }
 
-// AddSmartJitter adds intelligent jitter with occasional long pauses for more natural patterns
-// This helps bypass rate-based WAF detection
-func AddSmartJitter() {
-	base := 50 + GetRandomInt(150) // 50-200ms base
-
-	// 5% chance of a long pause (simulates user reading page)
-	if GetRandomInt(100) < 5 {
-		base += 1000 + GetRandomInt(2000) // +1-3 seconds
-	}
-
-	time.Sleep(time.Duration(base) * time.Millisecond)
-}
-
 // ====================================================================
 // LOGGING FUNCTIONS
 // ====================================================================

config/config_test.go

@@ -98,8 +98,8 @@ func TestInfoLog(t *testing.T) {
 }
 
 func TestConfigDefaults(t *testing.T) {
-	if MaxRetries != 2 {
-		t.Errorf("MaxRetries default should be 2, got %d", MaxRetries)
+	if MaxRetries != 0 {
+		t.Errorf("MaxRetries default should be 0 (retries disabled), got %d", MaxRetries)
 	}
 	if Workers != 100 {
 		t.Errorf("Workers default should be 100, got %d", Workers)

config/loader.go

@@ -0,0 +1,91 @@
+// Package config provides global configuration for the ipmap scanner.
+// loader.go handles configuration file loading and parsing.
+package config
+
+import (
+	"os"
+
+	"gopkg.in/yaml.v3"
+)
+
+// FileConfig represents the structure of the config.yaml file
+type FileConfig struct {
+	Workers    int      `yaml:"workers"`
+	Timeout    int      `yaml:"timeout"`
+	RateLimit  int      `yaml:"rate_limit"`
+	Proxy      string   `yaml:"proxy"`
+	DNSServers []string `yaml:"dns_servers"`
+	IPv6       bool     `yaml:"ipv6"`
+	Verbose    bool     `yaml:"verbose"`
+	Format     string   `yaml:"format"`
+}
+
+// LoadConfigFile loads configuration from a YAML file
+// Returns nil if file doesn't exist or is invalid
+func LoadConfigFile(path string) (*FileConfig, error) {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return nil, err
+	}
+
+	var cfg FileConfig
+	if err := yaml.Unmarshal(data, &cfg); err != nil {
+		return nil, err
+	}
+
+	return &cfg, nil
+}
+
+// ApplyFileConfig applies file configuration to global config
+// Only applies non-zero/non-empty values (allows CLI to override)
+func ApplyFileConfig(cfg *FileConfig) {
+	if cfg == nil {
+		return
+	}
+
+	if cfg.Workers > 0 {
+		Workers = cfg.Workers
+	}
+	if cfg.RateLimit > 0 {
+		RateLimit = cfg.RateLimit
+	}
+	if cfg.Proxy != "" {
+		ProxyURL = cfg.Proxy
+	}
+	if len(cfg.DNSServers) > 0 {
+		DNSServers = cfg.DNSServers
+	}
+	if cfg.IPv6 {
+		EnableIPv6 = cfg.IPv6
+	}
+	if cfg.Verbose {
+		Verbose = cfg.Verbose
+	}
+	if cfg.Format != "" {
+		Format = cfg.Format
+	}
+}
+
+// FindConfigFile looks for config file in common locations
+func FindConfigFile() string {
+	// Check common locations in order
+	locations := []string{
+		"config.yaml",
+		"config.yml",
+		".ipmap.yaml",
+		".ipmap.yml",
+	}
+
+	// Also check in user home directory
+	if home, err := os.UserHomeDir(); err == nil {
+		locations = append(locations, home+"/.ipmap.yaml", home+"/.ipmap.yml")
+	}
+
+	for _, loc := range locations {
+		if _, err := os.Stat(loc); err == nil {
+			return loc
+		}
+	}
+
+	return ""
+}

go.mod

@@ -6,6 +6,7 @@ require (
 	github.com/refraction-networking/utls v1.8.1
 	github.com/schollz/progressbar/v3 v3.14.1
 	golang.org/x/net v0.48.0
+	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
@@ -17,5 +18,4 @@ require (
 	golang.org/x/crypto v0.46.0 // indirect
 	golang.org/x/sys v0.39.0 // indirect
 	golang.org/x/term v0.38.0 // indirect
-	golang.org/x/text v0.32.0 // indirect
 )

go.sum

@@ -32,8 +32,8 @@ golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
 golang.org/x/term v0.14.0/go.mod h1:TySc+nGkYR6qt8km8wUhuFRTVSMIX3XPR58y2lC8vww=
 golang.org/x/term v0.38.0 h1:PQ5pkm/rLO6HnxFR7N2lJHOZX6Kez5Y1gDSJla6jo7Q=
 golang.org/x/term v0.38.0/go.mod h1:bSEAKrOT1W+VSu9TSCMtoGEOUcKxOKgl3LE5QEF/xVg=
-golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
-golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=