Release v.10 (#6)

* build to invoke as bitkit

* Add skip option to symbol chunking functions

that way you can skip prefixes that muddy up the actual symbols

* Fill in README and get ready to publish

---------

Co-authored-by: eay <eay>

Author: emclrk

Cargo.toml

@@ -3,6 +3,15 @@ name = "rf-bitkit"
 version = "0.1.0"
 edition = "2024"
 license = "MIT OR Apache-2.0"
+description = "A collection of analysis functions for reverse engineering RF protocols"
+repository = "https://github.com/emclrk/rf-bitkit"
+readme = "README.md"
+keywords = ["rf", "sdr", "protocol", "bitstream", "reverse-engineering"]
+categories = ["command-line-utilities", "algorithms"]
+
+[[bin]]
+name = "bitkit"
+path = "src/main.rs"
 
 [dependencies]
 clap = { version = "4.6.1", features = ["derive"] }

README.md

@@ -1 +1,110 @@
-A Rust library with utilities for RF protocol reverse engineering
+# rf-bitkit
+
+A Rust library and CLI tool for reverse engineering RF protocols.
+
+I wrote this after finding URH's protocol analysis tab a little clunky and limited. rf-bitkit provides a collection of analysis functions for working with demodulated bitstreams - finding fixed and varying fields, identifying symbol alphabets, measuring entropy, and correlating captures. It accepts plain text files (one bitstream per line) or XML exports directly from URH.
+
+## Installation
+
+```
+cargo install rf-bitkit
+```
+
+Or build from source:
+
+```
+git clone https://github.com/emclrk/rf-bitkit
+cd rf-bitkit
+cargo install --path .
+```
+
+## Quick Example
+
+```
+$ bitkit infer my_captures.txt
+
+Inferred Structure:
+  Fixed(3) | Varying(2) | Fixed(2) | Varying(2) | Fixed(2) | Varying(2) | Fixed(1)
+```
+
+## CLI Tool
+
+The `bitkit` binary provides the following subcommands. All commands accept either a `.txt` file (one bitstream per line) or a URH `.xml` export.
+
+### `info`
+Show basic stats and a hex representation of each bitstream.
+```
+bitkit info <file> [-s <symlen>] [--skip <n>]
+```
+
+### `infer`
+Compute positionwise entropy and infer the protocol field structure. This is the key command — given a series of bitstreams, it identifies which bit positions are fixed across all captures and which vary.
+```
+bitkit infer <file> [--eps <tolerance>]
+```
+
+### `prefix`
+Find the common prefix across all bitstreams. A long common prefix is a preamble or sync word candidate.
+```
+bitkit prefix <file>
+```
+
+### `sweep`
+Show normalized entropy at each symbol length to help identify the correct symbol size. Look for a sudden drop in entropy — that's a signal that the chunking is aligning with the actual symbol boundaries.
+```
+bitkit sweep <file> [--max-symlen <n>] [--skip <n>]
+```
+
+### `alphabet`
+Show the symbol alphabet and frequency counts across all bitstreams at a given symbol length.
+```
+bitkit alphabet <file> [-s <symlen>] [--skip <n>]
+```
+
+### `substrings`
+Show the most frequently occurring substrings of a given length. Useful for finding sync word candidates.
+```
+bitkit substrings <file> [-l <len>] [-t <top>] [--skip <n>]
+```
+
+### `correlate`
+Cross-correlate two bitstreams from a file by index. Useful for identifying misalignment between captures.
+```
+bitkit correlate <file> -a <index> -b <index> [-t <top>]
+```
+
+## Library
+
+rf-bitkit is also a Rust library. Add it to your `Cargo.toml`:
+
+```toml
+[dependencies]
+rf-bitkit = "0.1.0"
+```
+
+Key functions:
+
+- `from_txt` / `from_urh` — load bitstreams from a text file or URH XML export
+- `positionwise_entropy` — compute per-bit-position entropy across a set of bitstreams
+- `ProtocolStructure::infer_structure` — infer fixed/varying field layout from entropy values
+- `get_alphabet_counts` — count symbol occurrences at a given symbol length
+- `get_substr_counts` — count substring occurrences across all bitstreams
+- `get_cross_correlation` — cross-correlate two bitstreams across all offsets
+- `get_hamming_dist` — compute Hamming distance between two bitstreams
+- `find_common_prefix` — find the longest prefix shared by all bitstreams
+
+## Status and Roadmap
+
+This is an early release. Current planned work includes:
+
+- Sync word detection in the presence of misaligned packets (cross-correlation is implemented; evaluating Smith-Waterman for handling bit insertions/deletions)
+- CRC/checksum detection
+- User-defined tags for labeling bitstream families (e.g. Frame A vs Frame B)
+- JSON/TOML config file support for scripting multi-step analyses
+- Visualizations
+
+Longer term, I'd like to build a DSP layer and work toward a standalone URH replacement in Rust.
+
+## License
+
+Licensed under either of [MIT](LICENSE-MIT) or [Apache-2.0](LICENSE-APACHE) at your option.

src/lib.rs

@@ -97,6 +97,9 @@ impl Bitstream {
                 .collect::<String>(),
         })
     }
+    pub fn skip(&self, nbits: usize) -> Self {
+        Self::new(self.bits[nbits..].to_string()).unwrap()
+    }
     /// Number of bits in the Bitstream
     pub fn len(&self) -> usize {
         self.bits.len()
@@ -315,30 +318,56 @@ pub fn positionwise_entropy(bitstrs: &[Bitstream]) -> Vec<f32> {
 }
 /// Find the symbol alphabet (the full set of symbols that actually occur)
 /// across multiple Bitstreams.
-pub fn get_alphabet(bitstrs: &[Bitstream], symlen: usize) -> HashSet<String> {
+pub fn get_alphabet(bitstrs: &[Bitstream], symlen: usize, skip_bits: usize) -> HashSet<String> {
     let mut counts: HashMap<String, u32> = HashMap::new();
     for bitstr in bitstrs {
-        bitstr.accumulate_sym_counts(symlen, &mut counts);
+        if skip_bits > 0 {
+            bitstr
+                .skip(skip_bits)
+                .accumulate_sym_counts(symlen, &mut counts);
+        } else {
+            bitstr.accumulate_sym_counts(symlen, &mut counts);
+        }
     }
     counts.into_keys().collect::<HashSet<String>>()
 }
 
 /// Return frequency counts for the symbol alphabet (the full set of symbols that actually occur)
 /// across multiple Bitstreams.
-pub fn get_alphabet_counts(bitstrs: &[Bitstream], symlen: usize) -> HashMap<String, u32> {
+pub fn get_alphabet_counts(
+    bitstrs: &[Bitstream],
+    symlen: usize,
+    skip_bits: usize,
+) -> HashMap<String, u32> {
     let mut counts: HashMap<String, u32> = HashMap::new();
     for bitstr in bitstrs {
-        bitstr.accumulate_sym_counts(symlen, &mut counts);
+        if skip_bits > 0 {
+            bitstr
+                .skip(skip_bits)
+                .accumulate_sym_counts(symlen, &mut counts);
+        } else {
+            bitstr.accumulate_sym_counts(symlen, &mut counts);
+        }
     }
     counts
 }
 
 /// Return frequency counts for all possible substrings of length `strlen` across multiple
 /// Bitstreams.
-pub fn get_substr_counts(bitstrs: &[Bitstream], strlen: usize) -> HashMap<String, u32> {
+pub fn get_substr_counts(
+    bitstrs: &[Bitstream],
+    strlen: usize,
+    skip_bits: usize,
+) -> HashMap<String, u32> {
     let mut counts: HashMap<String, u32> = HashMap::new();
     for bitstr in bitstrs {
-        bitstr.accumulate_substr_counts(strlen, &mut counts);
+        if skip_bits > 0 {
+            bitstr
+                .skip(skip_bits)
+                .accumulate_substr_counts(strlen, &mut counts);
+        } else {
+            bitstr.accumulate_substr_counts(strlen, &mut counts);
+        }
     }
     counts
 }
@@ -452,7 +481,7 @@ mod tests {
             .entry("100".to_string())
             .and_modify(|ct| *ct += 1);
         hash_result.insert("000".to_string(), 2);
-        assert_eq!(hash_result, get_substr_counts(&vec![bs, bs2], 3));
+        assert_eq!(hash_result, get_substr_counts(&vec![bs, bs2], 3, 0));
     }
     #[test]
     fn test_bit_pcts() {
@@ -471,21 +500,21 @@ mod tests {
                 .iter()
                 .map(|s| s.to_string())
                 .collect::<HashSet<String>>(),
-            get_alphabet(&vec![bs_1.clone()], 1)
+            get_alphabet(&vec![bs_1.clone()], 1, 0)
         );
         assert_eq!(
             ["110", "101", "011"]
                 .iter()
                 .map(|s| s.to_string())
                 .collect::<HashSet<String>>(),
-            get_alphabet(&vec![bs_1.clone()], 3)
+            get_alphabet(&vec![bs_1.clone()], 3, 0)
         );
         assert_eq!(
             ["1101", "0110", "1011", "0101", "0011"]
                 .iter()
                 .map(|s| s.to_string())
                 .collect::<HashSet<String>>(),
-            get_alphabet(&vec![bs_1.clone(), bs_2.clone()], 4)
+            get_alphabet(&vec![bs_1.clone(), bs_2.clone()], 4, 0)
         );
         let hash_result = HashMap::from([
             (String::from("1101"), 1),
@@ -494,7 +523,7 @@ mod tests {
             (String::from("0101"), 1),
             (String::from("0011"), 1),
         ]);
-        assert_eq!(hash_result, get_alphabet_counts(&vec![bs_1, bs_2], 4));
+        assert_eq!(hash_result, get_alphabet_counts(&vec![bs_1, bs_2], 4, 0));
     }
     #[test]
     fn test_get_total_entropy() {

src/main.rs

@@ -20,6 +20,8 @@ enum Commands {
         file: String,
         #[arg(short, long, default_value_t = 4)]
         symlen: usize,
+        #[arg(long, default_value_t = 0)]
+        skip: usize,
     },
     /// Find the common prefix across all bitstreams (preamble candidate)
     Prefix { file: String },
@@ -35,12 +37,16 @@ enum Commands {
         file: String,
         #[arg(long, default_value_t = 8)]
         max_symlen: usize,
+        #[arg(long, default_value_t = 0)]
+        skip: usize,
     },
     /// Show symbol alphabet and frequency counts across all bitstreams
     Alphabet {
         file: String,
         #[arg(short, long, default_value_t = 1)]
         symlen: usize,
+        #[arg(long, default_value_t = 0)]
+        skip: usize,
     },
     /// Show the most frequent substrings of a given length
     Substrings {
@@ -51,6 +57,8 @@ enum Commands {
         /// Number of results to show
         #[arg(short, long, default_value_t = 10)]
         top: usize,
+        #[arg(long, default_value_t = 0)]
+        skip: usize,
     },
     /// Cross-correlate two bitstreams from a file by index
     Correlate {
@@ -62,7 +70,7 @@ enum Commands {
         #[arg(short)]
         b: usize,
         /// Number of top results to show
-        #[arg(long, default_value_t = 10)]
+        #[arg(short, long, default_value_t = 10)]
         top: usize,
     },
 }
@@ -92,7 +100,7 @@ fn main() {
 
 fn run(cli: Cli) -> Result<(), BitkitError> {
     match cli.command {
-        Commands::Info { file, symlen } => {
+        Commands::Info { file, symlen, skip } => {
             let bitstrs = load_file(&file)?;
             let lengths: Vec<usize> = bitstrs.iter().map(|b| b.len()).collect();
             let min_len = lengths.iter().min().copied().unwrap_or(0);
@@ -104,7 +112,11 @@ fn run(cli: Cli) -> Result<(), BitkitError> {
             println!("Lengths: min={min_len}, max={max_len}, avg={avg_len:.1}");
             println!();
             for (i, bs) in bitstrs.iter().enumerate() {
-                println!("[{i:3}] {}  ({} bits)", bs.to_hex(symlen), bs.len());
+                println!(
+                    "[{i:3}] {}  ({} bits)",
+                    bs.skip(skip).to_hex(symlen),
+                    bs.len()
+                );
             }
         }
 
@@ -152,8 +164,13 @@ fn run(cli: Cli) -> Result<(), BitkitError> {
             }
         }
 
-        Commands::Sweep { file, max_symlen } => {
+        Commands::Sweep {
+            file,
+            max_symlen,
+            skip,
+        } => {
             let bitstrs = load_file(&file)?;
+            let skipped: Vec<Bitstream> = bitstrs.iter().map(|bs| bs.skip(skip)).collect();
             println!("=== Entropy Sweep: {file} ===");
             println!();
             println!(
@@ -163,28 +180,22 @@ fn run(cli: Cli) -> Result<(), BitkitError> {
 
             let mut results = Vec::new();
             for symlen in 1..=max_symlen {
-                let avg = bitstrs
+                let avg = skipped
                     .iter()
                     .map(|bs| bs.get_normed_entropy(symlen))
                     .sum::<f32>()
-                    / bitstrs.len() as f32;
-                let unique = get_alphabet_counts(&bitstrs, symlen).len();
+                    / skipped.len() as f32;
+                let unique = get_alphabet_counts(&bitstrs, symlen, skip).len();
                 results.push((symlen, avg, unique));
             }
-            let min_entropy = results.iter().map(|(_, e, _)| *e).fold(f32::MAX, f32::min);
             for (symlen, entropy, unique) in &results {
-                let marker = if (*entropy - min_entropy).abs() < 1e-6 {
-                    " <-- minimum"
-                } else {
-                    ""
-                };
-                println!("{:>8}  {:>14.4}  {:>12}{marker}", symlen, entropy, unique);
+                println!("{:>8}  {:>14.4}  {:>12}", symlen, entropy, unique);
             }
         }
 
-        Commands::Alphabet { file, symlen } => {
+        Commands::Alphabet { file, symlen, skip } => {
             let bitstrs = load_file(&file)?;
-            let counts = get_alphabet_counts(&bitstrs, symlen);
+            let counts = get_alphabet_counts(&bitstrs, symlen, skip);
             let mut sorted: Vec<_> = counts.iter().collect();
             sorted.sort_by(|a, b| b.1.cmp(a.1));
 
@@ -196,9 +207,14 @@ fn run(cli: Cli) -> Result<(), BitkitError> {
             }
         }
 
-        Commands::Substrings { file, len, top } => {
+        Commands::Substrings {
+            file,
+            len,
+            top,
+            skip,
+        } => {
             let bitstrs = load_file(&file)?;
-            let counts = get_substr_counts(&bitstrs, len);
+            let counts = get_substr_counts(&bitstrs, len, skip);
             let mut sorted: Vec<_> = counts.iter().collect();
             sorted.sort_by(|a, b| b.1.cmp(a.1));