A command-line interface (CLI) tool to fetch CVE (Common Vulnerabilities and Exposures) and CPE (Common Platform Enumeration) data from NIST's National Vulnerability Database (NVD) API, along with CWE (Common Weakness Enumeration) data from MITRE.
- Multi-format Data Download: Fetch CVE, CPE, CPE Match, and CVE History data using NIST's NVD API 2.0
- Legacy URI Support: Download data using traditional URI-based methods
- Data Merging: Combine multiple data files into consolidated datasets
- Concurrent Processing: Multi-threaded download and processing for improved performance
- Flexible Output: Configurable output directories and file naming
- Query Filtering: Support for date ranges, modification filters, and pagination
- Java 21 or higher
- Maven 3.6 or higher
mvn clean packageThis creates an executable JAR file with all dependencies included in the target/ directory.
mvn clean post-clean packagemvn resources:copy-resources@generate-buildconfigThis project uses GitHub Actions for automated:
- CI: Automated testing with coverage validation (70% threshold)
- Code Quality: Formatting checks, SpotBugs security analysis, and PMD analysis
- Automated Releases: When a tag like
v0.1.0is created, the workflow automatically builds and attaches the executable JAR to the GitHub release
-
Create and push a tag following semantic versioning:
git tag v0.1.0 git push origin v0.1.0
-
Create a GitHub release using the tag - the CI/CD pipeline will automatically build and attach the executable JAR as a release asset.
Test the executable JAR:
java -jar target/nvd-tool-0.1.0-SNAPSHOT.jar --helpThe tool provides several commands for different operations:
java -jar nvd-tool-1.0-SNAPSHOT.jar [COMMAND] [OPTIONS]Download CVE/CPE data from NIST NVD repository and CWE data from MITRE.
java -jar nvd-tool-1.0-SNAPSHOT.jar download [SUBCOMMAND] [OPTIONS]Download Subcommands:
api- Download using NVD API 2.0uri- Download using traditional URI methods
# Download CVE data
java -jar nvd-tool-1.0-SNAPSHOT.jar download api cve [OPTIONS]
# Download CPE data
java -jar nvd-tool-1.0-SNAPSHOT.jar download api cpe [OPTIONS]
# Download CPE Match data
java -jar nvd-tool-1.0-SNAPSHOT.jar download api cpe-match [OPTIONS]
# Download CVE History data
java -jar nvd-tool-1.0-SNAPSHOT.jar download api cve-history [OPTIONS]Combine multiple downloaded data files.
java -jar nvd-tool-1.0-SNAPSHOT.jar merge [OPTIONS]-o, --output-dir DIR- Specify output directory--output-file FILE- Specify output filename-h, --help- Show help information-V, --version- Show version information
java -jar nvd-tool-1.0-SNAPSHOT.jar download api cve \
--output-dir ./data \
--last-mod-start-date 2024-01-01T00:00:00 \
--last-mod-end-date 2024-01-31T23:59:59java -jar nvd-tool-1.0-SNAPSHOT.jar download api cpe \
--output-dir ./cpe-data \
--results-per-page 500java -jar nvd-tool-1.0-SNAPSHOT.jar merge \
--input-dir ./data \
--output-file merged-cve-data.jsonThe tool uses a producer-consumer pattern for concurrent downloads with the following components:
sequenceDiagram
actor Coordinator
participant Producer
participant Queue as BlockingDeque
participant Consumer as ConsumerHelper
participant Tracker as RequestTracker
participant HttpUtil
Coordinator->>Producer: submit() (ExecutorService)
Producer->>Queue: put(QueueElement)
Note right of Producer: in finally -> addPoisonElements()<br/>-> put(poison QE)
alt Consumer threads running
Consumer->>Queue: take()
Queue-->>Consumer: QueueElement
Consumer->>Consumer: isPoisonPill(element)?
alt if poison
Consumer->>Consumer: log & break -> thread ends
else not poison
Consumer->>Tracker: addRequest(key) [may block]
alt Tracker full
Consumer->>Consumer: sleepQuietly(delay) and retry addRequest
end
Consumer->>HttpUtil: downloadHttpGetRequest(uri, outFile)
alt success
Consumer->>Consumer: success processing
else failure (e.g. 403)
Consumer->>Queue: offerFirst(element) [retry]
Consumer->>Consumer: incrementAttempts(), maybe sleepBackoff
end
end
end
classDiagram
class QueueElement {
-URI uri
-File outFile
-AtomicInteger attempts
+incrementAttempts()
+getAttempts()
+getKey()
}
class BaseProcessor {
-BlockingDeque~QueueElement~ downloadQueue
-FeedType feedType
-T poison
+createOutDir()
+getResults(URI, handler)
}
class ProducerHelper {
}
class ConsumerHelper {
-RequestTracker requestTracker
-IsPoisonPillFunction isPoisonPillFunction
+processQueue()
+consumeElement(QueueElement)
+addRequestToTracker(QueueElement)
}
class RequestTracker {
-Map~String, Long~ requestTimes
+addRequest(String) boolean
}
class IsPoisonPillFunction {
<<interface>>
+isPoisonPill(QueueElement) boolean
}
class StartIndexProducer
class DatesProducer
class StartIndexConsumer
class DatesConsumer
BaseProcessor <|-- StartIndexProducer
BaseProcessor <|-- DatesProducer
BaseProcessor <|-- StartIndexConsumer
BaseProcessor <|-- DatesConsumer
StartIndexProducer --> ProducerHelper : uses
DatesProducer --> ProducerHelper : uses
StartIndexConsumer --> ConsumerHelper : uses
DatesConsumer --> ConsumerHelper : uses
ConsumerHelper --> RequestTracker : uses
ConsumerHelper --> IsPoisonPillFunction : uses
ConsumerHelper --> QueueElement : processes
nvd-tool/
├── src/
│ ├── main/
│ │ ├── java/com/github/phanikb/nvd/
│ │ │ ├── cli/ # Command-line interface classes
│ │ │ ├── api2/ # Generated API 2.0 schema classes
│ │ │ ├── common/ # Common utilities and exceptions
│ │ │ ├── enums/ # Enumeration types
│ │ │ └── utils/ # Utility classes
│ │ └── resources/
│ │ ├── schema/ # JSON schema files
│ │ ├── spotbugs/ # SpotBugs configuration
│ │ └── pmd/ # PMD ruleset
│ └── test/
│ └── java/ # Unit tests
├── target/ # Build artifacts
├── logs/ # Application logs
├── pom.xml # Maven configuration
└── README.md
The project uses jsonschema2pojo-maven-plugin to generate Java classes from JSON schemas for:
- CVE API 2.0 schema
- CPE API 2.0 schema
- CPE Match API 2.0 schema
- CVE History API 2.0 schema
Key dependencies include:
- PicoCLI: Command-line interface framework
- Apache HttpClient 5: HTTP client for API requests
- Jackson: JSON processing
- Lombok: Boilerplate code reduction
- JUnit 5: Testing framework
- Mockito: Mocking framework for tests
- Log4j 2: Logging framework
Run the test suite:
mvn testRun code quality checks:
# Spotless formatting check
mvn spotless:check
# Apply Spotless formatting
mvn spotless:apply
# Run SpotBugs analysis
mvn spotbugs:check
# Run PMD analysis
mvn pmd:checkThe application uses Log4j 2 for logging. Log files are stored in the logs/ directory with automatic archiving.
- Ensure code follows the project's formatting standards (run
mvn spotless:apply) - Add appropriate unit tests for new functionality
- Run the full test suite before submitting changes
- Follow the existing code structure and naming conventions
This project is licensed under the terms specified in the LICENSE file.
This tool uses data from the NVD API but is not endorsed or certified by the NVD.
Happy vulnerability hunting! 🔍
![@dependabot[bot]](https://avatars.githubusercontent.com/in/29110?s=64&v=4){kind=small}