Project 26: AI-Powered Diagnostic Agent for Edge Devices

[MrMajnu112]

Hi mentors,

During my exploration of Project 26, I've been pondering on the architectural trade-offs between:

A) A purely LLM-based diagnostic pipeline

B) A hybrid approach that uses a combination of deterministic rule-based log pattern matching, lightweight embedding retrieval (RAG), and LLM for root cause summarization

Considering the edge device limitations (in terms of memory, connectivity, and CPU-only inference), I'm trying to comprehend the trade-offs between:

Is there a preference for a deterministic anomaly detection-focused diagnostic agent that uses the LLM for additional reasoning?

Is there an advantage to utilizing small-sized OpenVINO-optimized language models (e.g., 3B-7B quantized models) over API-based models?

Is there an assumption that this project would interface with existing OpenVINO telemetry/benchmarking tools?

Also, I'd like to understand the evaluation criteria better:

What would mentors consider successful from a measurable standpoint?

Diagnosis accuracy?

Latency?

Resource optimization on constrained devices?

Decreased manual debugging efforts?

It would be great to understand which architectural path would best align with the overall vision for OpenVINO.

[andy-vm]

Hi @MrMajnu112
Please take note that the agent is running in a resource constrained environment.

You are free to use the tech stacks on your own preference, the outcome is a solution to convert from 10,000+ raw logs to few useful insights without burning the CPU.

For other details, you may refer to this discussion below
#34406