Adding Skills for AI Agents (for users, not devs)#21634
Draft
ktsaou wants to merge 3 commits intonetdata:masterfrom
Draft
Adding Skills for AI Agents (for users, not devs)#21634ktsaou wants to merge 3 commits intonetdata:masterfrom
ktsaou wants to merge 3 commits intonetdata:masterfrom
Conversation
Add the new node states documentation page to the navigation map under Netdata Cloud section with appropriate keywords.
Contributor
There was a problem hiding this comment.
2 issues found across 19 files
Confidence score: 3/5
- The most significant concern is in
src/ai-skills/tests/framework/round_trip.py: the test runner never loads test cases, so it can silently skip all tests and hide regressions. src/ai-skills/tests/framework/normalize.pystrips indentation before checking for continuations, which can misclassify indented:lines as new keys and lead to incorrect normalization behavior.- Given the test framework issues and potential false negatives, there is some regression risk despite the changes being localized.
- Pay close attention to
src/ai-skills/tests/framework/round_trip.pyandsrc/ai-skills/tests/framework/normalize.py- test runner loading and indentation handling.
Prompt for AI agents (all issues)
Check if these issues are valid — if so, understand the root cause of each and fix them.
<file name="src/ai-skills/tests/framework/normalize.py">
<violation number="1" location="src/ai-skills/tests/framework/normalize.py:126">
P2: Stripping the line before the indentation check makes `line.startswith(' ')` always false, so indented continuation lines containing `:` are incorrectly treated as new keys. Preserve indentation for the key/continuation check to avoid mis-parsing multiline values.</violation>
</file>
<file name="src/ai-skills/tests/framework/round_trip.py">
<violation number="1" location="src/ai-skills/tests/framework/round_trip.py:337">
P2: The test runner never loads test cases (placeholder list stays empty), so it always exits early without running any tests. Implement basic test case loading from the provided path to make the runner functional.</violation>
</file>
Architecture diagram
sequenceDiagram
participant CLI as Test Runner<br/>(skill_tester.py)
participant FS as File System
participant Norm as Normalizer<br/>(normalize.py)
participant Model as LLM Client<br/>(models.py)
participant API as External AI API<br/>(OpenAI/Compatible)
Note over CLI,API: NEW: AI Skill Validation Flow (Three-Call Pattern)
CLI->>FS: Load Skill Content (System Prompt)
FS-->>CLI: netdata-alerts-config.md
CLI->>FS: Load Test Case (Original Artifact)
FS-->>CLI: alert_case.conf
rect rgb(20, 25, 35)
Note over CLI,API: Phase 1: Description (Artifact -> Natural Language)
CLI->>Model: call_model(System=Skill, User=Original)
Model->>API: POST /chat/completions
API-->>Model: Response (Description)
Model-->>CLI: Natural Language Description
end
rect rgb(20, 25, 35)
Note over CLI,API: Phase 2: Recreation (Natural Language -> Artifact)
CLI->>Model: call_model(System=Skill, User=Description)
Model->>API: POST /chat/completions
API-->>Model: Response (Regenerated Config)
Model-->>CLI: Regenerated Artifact
end
rect rgb(30, 20, 20)
Note over CLI,Norm: Phase 3: Validation & Assessment
CLI->>Norm: normalize_alert(Original)
CLI->>Norm: normalize_alert(Regenerated)
CLI->>Norm: alerts_equal(NormOrig, NormRegen)
Norm-->>CLI: Boolean Match + Differences List
alt Mismatch or Analysis Mode
CLI->>Model: run_step3_assess(Diffs, Reasoning)
Model->>API: POST /chat/completions
API-->>Model: JSON Assessment
Model-->>CLI: {is_equivalent, errors, suggestions}
end
end
CLI->>FS: Save Results (JSON)
CLI-->>CLI: Print Summary (PASS/FAIL)
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| current_key = None | ||
| current_value = [] | ||
|
|
||
| for line in config_text.strip().split('\n'): |
Contributor
There was a problem hiding this comment.
P2: Stripping the line before the indentation check makes line.startswith(' ') always false, so indented continuation lines containing : are incorrectly treated as new keys. Preserve indentation for the key/continuation check to avoid mis-parsing multiline values.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/ai-skills/tests/framework/normalize.py, line 126:
<comment>Stripping the line before the indentation check makes `line.startswith(' ')` always false, so indented continuation lines containing `:` are incorrectly treated as new keys. Preserve indentation for the key/continuation check to avoid mis-parsing multiline values.</comment>
<file context>
@@ -0,0 +1,238 @@
+ current_key = None
+ current_value = []
+
+ for line in config_text.strip().split('\n'):
+ line = line.strip()
+
</file context>
| # Load test cases | ||
| # TODO: Implement test case loading based on format | ||
| print("Loading test cases...") | ||
| test_cases = [] # Placeholder |
Contributor
There was a problem hiding this comment.
P2: The test runner never loads test cases (placeholder list stays empty), so it always exits early without running any tests. Implement basic test case loading from the provided path to make the runner functional.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/ai-skills/tests/framework/round_trip.py, line 337:
<comment>The test runner never loads test cases (placeholder list stays empty), so it always exits early without running any tests. Implement basic test case loading from the provided path to make the runner functional.</comment>
<file context>
@@ -0,0 +1,371 @@
+ # Load test cases
+ # TODO: Implement test case loading based on format
+ print("Loading test cases...")
+ test_cases = [] # Placeholder
+
+ if not test_cases:
</file context>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. WIP.
Planned skills:
Summary by cubic
Adds AI-friendly Netdata skills and a testing framework, enabling assistants to configure alerts and query the API accurately. Aligns health docs with current alert semantics and syntax.
New Features
Documentation
Written for commit f25fe02. Summary will update on new commits.