Phase 1: AI Tool Cost Comparison Plan¶

Roo Code plus Kong AI vs GitHub Copilot for Architecture Practice¶

This is Phase 1 of the Continuous Architecture Platform POC. It establishes the AI toolchain selection that will power all subsequent phases.

1 Goal¶

Determine the monthly cost per architect seat for two AI-assisted architecture toolchains:

The key question: What does it cost per month to give one solution architect AI-assisted tooling under each option?

GitHub Copilot has a simple per-seat monthly price. Roo Code + Kong AI has a usage-based cost model (per-token LLM charges routed through Kong AI) that depends on actual usage volume. To compare them fairly, we need to measure what a typical architecture month looks like in terms of token consumption, then price that against both models.

The comparison must be performed on a separate computer using entirely synthetic data so that no the enterprise proprietary information leaves the corporate environment.

2 Approach Summary¶

Phase 1 (This Computer): Use real corporate architecture data to create a synthetic dataset that is structurally representative but contains zero company data.

Phase 2 (Separate Computer): Use the synthetic dataset on a clean machine to execute a representative month of architecture work against both toolchains, measuring token usage and quality to calculate monthly cost per seat.

3 Data Sanitization Principles¶

These rules apply to ALL synthetic data produced in Phase 1:

4 What Needs to Be Synthesized¶

The architecture workflow involves the following artifacts and activities. Each must be recreated in synthetic form.

4.1 Corporate Service Registry (Synthetic corporate-services Equivalent)¶

Source on this machine: corporate-services/services/ (120+ OpenAPI/Swagger specs)

Synthetic equivalent to produce: - 15-20 fictional microservice Swagger/OpenAPI YAML specs - Each spec should be realistic in complexity (endpoints, schemas, error codes) - Service names from a fictional domain (e.g., a fictional streaming platform, or a fictional logistics company) - Cross-service references and shared models to preserve integration complexity

Approach: Select 15-20 representative real specs, study their structure and complexity, then write new specs in a completely different domain.

4.2 Corporate Architecture Diagrams (Synthetic Diagrams Equivalent)¶

Source on this machine: corporate-services/diagrams/ (PlantUML component, sequence, system diagrams)

Synthetic equivalent to produce: - 8-12 PlantUML component diagrams showing the synthetic microservice ecosystem - Include standard diagram patterns: include files, macros, component groupings, Kafka topics - Preserve the architectural style (box groupings, color coding, participant naming)

Approach: Study the diagram structure and style, create equivalent diagrams for the synthetic domain.

4.3 Ticket Workflow Artifacts¶

Source on this machine: private-analysis/docs/work-items/tickets/ (9 ticket folders with full analysis)

Synthetic equivalent to produce: - 3-5 synthetic ticket folders, each at different stages of completeness - Each ticket folder following the standard structure:

_SYN-XXXXX-brief-title/
  SYN-XXXXX-solution-design.md
  1.requirements/
    SYN-XXXXX.ticket.report.md
  2.analysis/
    simple.explanation.md
  3.solution/
    a.assumptions/assumptions.md
    b.current.state/investigations.md
    c.decisions/decisions.md
    g.guidance/guidance.md
    i.impacts/
      impacts.md
      impact.1/impact.1.md
      impact.2/impact.2.md
    r.risks/risks.md
    u.user.stories/user-stories.md

Ticket complexity tiers to cover:

Synthesis Mapping: Real Tickets to Synthetic Equivalents

Each synthetic ticket is structurally based on a real internal ticket. The domain, names, services, business rules, and all identifiable content are fully replaced, but the structural complexity, artifact count, comment thread depth, and iterative clarification patterns are preserved.

4.4 Architecture Standards Framework (Public, Pre-existing)¶

Instead of sanitizing our internal standards, we will adopt an existing public architecture standards framework. This eliminates IP risk entirely and adds credibility (the AI tools must work with real, industry-recognized standards, not something we invented for the test).

Primary framework: arc42 (https://arc42.org)

arc42 is an open-source, Creative Commons licensed architecture documentation template used industry-wide. It provides:

Download format: GitHub Markdown Multi-Page (with help text) from arc42-template-EN-withhelp-gitHubMarkdownMP.zip

Supplementary frameworks to layer on top:

What to download and include in the synthetic workspace: 1. arc42 template (GitHub Markdown MP with help) - 12 section files with detailed guidance 2. arc42 documentation site content (144 tips, 35 examples) - scraped or referenced 3. MADR template and examples from GitHub 4. ADR templates collection (Nygard, Tyree-Akerman, business case, Alexandrian) 5. C4 model diagramming rules and notation standards 6. arc42 Quality Model properties list

Then create a .ai-instructions/ directory that: - Tells the AI agent to follow arc42 structure for all documentation - Specifies which ADR template to use for decisions - Mandates C4 model compliance for all diagrams - Defines the ticket-to-arc42 mapping (how a JIRA ticket becomes arc42 artifacts) - Includes formatting rules, header conventions, and cross-referencing standards - This instruction file IS synthetic but is small, contains no company data, and simply wires the public frameworks together

Why this is better than sanitizing our standards: - ZERO risk of company data leakage (all public domain / Creative Commons) - More rigorous test (real industry standards are more comprehensive than custom ones) - More convincing comparison (stakeholders can verify the standards independently) - Saves significant effort (no need to rewrite 40+ customization documents)

4.5 Source Code References (Synthetic Microservice Code)¶

Source on this machine: source-code/microservices/ (10 reference microservices)

Synthetic equivalent to produce: - 5-8 synthetic microservice skeletons (Java/Spring Boot or similar) - Each with representative structure: controllers, services, repositories, DTOs, configs - Enough code to simulate "read source code for investigation" workflow steps - Not full implementations, just enough structure and placeholder logic

4.6 Tool Scripts (Synthetic Tooling)¶

Source on this machine: private-analysis/scripts/ (JIRA client, GitLab client, Elastic searcher)

Synthetic equivalent to produce: - Synthetic JIRA ticket extraction script (outputs synthetic ticket JSON) - Synthetic merge request analysis script (outputs synthetic MR details) - Synthetic Elastic query script (outputs synthetic log search results) - These scripts should produce canned/mock outputs rather than calling real APIs

4.7 Workspace Configuration¶

Source on this machine: .code-workspace, .vscode/, .github/copilot-instructions.md

Synthetic equivalent to produce: - A synthetic .code-workspace file with the correct multi-root structure - Synthetic VS Code settings - Synthetic Roo Code configuration (.roo/ equivalent) - Synthetic GitHub Copilot workspace instructions

5 Workflow Scenarios to Reproduce¶

These are the specific architecture tasks to execute on both toolchains for cost comparison. Each scenario should be documented as a "script" that can be followed identically on both tools.

Scenario 1: New Ticket Intake and Classification¶

Steps: 1. Query synthetic JIRA for open tickets 2. Select a ticket and extract details 3. Classify the ticket (architecture-relevant vs code bug) 4. Create the ticket workspace folder structure 5. Generate the simple explanation document

Measures: Input tokens, output tokens, API calls, wall clock time, output quality

Scenario 2: Current State Investigation¶

Steps: 1. Read the synthetic JIRA ticket report 2. Identify affected microservices from the ticket description 3. Analyze relevant Swagger/OpenAPI specs 4. Review relevant source code 5. Query synthetic Elastic logs for error patterns 6. Produce current state investigation documents

Measures: Input tokens, output tokens, API calls, wall clock time, output quality

Scenario 3: Solution Design Creation¶

Steps: 1. Using the investigation from Scenario 2, create a solution design 2. Identify impacted components 3. Create impact documents for each component 4. Create PlantUML sequence diagrams (target state) 5. Create assumptions, decisions, and risks documents 6. Generate user stories with acceptance criteria

Measures: Input tokens, output tokens, API calls, wall clock time, output quality

Scenario 4: Merge Request Review¶

Steps: 1. Analyze a synthetic merge request (Swagger spec changes, PlantUML changes) 2. Validate changes against the solution design 3. Produce a peer review document with findings

Measures: Input tokens, output tokens, API calls, wall clock time, output quality

Scenario 5: Confluence Publishing Preparation¶

Steps: 1. Review all documents with CONFLUENCE-PUBLISH markers 2. Validate document formatting and structure against standards 3. Verify cross-references and links between documents 4. Produce a publishing readiness checklist

Measures: Input tokens, output tokens, API calls, wall clock time, output quality

Monthly Volume Estimation¶

To convert scenario-level measurements into a monthly per-seat cost, we need to estimate how many of each scenario a single architect performs per month:

These frequencies will be validated during Phase 1 by examining actual ticket history patterns (using only counts and frequencies, not content).

6 Monthly Cost Per Seat Calculation¶

Per-Scenario Metrics to Capture¶

Monthly Cost Formula¶

Option A: Roo Code + Kong AI (usage-based)

Monthly Cost Per Seat =
  (Total Monthly Input Tokens x Input Token Price)
  + (Total Monthly Output Tokens x Output Token Price)
  + Kong AI Gateway Monthly Fee (if any per-seat component)
  + Roo Code License (if any)

Where: - Total Monthly Input Tokens = SUM(scenario input tokens x monthly frequency) across all scenarios - Total Monthly Output Tokens = SUM(scenario output tokens x monthly frequency) across all scenarios - Token prices come from AWS Bedrock pricing for the Claude model used via Kong AI

Option B: GitHub Copilot (seat-based)

Monthly Cost Per Seat =
  Copilot Business/Enterprise subscription per seat
  + Premium model request overage (if applicable)

GitHub Copilot Business is currently $19/seat/month. Copilot Enterprise is $39/seat/month. Premium requests for models like Claude Sonnet or GPT-4o may have usage limits with overage charges.

Pricing Inputs to Research (Phase 2)¶

Comparison Output¶

The final deliverable will be a table like:

7 Execution Plan¶

Phase 1: Synthetic Data Generation (This Computer)¶

Total Phase 1 estimate: 24-34 hours

Phase 2: Cost Comparison Execution (Separate Computer)¶

Total Phase 2 estimate: 14-24 hours

8 Sanitization Audit Checklist¶

Before transferring the synthetic workspace to the separate computer, run these checks:

# Search for any real microservice names
grep -ri "REAL-SVC-1\|REAL-SVC-2\|REAL-SVC-3" .  # Replace with actual internal service name patterns

# Search for any real ticket IDs
grep -ri "UPT-\|ARCH-\|UDX-" .

# Search for any real URLs
grep -ri "nbcu-ot\|atlassian\|gitlab\.use\|ucdp\.net\|kong\." .

# Search for any real people names (add known names)
grep -ri "blaisdell\|christopher\|specific-team-member-names" .

# Search for any real infrastructure references
grep -ri "bedrock\|elastic\|kibana\|confluence\|silverpop" .

# Search for "Universal" or company references
grep -ri "universal\|nbcuniversal\|comcast\|UPR\|UDX\|theme park\|resort" .

9 Deliverables¶

Everything below is what gets packaged and transferred to the other PC. The deliverable is a single portable directory that can be opened as a VS Code workspace immediately.

9.1 Portable Workspace Root Structure¶

ai-tool-cost-comparison-workspace/
  ai-tool-cost-comparison.code-workspace          # VS Code multi-root workspace file
  .github/
    copilot-instructions.md                        # GitHub Copilot workspace-level instructions
  .roo/                                            # Roo Code configuration equivalent
    rules/
  .vscode/
    settings.json                                  # VS Code workspace settings
  .ai-instructions/                                # AI agent instructions (wires public frameworks together)
    main-instructions.md                           # Primary AI agent instructions
    standards/                                     # Pointers and rules referencing arc42, C4, MADR
    templates/                                     # Solution design, impact, ADR templates
  architecture-standards/                          # PUBLIC FRAMEWORKS (downloaded, not synthesized)
    arc42/                                         # arc42 template (GitHub Markdown MP with help)
    adr-templates/                                 # ADR template collection (Nygard, Tyree-Akerman, etc.)
    madr/                                          # MADR template and examples
    c4-model/                                      # C4 model diagramming rules and notation
    quality-model/                                 # arc42 Quality Model properties
  corporate-services/                              # Synthetic service registry (equivalent of corporate-services)
    services/                                      # 15-20 synthetic OpenAPI/Swagger YAML specs
    diagrams/                                      # 8-12 synthetic PlantUML diagrams
      Components/
      Sequence/
      System/
      include.puml
      templates.puml
  source-code/                                     # Synthetic microservice code skeletons
    microservices/
      svc-one/                                     # 5-8 synthetic Java/Spring Boot skeletons
      svc-two/
      ...
  work-items/                                      # Synthetic ticket workspace (analysis docs)
    tickets/
      _SYN-10001-brief-title/                      # Simple tier ticket (complete)
      _SYN-10002-brief-title/                      # Medium tier ticket (complete)
      _SYN-10003-brief-title/                      # Complex tier ticket (complete)
      _SYN-10004-brief-title/                      # Ticket at investigation stage (partial)
      _SYN-10005-brief-title/                      # Ticket at intake stage (minimal)
  scripts/                                         # Synthetic tooling with mock outputs
    mock-jira-client.py
    mock-gitlab-client.py
    mock-elastic-searcher.py
    mock-data/                                     # Canned JSON responses for each script
  scenario-playbooks/                              # Step-by-step execution scripts
    scenario-1-ticket-intake.md
    scenario-2-current-state-investigation.md
    scenario-3-solution-design-creation.md
    scenario-4-merge-request-review.md
    scenario-5-publishing-preparation.md
    measurement-protocol.md                        # How to capture tokens, time, quality
  audit/                                           # Sanitization evidence
    sanitization-audit-log.md
    grep-results.txt

9.2 Deliverable Details¶

D1 Corporate Services (Synthetic)¶

D2 Architecture Standards (Public Frameworks)¶

D3 Ticket Workflow Artifacts (Synthetic)¶

Each complete ticket follows this structure:

_SYN-XXXXX-brief-title/
  SYN-XXXXX-solution-design.md
  1.requirements/
    SYN-XXXXX.ticket.report.md
  2.analysis/
    simple.explanation.md
  3.solution/
    a.assumptions/assumptions.md
    c.current.state/investigations.md
    d.decisions/decisions.md
    g.guidance/guidance.md
    i.impacts/
      impacts.md
      impact.1/impact.1.md    (+ .puml + .svg diagrams)
      impact.N/impact.N.md
    r.risks/risks.md
    u.user.stories/user-stories.md

D4 Source Code Skeletons (Synthetic)¶

D5 Tool Scripts (Synthetic with Mock Data)¶

Mock data files (JSON): - mock-data/jira-tickets.json - 8-12 synthetic tickets at various statuses - mock-data/jira-ticket-SYN-10001.json through SYN-10005.json - Individual ticket details - mock-data/gitlab-mr-101.json through mr-103.json - 3 synthetic merge requests - mock-data/elastic-error-logs.json - Synthetic error log search results - mock-data/elastic-api-traffic.json - Synthetic API traffic patterns

D6 AI Agent Instructions (Synthetic, Minimal)¶

D7 Scenario Playbooks¶

Each playbook includes: - Pre-conditions: What must exist before starting - Exact prompts: The literal text to send to each AI tool (identical wording for both) - Expected outputs: What files/artifacts should be created - Measurement checkpoints: When to record token counts and timing - Quality rubric: How to score output on a 1-5 scale per criterion - Standards compliance checklist: Specific arc42/C4/MADR rules to verify

D8 Sanitization Audit Log¶

9.3 Transfer Method¶

The entire ai-tool-cost-comparison-workspace/ directory will be: 1. Compressed into a single .zip or .tar.gz archive 2. Verified against the sanitization audit checklist one final time 3. Transferred to the separate computer via USB drive or secure file transfer 4. Extracted and opened as a VS Code workspace on the target machine

10 Fictional Domain Suggestions¶

The synthetic data needs a plausible domain that is complex enough to mimic theme park hospitality architecture. Suggestions:

Recommendation: NovaTrek Adventures or MetroFleet Logistics - different enough from the original domain to avoid any accidental overlap, but complex enough to exercise all the architecture patterns.

11 Next Steps¶

1. Choose the fictional domain for synthetic data
2. Begin Phase 1 execution, starting with Step 1.1
3. Track progress in this document by updating the execution plan table
4. Review each synthetic artifact against the sanitization checklist before moving on