Claude.Bricks

Claude.Bricks is two things at once. It is a system that generates modular LEGO buildings using Claude Code and PowerShell scripts that output LDraw (.ldr) files. It is also a working Azure ML platform that covers the Microsoft AI-300 (Operationalizing Machine Learning and Generative AI Solutions) exam objectives.

The project started as a DP-100 study lab and has been retooled for AI-300. The shift in emphasis is significant: AI-300 cares more about provisioning, versioning, promoting, evaluating, monitoring, and governing ML and GenAI systems than about building models from scratch.

The LEGO domain gives every scenario real data to work with. Images of rendered buildings feed the classifier. Structured .ldr files drive anomaly detection. Part-usage statistics power clustering. Natural language descriptions generate building specifications through a RAG pipeline. Four ML scenarios, all backed by Bicep provisioning, GitHub Actions CI/CD, and operational tooling (endpoints, evaluation, drift monitoring).

• Bicep-provisioned infrastructure (16 modules, dev/test environments)
• Five GitHub Actions CI/CD workflows (infra, training, deployment, evaluation, drift)
• Four ML scenarios with full operational lifecycle
• Foundry-native GenAI path with evaluation and monitoring
• Prompt versioning and A/B experiment framework
• Blue/green endpoint deployment with safe rollback
• Drift detection and automated retraining triggers
• Observability dashboards and KQL queries
• Cost analysis and teardown procedures

Bicep Module Inventory

#	Module	Resource	Purpose
1	storage-account.bicep	Storage Account	Blob containers for training data, models, images
2	key-vault.bicep	Key Vault	Secrets, connection strings, API keys
3	acr.bicep	Container Registry	Custom environment images
4	log-analytics.bicep	Log Analytics Workspace	Central logging and diagnostics
5	app-insights.bicep	Application Insights	Endpoint telemetry, latency tracking
6	ml-workspace.bicep	AML Workspace	Core ML workspace (depends on 1-5)
7	compute-cluster.bicep	Compute Cluster	CPU/GPU training clusters
8	compute-instance.bicep	Compute Instance	Dev VM for notebook work
9	openai.bicep	Azure OpenAI	GPT-4o-mini deployment (conditional)
10	ai-search.bicep	AI Search	Vector index for RAG (conditional)
11	custom-vision.bicep	Custom Vision	Training + prediction (conditional)
12	foundry-hub.bicep	AI Foundry Hub	Foundry management layer (conditional)
13	foundry-project.bicep	AI Foundry Project	Scenario 4 Foundry workspace (conditional)
14	aml-registry.bicep	AML Registry	Cross-workspace model promotion
15	rbac.bicep	Role Assignments	Service principal and managed identity roles
16	diagnostics.bicep	Diagnostic Settings	Route resource logs to Log Analytics

Deployment Command Flow

Wrapper scripts in scripts/deploy/ handle login, subscription selection, resource group creation, and deployment in one command. The flow follows a validate-preview-deploy pattern.

Scripts: scripts/deploy/deploy-infra.sh (bash), scripts/deploy/deploy-infra.ps1 (PowerShell), scripts/deploy/whatif.sh (preview only), scripts/deploy/teardown.sh (cleanup).

A complete Terraform implementation exists in deploymentcode/terraform/ and an earlier version in tf/. It provisions the same resources as Bicep using HashiCorp Configuration Language. Both paths produce equivalent Azure environments.

Terraform remains in the repository for reference and for teams that prefer the HashiCorp ecosystem. The Bicep path is documented as primary because it integrates natively with Azure CLI and requires no external tooling beyond az.

Two resource groups separate dev and test workloads. Parameter files in deploymentcode/bicep/parameters/ control what differs between environments.

Parameter	Dev (rg-claudebricks-dev)	Test (rg-claudebricks-test)
Compute cluster SKU	Standard_DS3_v2 (low cost)	Standard_DS3_v2
Compute min nodes	0	0
Compute max nodes	2	4
OpenAI deployment	Enabled	Enabled
AI Search	Enabled	Enabled
Custom Vision	Enabled	Disabled (cost savings)
Foundry Hub	Enabled	Enabled
Storage redundancy	LRS	LRS
Key Vault soft delete	7 days	90 days
Diagnostic logging	Minimal	Full

The CI/CD pipeline deploys to dev first, waits for manual approval, then deploys to test with the production-like parameter set.

Registration Scripts

Script	Layer	What It Registers
register_data_assets.py	2	4 data assets: facade-images:1 (uri_folder), ldr-files:1 (uri_folder), ldr-validation-features:1 (uri_file), reference-models:1 (uri_folder)
register_envs.py	2	2 environments: claudebricks-sklearn:1 (scikit-learn, Pillow, MLflow), claudebricks-clustering:1 (scikit-learn, UMAP, HDBSCAN, MLflow)
register_components.py	3	3 components: extract-features, run-clustering, evaluate-model (reusable pipeline steps with defined inputs/outputs)
publish_to_registry.py	4	Copies selected assets (models, environments, components) from the workspace to the shared AML registry for cross-workspace use

Shared AML Registry

The AML registry (claudebricks-registry) provides a central catalog that both dev and test workspaces can pull from. Models registered in dev can be promoted to the registry, then consumed by test without retraining. The same applies to environments and components.

This enables a clean promotion path: train in dev, validate in dev, publish to registry, consume in test, deploy to production endpoints.

Blue/Green Deployment

Scenario 1 is the primary demonstration of safe deployment. The endpoint supports two named deployments (blue and green) with traffic splitting controlled by percentage.

Drift monitoring runs weekly on this scenario via the drift-check.yml workflow. If the Population Stability Index (PSI) exceeds the threshold, a retraining job is triggered automatically.

Scripts

Script	Purpose
prepare_data.py	Upload facade images to blob, create versioned data asset
train.py	Training logic (runs on cluster): load images, train RandomForest, MLflow autolog
train_job.py	Submit training as AML command job
register_model.py	Register best model from experiment run
deploy_endpoint.py	Create endpoint and initial "blue" deployment (100% traffic)
deploy_canary.py	Deploy new model version as "green" (0% traffic)
promote_deployment.py	Shift traffic percentage to new deployment
rollback_deployment.py	Revert all traffic to previous deployment
smoke_test.py	Validate endpoint health (latency, accuracy, schema)
score.py	Client-side scoring example

Azure Resources Consumed by Scenario 1

Operational Lifecycle

The same blue/green deployment pattern from Scenario 1 applies here. When the model is retrained on new labeled .ldr files, a canary deployment is created and validated before traffic is shifted.

Feature engineering is the critical first step. The 4 numeric features (overhang ratio, collision count, height-to-base ratio, layer density) are derived from parsing the LDraw line format. This parsing logic lives in feature_engineering.py and must handle the full range of LDraw part types.

Azure Resources Consumed by Scenario 2

Component-Based Pipeline

Unlike Scenarios 1 and 2 (which use single command jobs), Scenario 3 uses the @dsl.pipeline decorator to define a multi-step pipeline. Each step is a registered component with typed inputs and outputs. This means Step 1's output (the features CSV) is automatically passed to Step 2 as input.

The pipeline is submitted via pipeline_job.py, which defines the step graph and submits it to AML. Both steps run on the same compute cluster but could be configured to use different compute targets if needed.

Because this is a batch analysis (no real-time scoring), there is no managed endpoint. Results are stored as MLflow artifacts and the clustering model is registered for later use (for example, to classify new reference models into discovered pattern groups).

Azure Resources Consumed by Scenario 3

Resource	Created By	Purpose
Blob container reference-models	Bicep	Reference .ldr file storage
Datastore reference_models	register_data_assets.py	ML workspace pointer to blob
Data asset reference-models:1	extract_stats.py	Input data (uri_folder)
Environment claudebricks-clustering:1	register_envs.py	Training runtime (sklearn, hdbscan)
CPU Cluster	Bicep	Pipeline compute
Pipeline job scenario3-pattern-extraction	pipeline_job.py	2-step extract + cluster pipeline
MLflow experiment	Auto-created	Metric and artifact tracking
Registered model clustering-model:1	train.py	Cluster labels + centroids

This scenario generates LEGO building specifications from natural language descriptions. It combines retrieval-augmented generation (RAG) with optional fine-tuning. There are two implementation paths: the original Azure OpenAI + Prompt Flow approach, and a Foundry-native path that aligns with AI-300 GenAIOps objectives.

Resources Consumed

Resource	Purpose
Azure OpenAI	GPT-4o for generation, text-embedding-3-small for vectorization
AI Search	Vector + hybrid search index for RAG retrieval
Foundry Hub + Project	GenAI operations, evaluation, monitoring (Path B)
Storage Account	Training data, inference logs
App Insights	Telemetry, tracing, latency metrics

Managed online endpoints serve trained models for real-time inference. Claude.Bricks uses blue/green deployment for safe rollout. The pattern applies to any scenario, but Scenario 1 (facade classification) is the primary demonstration.

Blue/Green Architecture

Deployment Flow

Smoke Test Criteria

Rollback triggers: latency spike above 2x baseline, error rate exceeds 1%, accuracy regression on validation set, or manual operator decision. See runbooks/endpoint-deployment.md for the full operational playbook.

Every change to prompts, evaluation datasets, or Scenario 4 code triggers an automated evaluation run through the eval-rag.yml GitHub Actions workflow. Deployment is blocked if quality metrics fall below defined thresholds.

Quality Metrics and Thresholds

Minimum scores required to pass the evaluation gate. Deployment is blocked if any metric falls below its threshold.

Evaluation Scripts

Script	Purpose
eval/run_evaluation.py	Main evaluation runner. Scores all test cases against deployed model.
eval/run_prompt_experiment.py	A/B testing across prompt variant combinations (system x RAG matrix).
eval/run_rag_tuning.py	RAG configuration optimization (chunk size, overlap, search mode, top-k).

All prompts are versioned in Git under the prompts/ directory. Changes to prompts trigger the evaluation workflow automatically, so regressions are caught before deployment.

Prompt Directory Structure

Experiment Framework

eval/run_prompt_experiment.py creates a matrix of all system prompt x RAG template combinations. For each combination, it runs the full evaluation dataset and measures quality metrics, token usage, latency, and estimated cost. The output identifies the best-performing combination by weighted composite score.

Example: 2 system prompts x 2 RAG templates = 4 combinations, each evaluated against 15 test cases = 60 evaluation runs.