Files

T

Frank 19d054096f initial implementation plan

2026-02-27 12:00:39 +01:00

16 KiB

Raw Blame History

Plan: Pluggable Character Import/Export with Microservice Adapters

This plan extends the existing importero package into a full-featured, extensible import/export system using Docker-based adapter microservices. The canonical CharacterImport format becomes the system-wide interchange format (BMRT-Format), and new external formats (starting with Foundry VTT) are handled by isolated adapter services. New master data is automatically flagged as personal items (house rules).

Key Decisions:

Microservice architecture for adapters (Docker containers)
Auto-flag imported master data as personal items
Foundry VTT JSON as first format
Backend-only implementation (no Vue components)
Keep transfero/ untouched (BaMoRT-to-BaMoRT transfers)
Extend importero/ as the adapter orchestration layer

1. Core Infrastructure (Backend)

1.1 Formalize BMRT-Format

Document importero/model.go CharacterImport as the canonical interchange format
Add JSON schema validation using github.com/xeipuuv/gojsonschema
Add BmrtVersion field to CharacterImport (start at "1.0")
Add SourceMetadata struct to track original format, adapter ID, import timestamp
Update existing VTT adapter to populate these fields

1.2 Database Migrations

Add new tables to models/model_character.go:

type ImportHistory struct {
    ID              uint   `gorm:"primaryKey"`
    UserID          uint   `gorm:"not null;index"`
    CharacterID     uint   `gorm:"index"`
    AdapterID       string `gorm:"type:varchar(100);not null"` // "foundry-vtt-v1"
    SourceFormat    string `gorm:"type:varchar(50)"`           // "foundry-vtt"
    SourceFilename  string
    SourceSnapshot  []byte `gorm:"type:MEDIUMBLOB"`            // Original file
    MappingSnapshot []byte `gorm:"type:JSON"`                  // Adapter->BMRT mappings
    ImportedAt      time.Time
    Status          string `gorm:"type:varchar(20)"` // "success", "partial", "failed"
    ErrorLog        string `gorm:"type:TEXT"`
}

type MasterDataImport struct {
    ID              uint `gorm:"primaryKey"`
    ImportHistoryID uint `gorm:"not null;index"`
    ItemType        string `gorm:"type:varchar(20)"` // "skill", "spell", "weapon", "equipment"
    ItemID          uint   `gorm:"not null"`
    ExternalName    string
    MatchType       string `gorm:"type:varchar(20)"` // "exact", "created_personal"
    CreatedAt       time.Time
}

Add to models/database.go MigrateStructure() function Add to models/model_character.go migration function

1.3 Adapter Service Registry

Create importero/registry.go:

type AdapterMetadata struct {
    ID                  string   // "foundry-vtt-v1"
    Name                string   // "Foundry VTT Character"
    Version             string   // "1.0"
    SupportedExtensions []string // [".json"]
    BaseURL             string   // "http://adapter-foundry:8181"
    Capabilities        []string // ["import", "export", "detect"]
}

type AdapterRegistry struct {
    adapters map[string]*AdapterMetadata
    mu       sync.RWMutex
}

func (r *AdapterRegistry) Register(meta AdapterMetadata) error
func (r *AdapterRegistry) Detect(data []byte) (string, float64, error) // Returns adapter ID + confidence
func (r *AdapterRegistry) Import(adapterID string, data []byte) (*CharacterImport, error)
func (r *AdapterRegistry) Export(adapterID string, char *CharacterImport) ([]byte, error)

Load adapters from config on startup (importero/routes.go):

Environment variable IMPORT_ADAPTERS (JSON array of adapter configs)
Ping each adapter's /metadata endpoint to register
Cache metadata in memory

1.4 Format Detection

Create importero/detector.go:

func DetectFormat(data []byte, filename string) (adapterID string, confidence float64, err error) {
    // Call all registered adapters' POST /detect endpoints in parallel
    // Return highest confidence match
    // Fallback to filename extension matching
}

1.5 Validation Framework

Create importero/validator.go:

type ValidationResult struct {
    Valid    bool
    Errors   []ValidationError
    Warnings []ValidationWarning
}

type ValidationRule interface {
    Validate(char *CharacterImport) ValidationResult
}

// Rules:
// - RequiredFieldsRule (name, gameSystem must exist)
// - StatsRangeRule (stats 0-100 for Midgard)
// - ReferentialIntegrityRule (skills reference valid categories)

1.6 Master Data Reconciliation

Enhance importero/importer.go existing CheckSkill(), CheckSpell() functions:

func ReconcileSkill(skill Fertigkeit, importHistoryID uint) (*models.Skill, string, error) {
    // 1. Exact match by (Name + GameSystem) → "exact"
    // 2. Not found → Create with PersonalItem=true → "created_personal"
    // 3. Log to MasterDataImport table
}

Apply to all types: skills, weapon skills, spells, equipment, weapons, containers Set PersonalItem = true for all created master data Chain user's UserID to created items via CreatedByUserID (add field to GSM models)

2. API Endpoints (Backend)

Add to importero/routes.go:

func RegisterRoutes(r *gin.RouterGroup) {
    importer := r.Group("/importer")
    
    // Existing endpoints remain
    
    // NEW endpoints:
    importer.POST("/detect", DetectHandler)           // Upload file, returns detected format
    importer.POST("/import", ImportHandler)           // Upload + import with adapter
    importer.GET("/adapters", ListAdaptersHandler)    // List registered adapters
    importer.GET("/history", ImportHistoryHandler)    // User's import history
    importer.GET("/history/:id", ImportDetailsHandler) // Details + errors
    importer.POST("/export/:id", ExportHandler)        // Export char to original format
}

Handler Implementations in importero/handlers.go:

DetectHandler:

Accept multipart file upload
Save to ./uploads/detect_<uuid>
Call DetectFormat()
Return {adapter_id, confidence, suggested_adapter_name}
Clean up temp file

ImportHandler:

Accept file + optional adapter_id (from detect)
Save original file to ./uploads/import_<uuid>
If no adapter_id, call DetectFormat()
Call registry.Import(adapterID, fileData)
Validate result with validator.Validate()
Create models.Char via existing CreateCharacterFromImport() helper (new function)
Reconcile all master data, log to MasterDataImport
Save original file to ImportHistory.SourceSnapshot
Return {character_id, warnings, created_items: {skills: 3, spells: 1}}

ListAdaptersHandler:

Return registry.GetAll() metadata

ImportHistoryHandler:

Query ImportHistory filtered by userID
Return paginated list

ExportHandler:

Load Char by ID (check ownership)
Load ImportHistory to get original AdapterID
Convert Char back to CharacterImport (reverse of import)
Call registry.Export(adapterID, charImport)
Return file download with Content-Disposition: attachment

3. Adapter Service Protocol

3.1 Adapter HTTP API Contract

All adapter services must implement:

GET /metadata

{
  "id": "foundry-vtt-v1",
  "name": "Foundry VTT Character",
  "version": "1.0",
  "supported_extensions": [".json"],
  "capabilities": ["import", "export", "detect"]
}

POST /detect

Body: raw file bytes
Response: {"confidence": 0.95, "version": "10.x"}

POST /import

Body: raw file bytes
Response: CharacterImport JSON (BMRT-Format)

POST /export

Body: CharacterImport JSON
Response: original format file bytes

3.2 Error Handling

400 Bad Request: malformed input
422 Unprocessable Entity: valid format but conversion failed
500 Internal Server Error: adapter crash

All adapter calls have 30-second timeout Retry logic: 3 attempts with exponential backoff

4. Foundry VTT Adapter Service (First Implementation)

4.1 Docker Service

Create docker/Dockerfile.adapter-foundry:

FROM golang:1.25-alpine AS builder
WORKDIR /app
COPY backend/adapters/foundry/ .
RUN go build -o adapter-foundry .

FROM alpine:latest
COPY --from=builder /app/adapter-foundry /adapter-foundry
EXPOSE 8181
CMD ["/adapter-foundry"]

4.2 Service Code

Create backend/adapters/foundry/main.go:

package main

import (
    "github.com/gin-gonic/gin"
    "bamort/importero"  // Import BMRT-Format types
)

type FoundryCharacter struct {
    Name   string `json:"name"`
    System struct {
        Abilities map[string]struct {
            Value int `json:"value"`
        } `json:"abilities"`
        // ... Foundry schema
    } `json:"system"`
}

func metadata(c *gin.Context) {
    c.JSON(200, gin.H{
        "id": "foundry-vtt-v1",
        "name": "Foundry VTT Character",
        "version": "1.0",
        "supported_extensions": []string{".json"},
        "capabilities": []string{"import", "export", "detect"},
    })
}

func detect(c *gin.Context) {
    data, _ := c.GetRawData()
    // Parse JSON, check for Foundry-specific fields
    // Return confidence 0.0-1.0
}

func importChar(c *gin.Context) {
    var foundry FoundryCharacter
    c.BindJSON(&foundry)
    
    // Convert to importero.CharacterImport
    bmrt := toBMRT(foundry)
    c.JSON(200, bmrt)
}

func exportChar(c *gin.Context) {
    var bmrt importero.CharacterImport
    c.BindJSON(&bmrt)
    
    // Convert back to Foundry format
    foundry := fromBMRT(bmrt)
    c.JSON(200, foundry)
}

4.3 Conversion Logic

Map Foundry abilities → BMRT stats (St, Gw, In...)
Map Foundry items → BMRT equipment
Map Foundry features → BMRT skills
Preserve unmapped fields in CharacterImport.Extensions["foundry"] (add Extensions map to model)

4.4 Docker Compose Integration

Add to docker/docker-compose.dev.yml:

adapter-foundry:
  build:
    context: ../
    dockerfile: docker/Dockerfile.adapter-foundry
  container_name: bamort-adapter-foundry-dev
  ports:
    - "8181:8181"
  networks:
    - bamort-network
  environment:
    - PORT=8181
  restart: unless-stopped

Update backend environment to register adapter:

bamort-backend-dev:
  environment:
    - IMPORT_ADAPTERS=[{"id":"foundry-vtt-v1","base_url":"http://adapter-foundry:8181"}]

5. Testing Strategy

5.1 Unit Tests

Create importero/registry_test.go:

Test adapter registration
Test detection with multiple adapters
Mock HTTP responses using httptest

Create importero/validator_test.go:

Test each validation rule
Test warning vs error distinction

5.2 Integration Tests

Create importero/import_integration_test.go:

Use testutils.SetupTestDB()
Test full import flow with mock adapter
Verify ImportHistory created
Verify personal items flagged
Test character creation

5.3 Adapter Tests

Create backend/adapters/foundry/adapter_test.go:

Golden file tests: testdata/foundry_character.json → BMRT → compare
Round-trip tests: Foundry → BMRT → Foundry (structural equality)
Detection tests with sample files

5.4 End-to-End Tests

Create backend/api/import_e2e_test.go:

Start real adapter service in Docker
Upload Foundry character via API
Verify character created
Verify export produces valid Foundry JSON
Use docker-compose -f docker/docker-compose.test.yml with test services

6. Documentation

6.1 Update Files

backend/PlanNewFeature.md → Mark as "Implemented, see IMPORT_EXPORT_GUIDE.md"
Create backend/IMPORT_EXPORT_GUIDE.md with architecture overview
Create backend/adapters/ADAPTER_DEVELOPMENT.md with adapter creation guide
Update docker/SERVICES_REFERENCE.md with adapter services

6.2 API Documentation

Add OpenAPI/Swagger annotations to handlers (use swaggo/swag) Generate docs with swag init

7. Deployment Considerations

7.1 Production Configuration

Adapter URLs from environment variables
Health checks for adapter services
Graceful degradation if adapter unavailable (return error, don't crash)
Rate limiting on import endpoints (prevent abuse)

7.2 Monitoring

Log all import attempts (success/failure) with logger package
Metrics: imports per adapter, detection accuracy, errors by adapter
Alert on adapter unavailability

7.3 File Cleanup

Cron job to delete old uploads (>30 days)
ImportHistory.SourceSnapshot compressed with gzip
Configurable retention policy

8. Future Extensibility

8.1 Adding New Adapters

Create adapter service in backend/adapters/<format>/
Add Dockerfile
Add to docker-compose.dev.yml
Register in backend env vars
Deploy container
No backend code changes required

8.2 Master Data Approval Workflow (Future)

Add MasterDataPending table
Admin UI in Vue to approve/reject
Change reconciliation to create pending records instead of auto-creating

8.3 Fuzzy Matching (Future)

Add github.com/texttheater/golang-levenshtein for string distance
Configurable threshold (e.g., distance < 3)
Return suggestions to user for manual mapping

Verification

Step-by-Step Testing

Start dev environment: cd docker && ./start-dev.sh
Verify adapter container running: docker ps | grep bamort-adapter-foundry
Check adapter metadata: curl http://localhost:8181/metadata
Run backend tests: cd backend && go test ./importero/... -v
Run adapter tests: go test ./adapters/foundry/... -v
Upload test character: curl -F "file=@testdata/foundry_sample.json" http://localhost:8180/api/importer/import -H "Authorization: Bearer <token>"
Verify character created in database via phpMyAdmin
Check ImportHistory table populated
Export character: curl http://localhost:8180/api/importer/export/1 -H "Authorization: Bearer <token>" -o exported.json
Compare original vs exported (structural equivalence)

Database Verification

SELECT * FROM import_histories ORDER BY imported_at DESC LIMIT 10;
SELECT * FROM master_data_imports WHERE item_type='skill';
SELECT * FROM skills WHERE personal_item = true;

Key Decisions

Microservice vs Monolith: Chose microservices for adapters despite added complexity, enables language-agnostic adapters and crash isolation
Master Data Handling: Auto-flag as personal items (no approval workflow) to avoid blocking imports
Format Priority: Foundry VTT first, enables testing of full architecture before adding more formats
Frontend Scope: Backend-only to establish stable API before UI/UX work
BMRT-Format: Use existing CharacterImport rather than create new structure, reduces refactoring
transfero Separation: Keep untouched, serves different purpose (BaMoRT-to-BaMoRT lossless transfer)

Implementation Phases

Phase 1: Core Infrastructure (Week 1-2)

Database migrations
Adapter registry
Format detection
Validation framework
Master data reconciliation updates

Phase 2: API Endpoints (Week 2-3)

Implement all handlers
Error handling
File management
Testing infrastructure

Phase 3: Foundry Adapter (Week 3-4)

Docker service setup
Conversion logic
Round-trip testing
Integration with backend

Phase 4: Testing & Documentation (Week 4-5)

Comprehensive test suite
Documentation updates
E2E testing
Performance testing

Phase 5: Deployment & Monitoring (Week 5-6)

Production configuration
Monitoring setup
File cleanup jobs
Security hardening

Success Criteria

Foundry VTT characters import successfully
Round-trip export produces valid Foundry JSON
Personal items flagged automatically
ImportHistory tracks all imports
Adapters run in isolated Docker containers
90%+ test coverage on new code
API documentation complete
Adding new adapter requires no backend changes
Zero data loss on import/export cycle
Performance: <5s for typical character import

16 KiB Raw Blame History