scry/docs/CARD_RECOGNITION.md

Card Recognition Architecture

This document explores approaches for implementing robust MTG card recognition in Scry.

Goals

1. Robustness - Work reliably across varying lighting, angles, and card conditions
2. Speed - Fast enough for real-time scanning (<500ms per card)
3. Accuracy - High precision to avoid misidentifying valuable cards
4. Offline-capable - Core recognition should work without network

Data Sources

Scryfall API

Scryfall is the de-facto source of truth for MTG card data.

Key endpoints:

Endpoint	Purpose
GET /cards/named?fuzzy={name}	Fuzzy name lookup
GET /cards/{scryfall_id}	Get card by ID
GET /cards/search?q={query}	Full-text search
GET /bulk-data	Daily JSON exports

Rate limits: 50-100ms between requests (~10/sec). Images at *.scryfall.io have no rate limit.

Bulk data options:

File	Size	Use Case
Oracle Cards	~161 MB	One card per Oracle ID (recognition)
Unique Artwork	~233 MB	One per unique art (art-based matching)
Default Cards	~501 MB	Every English printing
All Cards	~2.3 GB	Every card, every language

Recommended approach: Download "Unique Artwork" bulk data, extract image URLs and hashes for all cards. Update weekly or after new set releases.

Card Image Fields

{
  "id": "uuid",
  "oracle_id": "uuid",
  "name": "Lightning Bolt",
  "set": "2xm",
  "collector_number": "129",
  "illustration_id": "uuid",
  "image_uris": {
    "small": "https://cards.scryfall.io/.../small/...",
    "normal": "https://cards.scryfall.io/.../normal/...",
    "large": "https://cards.scryfall.io/.../large/...",
    "art_crop": "https://cards.scryfall.io/.../art_crop/..."
  }
}

Key identifiers:

• id - Unique per printing
• oracle_id - Same across reprints (same card conceptually)
• illustration_id - Same across reprints with identical artwork

---

Recognition Approaches

1. Perceptual Hashing (Recommended Primary)

How it works: Convert image to fixed-size fingerprint resistant to minor transformations.

Algorithm:

1. Resize image to small size (e.g., 32x32)
2. Convert to grayscale (or keep RGB for color-aware variant)
3. Apply DCT (Discrete Cosine Transform)
4. Keep low-frequency components
5. Compute hash from median comparison

Variants:

Type	Description	Use Case
aHash	Average hash	Fast, less accurate
pHash	Perceptual hash	Good balance
dHash	Difference hash	Edge-focused
wHash	Wavelet hash	Most robust
Color pHash	Separate RGB channel hashes	Best for colorful art

Performance (from MTG Card Detector project):

• Hash size 16 (256-bit with RGB): ~16ms per comparison
• Hash size 64: ~65ms per comparison
• Database of 30k+ cards: still feasible with proper indexing

Implementation:

// Pseudo-code for color-aware pHash
public byte[] ComputeColorHash(Image image)
{
    var resized = Resize(image, 32, 32);
    var rHash = ComputePHash(resized.RedChannel);
    var gHash = ComputePHash(resized.GreenChannel);
    var bHash = ComputePHash(resized.BlueChannel);
    return Concat(rHash, gHash, bHash); // 768-bit hash
}

public int HammingDistance(byte[] a, byte[] b)
{
    int distance = 0;
    for (int i = 0; i < a.Length; i++)
        distance += PopCount(a[i] ^ b[i]);
    return distance;
}

Matching strategy:

confidence = (mean_distance - best_match_distance) / (4 * std_deviation)

Accept match if best match is >4 standard deviations better than average.

2. OCR-Based Recognition (Fallback)

When to use: Stacked/overlapping cards where only name is visible.

Approach:

1. Detect text regions in image
2. Run OCR on card name area
3. Fuzzy match against card database using SymSpell (edit distance ≤6)

Libraries:

• Azure Computer Vision / Google Cloud Vision (best accuracy)
• Tesseract (open source, but poor on stylized MTG fonts)
• ML Kit (on-device, good for mobile)

Accuracy: ~90% on test sets with cloud OCR.

3. Art-Only Matching

When to use: Cards with same name but different art (reprints).

Approach:

1. Detect card boundaries
2. Crop to art box only (known position relative to card frame)
3. Compute hash of art region
4. Match against art-specific hash database

Benefits:

• More robust to frame changes between editions
• Smaller hash database (unique artwork only)
• Less affected by card condition (art usually best preserved)

4. Neural Network (Future Enhancement)

Potential approaches:

Method	Pros	Cons
YOLO detection	Finds cards in complex scenes	Slow (~50-60ms/frame)
CNN classification	High accuracy	Needs training per card
CNN embeddings	Similarity search	Requires pre-trained model
Siamese networks	Few-shot learning	Complex training

Recommendation: Start with pHash, add neural detection for card localization only if contour detection proves insufficient.

---

Robustness Strategies

Pre-processing Pipeline

Input Image
    │
    ▼
┌─────────────────┐
│ Resize (max 1000px) │
└─────────────────┘
    │
    ▼
┌─────────────────┐
│ CLAHE Normalization │  ← Fixes uneven lighting
│ (LAB color space)   │
└─────────────────┘
    │
    ▼
┌─────────────────┐
│ Card Detection      │  ← Contour or ML-based
│ (find boundaries)   │
└─────────────────┘
    │
    ▼
┌─────────────────┐
│ Perspective Warp    │  ← Normalize to rectangle
└─────────────────┘
    │
    ▼
┌─────────────────┐
│ Hash Computation    │
└─────────────────┘
    │
    ▼
┌─────────────────┐
│ Database Matching   │
└─────────────────┘

CLAHE (Contrast Limited Adaptive Histogram Equalization)

Critical for handling varying lighting:

// Convert to LAB, apply CLAHE to L channel, convert back
var lab = ConvertToLab(image);
lab.L = ApplyCLAHE(lab.L, clipLimit: 2.0, tileSize: 8);
var normalized = ConvertToRgb(lab);

Multi-Threshold Card Detection

Use multiple thresholding approaches in parallel:

1. Adaptive threshold on grayscale
2. Separate thresholds on R, G, B channels
3. Canny edge detection

Combine results to find card contours that appear in multiple methods.

Confidence Scoring

public class MatchResult
{
    public Card Card { get; set; }
    public float Confidence { get; set; }
    public int HashDistance { get; set; }
    public MatchMethod Method { get; set; }
}

public MatchResult Match(byte[] queryHash, CardDatabase db)
{
    var distances = db.Cards
        .Select(c => (Card: c, Distance: HammingDistance(queryHash, c.Hash)))
        .OrderBy(x => x.Distance)
        .ToList();
    
    var best = distances[0];
    var mean = distances.Average(x => x.Distance);
    var stdDev = StandardDeviation(distances.Select(x => x.Distance));
    
    // Z-score: how many std devs better than mean
    var zScore = (mean - best.Distance) / stdDev;
    
    return new MatchResult
    {
        Card = best.Card,
        Confidence = Math.Min(zScore / 4f, 1f), // Normalize to 0-1
        HashDistance = best.Distance
    };
}

Edge Cases

Scenario	Strategy
Foil cards	Pre-process to reduce glare; may need separate foil hash DB
Worn/played	Lower confidence threshold, flag for manual review
Foreign language	Match by art hash (language-independent)
Tokens/emblems	Include in database with separate type flag
Partial visibility	Fall back to OCR on visible portion
Similar cards	Color-aware hashing helps; art-only match as tiebreaker

---

Recommended Architecture

Phase 1: MVP (pHash + Scryfall)

┌─────────────────────────────────────────────────────┐
│                    Scry App                          │
├─────────────────────────────────────────────────────┤
│  ┌─────────────┐    ┌──────────────────┐            │
│  │ CameraView  │───▶│ CardRecognition  │            │
│  └─────────────┘    │ Service          │            │
│                     ├──────────────────┤            │
│                     │ • PreProcess()   │            │
│                     │ • DetectCard()   │            │
│                     │ • ComputeHash()  │            │
│                     │ • MatchCard()    │            │
│                     └────────┬─────────┘            │
│                              │                      │
│                     ┌────────▼─────────┐            │
│                     │ CardHashDatabase │            │
│                     │ (SQLite)         │            │
│                     └────────┬─────────┘            │
│                              │                      │
└──────────────────────────────┼──────────────────────┘
                               │ Weekly sync
                     ┌─────────▼─────────┐
                     │  Scryfall Bulk    │
                     │  Data API         │
                     └───────────────────┘

Components

1. CardHashDatabase - SQLite with pre-computed hashes for all cards
2. ImagePreprocessor - CLAHE, resize, normalize
3. CardDetector - Contour detection, perspective correction
4. HashComputer - Color-aware pHash implementation
5. CardMatcher - Hamming distance search with confidence scoring
6. ScryfallSyncService - Downloads bulk data, computes hashes, updates DB

Database Schema

CREATE TABLE cards (
    id TEXT PRIMARY KEY,          -- Scryfall ID
    oracle_id TEXT NOT NULL,
    name TEXT NOT NULL,
    set_code TEXT NOT NULL,
    collector_number TEXT,
    illustration_id TEXT,
    image_url TEXT,
    art_hash BLOB,                -- 256-bit hash of art region
    full_hash BLOB,               -- 256-bit hash of full card
    color_hash BLOB               -- 768-bit color-aware hash
);

CREATE INDEX idx_cards_oracle ON cards(oracle_id);
CREATE INDEX idx_cards_illustration ON cards(illustration_id);

Phase 2: Enhanced (Add OCR Fallback)

Add ML Kit or Tesseract for OCR when hash matching confidence is low.

Phase 3: Advanced (Neural Detection)

Replace contour-based card detection with YOLO or similar for complex scenes (multiple overlapping cards, cluttered backgrounds).

---

Libraries & Tools

.NET/MAUI Compatible

Library	Purpose	Platform
SkiaSharp	Image processing	All
OpenCvSharp4	Advanced CV	Android/iOS/Windows
ImageSharp	Image manipulation	All
Emgu.CV	OpenCV wrapper	All
ML.NET	Machine learning	All
Plugin.Maui.OCR	On-device OCR	Android/iOS

Recommended Stack

<PackageReference Include="SkiaSharp" Version="2.88.7" />
<PackageReference Include="SkiaSharp.Views.Maui.Controls" Version="2.88.7" />
<PackageReference Include="ImageHash" Version="3.1.0" />  <!-- If porting from Python -->
<PackageReference Include="Microsoft.Data.Sqlite" Version="9.0.0" />

For perceptual hashing in C#, we'll need to implement it using SkiaSharp (no direct port of Python's imagehash exists).

---

Test Image Categories

The TestImages/ directory contains reference images for testing:

TestImages/
├── varying_quality/     # Different lighting, blur, exposure
│   ├── black.jpg
│   ├── counterspell_bgs.jpg
│   ├── dragon_whelp.jpg
│   ├── evil_eye.jpg
│   ├── instill_energy.jpg
│   ├── ruby.jpg
│   ├── card_in_plastic_case.jpg
│   ├── test1.jpg
│   ├── test2.jpg
│   └── test3.jpg
├── hands/               # Cards held in hand (partial visibility)
│   ├── hand_of_card_1.png
│   ├── hand_of_card_green_1.jpg
│   ├── hand_of_card_green_2.jpeg
│   ├── hand_of_card_ktk.png
│   ├── hand_of_card_red.jpeg
│   └── hand_of_card_tron.png
├── angled/              # Perspective distortion
│   ├── tilted_card_1.jpg
│   └── tilted_card_2.jpg
└── multiple_cards/      # Multiple cards in frame
    ├── alpha_deck.jpg
    ├── geyser_twister_fireball.jpg
    ├── lands_and_fatties.jpg
    ├── pro_tour_table.png
    └── pro_tour_side.png

Test Scenarios to Add

• Foil cards with glare
• Heavily played/worn cards
• Cards under glass/sleeve
• Low-light conditions
• Overexposed images
• Cards with shadows across them
• Non-English cards
• Tokens and emblems
• Old frame vs new frame cards

---

References

• Scryfall API Docs
• MTG Card Detector (Python)
• Magic Card Detector Blog
• mtgscan (OCR approach)
• Moss Machine (pHash + sorting)