vibe coded the shit out of a db migration system
This commit is contained in:
264
AGENTS.md
264
AGENTS.md
@@ -53,14 +53,270 @@ go test -coverprofile=coverage.out ./...
|
||||
go tool cover -html=coverage.out
|
||||
```
|
||||
|
||||
### Database
|
||||
### Database Migrations
|
||||
|
||||
**oslstats uses Bun's migration framework for safe, incremental schema changes.**
|
||||
|
||||
#### Quick Reference
|
||||
|
||||
```bash
|
||||
# Run migrations
|
||||
# Show migration status
|
||||
make migrate-status
|
||||
|
||||
# Preview pending migrations (dry-run)
|
||||
make migrate-dry-run
|
||||
|
||||
# Run pending migrations (with automatic backup)
|
||||
make migrate
|
||||
# OR
|
||||
./bin/oslstats --migrate
|
||||
|
||||
# Rollback last migration group
|
||||
make migrate-rollback
|
||||
|
||||
# Create new migration
|
||||
make migrate-create NAME=add_email_to_users
|
||||
|
||||
# Dev: Run migrations without backup (faster)
|
||||
make migrate-no-backup
|
||||
|
||||
# Dev: Reset database (DESTRUCTIVE - deletes all data)
|
||||
make reset-db
|
||||
```
|
||||
|
||||
#### Creating a New Migration
|
||||
|
||||
**Example: Adding an email field to users table**
|
||||
|
||||
1. **Generate migration file:**
|
||||
```bash
|
||||
make migrate-create NAME=add_email_to_users
|
||||
```
|
||||
Creates: `cmd/oslstats/migrations/20250124150030_add_email_to_users.go`
|
||||
|
||||
2. **Edit the migration file:**
|
||||
```go
|
||||
package migrations
|
||||
|
||||
import (
|
||||
"context"
|
||||
"github.com/uptrace/bun"
|
||||
)
|
||||
|
||||
func init() {
|
||||
Migrations.MustRegister(
|
||||
// UP: Add email column
|
||||
func(ctx context.Context, db *bun.DB) error {
|
||||
_, err := db.ExecContext(ctx,
|
||||
"ALTER TABLE users ADD COLUMN email VARCHAR(255)")
|
||||
return err
|
||||
},
|
||||
// DOWN: Remove email column (for rollback)
|
||||
func(ctx context.Context, db *bun.DB) error {
|
||||
_, err := db.ExecContext(ctx,
|
||||
"ALTER TABLE users DROP COLUMN email")
|
||||
return err
|
||||
},
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
3. **Update the model** (`internal/db/user.go`):
|
||||
```go
|
||||
type User struct {
|
||||
bun.BaseModel `bun:"table:users,alias:u"`
|
||||
|
||||
ID int `bun:"id,pk,autoincrement"`
|
||||
Username string `bun:"username,unique"`
|
||||
Email string `bun:"email"` // NEW FIELD
|
||||
CreatedAt int64 `bun:"created_at"`
|
||||
DiscordID string `bun:"discord_id,unique"`
|
||||
}
|
||||
```
|
||||
|
||||
4. **Preview the migration (optional):**
|
||||
```bash
|
||||
make migrate-dry-run
|
||||
```
|
||||
Output:
|
||||
```
|
||||
[INFO] Pending migrations (dry-run):
|
||||
📋 20250124150030_add_email_to_users
|
||||
[INFO] Would migrate to group 2
|
||||
```
|
||||
|
||||
5. **Apply the migration:**
|
||||
```bash
|
||||
make migrate
|
||||
```
|
||||
Output:
|
||||
```
|
||||
[INFO] Step 1/5: Validating migrations...
|
||||
[INFO] Migration validation passed ✓
|
||||
[INFO] Step 2/5: Checking for pending migrations...
|
||||
[INFO] Step 3/5: Creating backup...
|
||||
[INFO] Backup created: backups/20250124_150145_pre_migration.sql.gz (2.3 MB)
|
||||
[INFO] Step 4/5: Acquiring migration lock...
|
||||
[INFO] Migration lock acquired
|
||||
[INFO] Step 5/5: Applying migrations...
|
||||
[INFO] Migrated to group 2
|
||||
✅ 20250124150030_add_email_to_users
|
||||
[INFO] Migration lock released
|
||||
```
|
||||
|
||||
#### Migration Patterns
|
||||
|
||||
**Create Table:**
|
||||
```go
|
||||
// UP
|
||||
_, err := db.NewCreateTable().
|
||||
Model((*db.Game)(nil)).
|
||||
Exec(ctx)
|
||||
return err
|
||||
|
||||
// DOWN
|
||||
_, err := db.NewDropTable().
|
||||
Model((*db.Game)(nil)).
|
||||
IfExists().
|
||||
Exec(ctx)
|
||||
return err
|
||||
```
|
||||
|
||||
**Add Column:**
|
||||
```go
|
||||
// UP
|
||||
_, err := db.ExecContext(ctx,
|
||||
"ALTER TABLE users ADD COLUMN email VARCHAR(255)")
|
||||
return err
|
||||
|
||||
// DOWN
|
||||
_, err := db.ExecContext(ctx,
|
||||
"ALTER TABLE users DROP COLUMN email")
|
||||
return err
|
||||
```
|
||||
|
||||
**Create Index:**
|
||||
```go
|
||||
// UP
|
||||
_, err := db.NewCreateIndex().
|
||||
Model((*db.User)(nil)).
|
||||
Index("idx_username").
|
||||
Column("username").
|
||||
Exec(ctx)
|
||||
return err
|
||||
|
||||
// DOWN
|
||||
_, err := db.ExecContext(ctx,
|
||||
"DROP INDEX IF EXISTS idx_username")
|
||||
return err
|
||||
```
|
||||
|
||||
**Data Migration:**
|
||||
```go
|
||||
// UP
|
||||
_, err := db.NewUpdate().
|
||||
Model((*db.User)(nil)).
|
||||
Set("status = ?", "active").
|
||||
Where("status IS NULL").
|
||||
Exec(ctx)
|
||||
return err
|
||||
|
||||
// DOWN (if reversible)
|
||||
_, err := db.NewUpdate().
|
||||
Model((*db.User)(nil)).
|
||||
Set("status = NULL").
|
||||
Where("status = ?", "active").
|
||||
Exec(ctx)
|
||||
return err
|
||||
```
|
||||
|
||||
#### Safety Features
|
||||
|
||||
✅ **Automatic Backups**
|
||||
- Compressed with gzip (saves ~80% space)
|
||||
- Created before every migration and rollback
|
||||
- Stored in `backups/` directory
|
||||
- Retention policy keeps last 10 backups
|
||||
- Graceful fallback if `pg_dump` not installed
|
||||
|
||||
✅ **Migration Locking**
|
||||
- PostgreSQL advisory locks prevent concurrent migrations
|
||||
- 5-minute timeout prevents hung locks
|
||||
- Safe for multi-instance deployments
|
||||
|
||||
✅ **Validation**
|
||||
- Pre-migration `go build` check ensures migrations compile
|
||||
- Dry-run mode previews changes without applying
|
||||
- Status command shows exactly what's applied
|
||||
|
||||
✅ **Rollback Support**
|
||||
- Every migration must have a DOWN function
|
||||
- Rollbacks are grouped (undo last batch together)
|
||||
- Automatic backup before rollback
|
||||
|
||||
#### Environment Variables
|
||||
|
||||
```bash
|
||||
# Backup directory (default: backups)
|
||||
DB_BACKUP_DIR=backups
|
||||
|
||||
# Number of backups to keep (default: 10)
|
||||
DB_BACKUP_RETENTION=10
|
||||
```
|
||||
|
||||
#### Troubleshooting
|
||||
|
||||
**"pg_dump not found"**
|
||||
- Migrations will still run, but backups will be skipped
|
||||
- Install PostgreSQL client tools for backups:
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get install postgresql-client
|
||||
|
||||
# macOS
|
||||
brew install postgresql
|
||||
|
||||
# Arch
|
||||
sudo pacman -S postgresql-libs
|
||||
```
|
||||
|
||||
**"migration already in progress"**
|
||||
- Another instance is running migrations
|
||||
- Wait for it to complete (max 5 minutes)
|
||||
- If stuck, check for hung database connections
|
||||
|
||||
**"migration build failed"**
|
||||
- Migration file has syntax errors
|
||||
- Fix the errors and try again
|
||||
- Use `go build ./cmd/oslstats/migrations` to debug
|
||||
|
||||
#### Best Practices
|
||||
|
||||
1. **Always test in development first**
|
||||
- Use `make migrate-dry-run` to preview
|
||||
- Test rollback: `make migrate-rollback`
|
||||
- Verify with `make migrate-status`
|
||||
|
||||
2. **Write reversible migrations**
|
||||
- DOWN function should undo UP changes
|
||||
- Test rollbacks work correctly
|
||||
- Some operations (data deletion) may not be reversible
|
||||
|
||||
3. **Keep migrations focused**
|
||||
- One migration = one logical change
|
||||
- Don't mix schema changes with data migrations
|
||||
- Use descriptive names
|
||||
|
||||
4. **Production checklist**
|
||||
- ✅ Tested in development
|
||||
- ✅ Tested rollback
|
||||
- ✅ Verified backup works
|
||||
- ✅ Communicated downtime (if any)
|
||||
- ✅ Have rollback plan ready
|
||||
|
||||
5. **Development workflow**
|
||||
- Use `--no-backup` for speed: `make migrate-no-backup`
|
||||
- Use `--reset-db` to start fresh (loses data!)
|
||||
- Commit migrations to version control
|
||||
|
||||
### Configuration Management
|
||||
```bash
|
||||
# Generate .env template file
|
||||
|
||||
Reference in New Issue
Block a user