pizofreude/git.md

Here’s a comprehensive Git Gist tailored for both Fellow-level BI & Analytics Engineers and Fellow-level Fullstack Software Engineers. This Gist covers advanced workflows, best practices, and tips for collaboration, scalability, and maintainability—key for senior engineers in both domains.

Git Gist: Advanced Workflows & Best Practices

For Fellow-Level BI & Analytics Engineers & Fullstack Software Engineers

1. Core Git Concepts (Refresher)

Branching Models: Use git flow or GitHub Flow for structured releases.
Rebase vs. Merge: Prefer rebase for clean history, merge for public branches.
Stashing: git stash for temporary changes without committing.

2. Advanced Collaboration

Pull Requests (PRs):
- Template: Enforce PR templates with context, impact, and testing notes.
- Review: Require 2+ approvals for critical changes.
- CI/CD: Integrate automated tests (e.g., GitHub Actions, GitLab CI).
Squashing: Use git merge --squash to consolidate commits before merging.

3. Scalability & Performance

Shallow Clones: git clone --depth 1 for large repos.
Submodules: Manage dependencies with git submodule.
Large Files: Use git lfs for datasets, binaries, or models.

4. BI/Analytics-Specific Tips

Data Versioning: Track datasets with DVC (Data Version Control) alongside Git.
Notebooks: Use nbdime for Jupyter notebook diffs.
SQL Scripts: Version-control SQL in /sql/ with clear naming conventions.

5. Fullstack-Specific Tips

Monorepos: Use tools like lerna or nx for multi-package projects.
Environment Parity: Use .gitignore to exclude local configs (e.g., .env).
Docker Integration: Commit Dockerfiles and docker-compose.yml for reproducibility.

6. Security & Compliance

Secrets: Never commit API keys; use git-secrets or .gitignore.
Signed Commits: Enforce GPG-signed commits for critical projects.
Audit Logs: Regularly audit git log for anomalies.

7. Debugging & Recovery

Bisect: git bisect to find regression commits.
Reflog: git reflog to recover lost commits/branches.
Blame: git blame to trace changes (use sparingly in collaborative settings).

8. Automation & Tooling

Hooks: Use pre-commit hooks for linting (e.g., black, flake8).
Bots: Automate PR labeling/merging with tools like Renovate.
CLI Aliases: Customize Git with aliases (e.g., git config --global alias.lg "log --graph").

9. Git Stash & Merge Workflow Gist

When working with untracked local files in a feature branch and preparing to merge:

Stash local changes (including untracked files):
```
git stash -u -m "local test files"
```

Switch to main and merge:

git checkout main
git merge feature-branch

Create new branch from main and apply stashed files:
```
git checkout -b new-feature-branch
git stash apply
```

✅ Only stashed changes meant for reuse are applied — keep main clean of local-only files.

10. Documentation

README.md: Include setup, architecture, and contribution guidelines.
CHANGELOG.md: Track releases and breaking changes.
GitHub Wiki: For project-specific workflows.

11. Culture & Leadership

Mentorship: Document Git best practices for junior engineers.
Postmortems: Use Git history to analyze incidents.
Open Source: Contribute back to tools like dbt, Airflow, or React.

Example Workflow for a BI Project

# Feature branch
git checkout -b feature/analytics-dashboard

# Commit with context
git commit -m "feat: add revenue KPI dashboard
- Uses Redshift data
- Tests in Looker"

# Push and open PR
git push origin feature/analytics-dashboard
gh pr create --fill

Example Workflow for a Fullstack Project

# Rebase on main
git fetch origin
git rebase origin/main

# Squash commits
git reset --soft HEAD~3
git commit -m "feat: implement OAuth2 flow"

# Tag release
git tag -a v1.2.0 -m "Release v1.2.0"
git push origin v1.2.0

Key Tools

Tool	Use Case
`DVC`	Data versioning
`GitHub Actions`	CI/CD pipelines
`pre-commit`	Code quality hooks
`Renovate`	Dependency updates

Final Tips

Consistency: Agree on conventions (e.g., commit messages, branch names).
Transparency: Document decisions in PRs/issues.
Adaptability: Evolve workflows as the team scales.

Here’s a tailored, stack-specific Git Gist for Fellow-level BI & Analytics Engineers (AWS + dbt) and Fullstack Software Engineers, with a focus on cross-functional collaboration and scalability for teams of 10+ engineers.

Git Gist: Advanced Workflows for AWS + dbt (BI) & Fullstack (React/Node)

For Teams of 10+ Engineers

1. BI/Analytics: AWS + dbt Workflow

Branching:

Use main (production), staging, and feature branches (e.g., feature/add-snowflake-source).
Enforce linear history with rebase (no merge commits).

dbt Integration:

Version-control dbt models in /models/ with clear prefixes:

/models/
  ├── staging/
  ├── marts/
  └── seeds/

Use dbt docs generate and commit target/catalog.json for documentation.

AWS-Specific:

Terraform/IaC: Store in /infrastructure/; use terragrunt for DRY config.
Glue/EMR Scripts: Version-control in /scripts/ with .gitignore for *.log.
Data Pipelines: Track Airflow DAGs in /dags/ with git lfs for large dependencies.

Example:

# dbt workflow
git checkout -b feature/add-customer-mart
dbt run --models +customer_mart
git add models/marts/customer/
git commit -m "feat: add customer lifetime value model"
git push origin feature/add-customer-mart

2. Fullstack: React/Node + Docker

Branching:

main (prod), develop (staging), and feature branches (e.g., feature/auth0-integration).
Use semantic commits (e.g., fix:, feat:, chore:).

Monorepo Tips:

Use lerna or nx for shared libraries (e.g., UI components, utilities).

Example structure:

/packages/
  ├── frontend/ (React)
  ├── backend/  (Node)
  └── shared/   (TypeScript libs)

Docker:

Commit Dockerfile and docker-compose.yml; ignore node_modules/ and .env.
Use docker build --tag myapp:v1.2.0 with Git tags.

Example:

# Fullstack feature
git checkout -b feature/add-payment-gateway
cd packages/backend && npm install stripe
git add packages/backend/package.json
git commit -m "feat: add Stripe integration"

3. Cross-Functional Collaboration

Shared Ownership:

BI + DevOps: Use CODEOWNERS to auto-assign PRs for infrastructure changes.
Frontend + BI: Document API contracts in /docs/api/ (OpenAPI/Swagger).

PR Templates:

## [BI/Fullstack] Description
- **Jira Ticket**: [LINK]
- **Impact**: [Metrics/Dashboards/APIs affected]
- **Testing**: [dbt tests/Postman collection]
- **Screenshots**: [For UI changes]

CI/CD:

BI: Run dbt test in GitHub Actions on PR.
Fullstack: Run npm test + docker build in parallel.

4. Scaling for 10+ Engineers

Code Reviews:

Rotate reviewers weekly; use pair programming for complex changes.
Require video walkthroughs for major features.

Documentation:

Architecture Decisions: Track in /docs/adr/ (e.g., 001-use-dbt-cloud.md).
Onboarding: Maintain a CONTRIBUTING.md with setup guides.

Tools:

Tool	BI Use Case	Fullstack Use Case
`DVC`	Version Snowflake data	N/A
`Sentry`	Monitor dbt runs	Frontend error tracking
`Datadog`	Pipeline observability	Backend APM

5. Disaster Recovery

BI: Backup dbt manifest.json and catalog.json in S3.
Fullstack: Use git archive for point-in-time restores.

Example: Cross-Team PR

Title: feat: add user activity dashboard Description:

BI: New dbt model (user_activity.sql) + Looker dashboard.
Fullstack: New API endpoint (/api/activity) + React component.
Testing: dbt tests + Jest/RTL.

Here’s your expanded Git Gist, now including detailed merge conflict resolution for dbt and React, plus GitHub Actions workflows for both BI (AWS + dbt) and Fullstack (React/Node) stacks.

Git Gist: Advanced Workflows for AWS + dbt & React/Node

For Fellow-Level Engineers (Teams 10+)

6. Handling Merge Conflicts

For dbt Models

Merge conflicts in dbt often occur in:

SQL files (e.g., models/marts/customer_dimension.sql)
YAML files (e.g., dbt_project.yml, schema.yml)

How to Resolve:

Pull Latest Changes:

git fetch origin
git rebase origin/develop  # or main

Identify Conflicts:
- Use git status to find conflicted files.
- Open the file; conflicts are marked with <<<<<<<, =======, >>>>>>>.

Resolve SQL Conflicts:

Example:

<<<<<<< HEAD
select user_id, sum(revenue) as lifetime_value
from {{ ref('stg_orders') }}
=======
select user_id, sum(amount) as lifetime_value
from {{ ref('stg_payments') }}
>>>>>>> feature/new-metrics

Fix: Decide which logic to keep (or combine both):

select user_id, sum(revenue) as lifetime_value
from {{ ref('stg_orders') }}
-- Added payments data for accuracy
union all
select user_id, sum(amount) as lifetime_value
from {{ ref('stg_payments') }}

Resolve YAML Conflicts:

Example (dbt_project.yml):

models:
  marts:
    +schema: analytics
<<<<<<< HEAD
    +materialized: table
=======
    +materialized: incremental
>>>>>>> feature/incremental-models

Fix: Choose the correct materialization strategy.

Mark as Resolved:

git add models/marts/customer_dimension.sql
git rebase --continue

For React/Node Projects

Merge conflicts in React/Node often occur in:

Component files (e.g., Button.jsx)
Package files (e.g., package.json)
Configuration files (e.g., webpack.config.js)

How to Resolve:

Pull Latest Changes:

git fetch origin
git rebase origin/develop

Resolve Component Conflicts:

Example (Button.jsx):

<<<<<<< HEAD
const Button = ({ onClick, label }) => (
  <button className="btn-primary" onClick={onClick}>
    {label}
  </button>
);
=======
const Button = ({ onClick, children }) => (
  <button className="btn-secondary" onClick={onClick}>
    {children}
  </button>
);
>>>>>>> feature/new-button-styles

Fix: Combine props and styles:

const Button = ({ onClick, label, children }) => (
  <button className="btn-primary btn-secondary" onClick={onClick}>
    {label || children}
  </button>
);

Resolve package.json Conflicts:
- Use npm install after resolving to update package-lock.json.

Mark as Resolved:

git add src/components/Button.jsx
git rebase --continue

7. GitHub Actions Workflows

For BI (AWS + dbt)

Workflow: Run dbt on PR

# .github/workflows/dbt-pr-check.yml
name: dbt PR Check

on:
  pull_request:
    branches: [develop, main]
    paths:
      - 'models/**'
      - 'dbt_project.yml'

jobs:
  dbt-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install dbt
        run: pip install dbt-core dbt-postgres

      - name: Run dbt deps
        run: dbt deps

      - name: Run dbt test
        run: dbt test --select state:modified
        env:
          DBT_PROFILES_DIR: .

Workflow: Deploy dbt to Production

# .github/workflows/dbt-prod-deploy.yml
name: dbt Production Deploy

on:
  push:
    branches: [main]
    paths:
      - 'models/**'
      - 'dbt_project.yml'

jobs:
  dbt-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install dbt
        run: pip install dbt-core dbt-postgres

      - name: Run dbt prod
        run: dbt run --full-refresh
        env:
          DBT_PROFILES_DIR: .
          DBT_ENV_SECRET: ${{ secrets.DBT_PROD_ENV }}

For Fullstack (React/Node)

Workflow: Test and Build on PR

# .github/workflows/frontend-pr-check.yml
name: Frontend PR Check

on:
  pull_request:
    branches: [develop, main]
    paths:
      - 'packages/frontend/**'

jobs:
  test-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Node
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

      - name: Build
        run: npm run build

Workflow: Deploy Backend to AWS

# .github/workflows/backend-prod-deploy.yml
name: Backend Production Deploy

on:
  push:
    branches: [main]
    paths:
      - 'packages/backend/**'

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Node
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

      - name: Deploy to AWS
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ap-southeast-1

      - name: Deploy with Serverless
        run: npx serverless deploy --stage prod

8. Pro Tips for Both Stacks

Use git rerere: Enable "reuse recorded resolution" to auto-resolve recurring conflicts:
```
git config --global rerere.enabled true
```
Automate PR Labels: Use GitHub Actions to label PRs by size (e.g., small, large) or type (e.g., bug, feature).
Branch Protection: Require status checks (CI) and 2+ approvals for main/develop.

Here’s your final, comprehensive Git Gist, now including a troubleshooting guide for CI/CD failures and an onboarding checklist for new engineers.

Git Gist: Advanced Workflows for AWS + dbt & React/Node

For Fellow-Level Engineers (Teams 10+)

9. Troubleshooting CI/CD Failures

For BI (dbt + GitHub Actions)

Common Issues & Fixes

Issue	Root Cause	Solution
`dbt test` fails on PR	Missing or incorrect test data	Update `data/tests/` or seed files; run locally first.
`dbt run` fails in production	Profile misconfiguration	Check `profiles.yml` and secrets; test with `dbt debug`.
`dbt deps` fails	Missing or incorrect package version	Update `packages.yml`; run `dbt clean` and retry.
GitHub Actions stuck	Runner out of memory	Use a larger runner (e.g., `ubuntu-20.04-8core`).
Permission denied on S3/Redshift	Incorrect IAM roles or secrets	Verify `DBT_ENV_SECRET` and AWS credentials in GitHub Secrets.

Debugging Steps

Reproduce Locally:
```
dbt deps
dbt run --models +your_model
```
Check Logs:
- Look for ERROR or FAIL in GitHub Actions logs.
Validate Secrets:
- Ensure DBT_PROFILES_DIR and environment variables are set.

For Fullstack (React/Node + GitHub Actions)

Common Issues & Fixes

Issue	Root Cause	Solution
`npm install` fails	Corrupted `package-lock.json`	Delete `node_modules/` and `package-lock.json`; run `npm install`.
Tests fail on PR	Missing test data or environment vars	Mock API calls; check `.env.test`.
Build fails	Webpack/config issue	Run `npm run build` locally; check `webpack.config.js`.
AWS deployment fails	Incorrect IAM permissions	Verify `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in GitHub Secrets.
Serverless deploy hangs	Timeout or missing dependencies	Increase timeout; check `serverless.yml` for missing plugins.

Debugging Steps

Reproduce Locally:
```
npm ci
npm test
npm run build
```
Check Logs:
- Look for ERROR or stack traces in GitHub Actions.
Validate Secrets:
- Ensure AWS credentials and environment variables are correct.

10. Onboarding Checklist for New Engineers

Day 1: Setup & Access

GitHub Access: Invite to the org/repo; assign to the correct team.
SSH Keys: Add public key to GitHub.
Secrets: Share .env.example and instructions for local setup.
AWS Access: Grant IAM user with least-privilege permissions (if applicable).

Day 2: Local Development

Clone Repo:

git clone git@github.com:your-org/your-repo.git
cd your-repo

Install Dependencies:
- BI: pip install -r requirements.txt
- Fullstack: npm ci
Run Locally:
- BI: dbt run
- Fullstack: npm start

Day 3: Git & Workflow

Branch Strategy: Explain main/develop/feature branches.
Commit Conventions: Share commit message guidelines (e.g., feat:, fix:).
PR Process: Walk through creating a PR, requesting reviews, and merging.
CI/CD: Explain how GitHub Actions works for the project.

Day 4: Project-Specific

BI:
- Review dbt model structure (staging, marts).
- Explain how to add a new model and document in schema.yml.
Fullstack:
- Review React component structure and state management.
- Explain API routes and database schema.

Day 5: Collaboration

Standups: Invite to daily/weekly syncs.
Documentation: Share links to README.md, CONTRIBUTING.md, and docs/.
Mentorship: Assign a buddy for the first two weeks.

11. Pro Tips for Onboarding

Pair Programming: Schedule a session to work through a small feature or bug fix.
Shadowing: Have the new engineer shadow a PR review or incident response.
Feedback Loop: Schedule a 1:1 after the first week to address questions.