Build an Internal Developer Platform with Backstage

Introduction

If you've been following DevOps trends, you've probably heard the buzz around platform engineering. But what started as a Gartner hype cycle prediction is now becoming a business necessity. Organisations worldwide are adopting Internal Developer Platforms (IDPs) to accelerate delivery, improve security, and reduce operational overhead.

According to recent surveys, over 50% of organisations are actively investing in platform engineering initiatives. By 2026, Gartner predicts 80% of organisations will establish dedicated platform engineering teams.

But what exactly is an IDP, and more importantly, how do you build one?

This comprehensive guide will walk you through everything you need to know—from fundamentals to hands-on implementation—to build your first Internal Developer Platform.

Who should read this? DevOps engineers, infrastructure teams, and engineering leaders looking to improve developer productivity and operational efficiency.

What is Platform Engineering?

Platform engineering is a discipline that sits at the intersection of DevOps, infrastructure, and developer experience. It's about building tools and abstractions that enable developers to self-serve infrastructure and deployment needs without deep infrastructure knowledge.

Think of it as building an internal product for your developers. Instead of developers wrestling with Kubernetes YAML, networking policies, and deployment procedures, they interact with a simplified, curated experience—your platform.

The Problems It Solves

Before diving into implementation, let's understand why platform engineering matters:

1. Cognitive Load Reduction

Developers shouldn't need to be Kubernetes experts to deploy applications. When developers spend time learning infrastructure details instead of writing business logic, you lose productivity.

2. Inconsistent Deployments

Without standardisation, each team deploys differently. This leads to security vulnerabilities, compliance violations, operational inconsistencies, and higher troubleshooting costs.

3. Slow Time-to-Deployment

When developers need to file tickets and wait for platform/ops teams, deployment cycles stretch from hours to days.

4. Knowledge Silos

Critical infrastructure knowledge lives in a few experts' heads. When they leave or go on vacation, operations suffer.

5. Cost Inefficiency

Without governance, resource sprawl is inevitable. Teams provision more than they need, leading to wasted cloud spend.

Core Components of an IDP

An effective IDP consists of four interconnected layers:

Layer 1: Developer Experience (Portal & APIs)

Self-service portals
Service templates
API-first design
Golden paths

Layer 2: Platform Orchestration

Service catalog
Policy engine
GitOps integration
Routing logic

Layer 3: Capabilities & Integrations

CI/CD pipeline integration
Observability platform
Security scanners
Cost management

Layer 4: Infrastructure Foundation

Kubernetes clusters
Cloud services
Observability stacks
Supporting infrastructure

Why Now? The Perfect Storm

Three converging trends make now the ideal time to build an IDP:

1. Kubernetes Maturity

Kubernetes abstracts away infrastructure complexity. Building platforms on top of K8s is now practical for teams of all sizes.

2. GitOps Tooling

Tools like ArgoCD and Flux have matured enough to become enterprise standards.

3. Platform Engineering Consolidation

We now have proven patterns and tools:

Backstage: By Spotify, the leading open-source platform portal
Score: CNCF project for standardizing deployment metadata
Crossplane: Open-source infrastructure as code at scale
OPA/Gatekeeper: Policy enforcement across clusters

Building Your First IDP: Step-by-Step

Prerequisites

You'll need:

A Kubernetes cluster (EKS, GKE, AKS, or local Minikube)
Docker installed
Basic kubectl knowledge
Git repository for storing service manifests
~2-3 hours for initial setup
Node.js 16+ (for Backstage)

Step 1: Set Up a Kubernetes Cluster

If you don't have a cluster, create one locally:

curl -LO https://github.com/kubernetes/minikube/releases/latest/download/minikube-darwin-amd64
chmod +x minikube-darwin-amd64
sudo mv minikube-darwin-amd64 /usr/local/bin/minikube

minikube start --cpus=4 --memory=8192 --disk-size=50g

kubectl cluster-info
kubectl get nodes

Step 2: Install ArgoCD

kubectl create namespace argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

kubectl wait --for=condition=available --timeout=300s deployment/argocd-server -n argocd

ARGOCD_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)

echo "ArgoCD Admin Password: $ARGOCD_PASSWORD"

kubectl port-forward svc/argocd-server -n argocd 8080:443 &

Access ArgoCD https://localhost:8080 with the username admin and the password from above.

Step 3: Create a Git Repository for Your Platform

mkdir -p idp-demo/{apps,services,platform}
cd idp-demo

git init
git config user.email "platform@example.com"
git config user.name "Platform Team"
git branch -M main

cat > README.md << 'EOF'
# Internal Developer Platform

This repository contains all service definitions and platform configurations.

## Directory Structure

├── services/          # Service deployment manifests
├── apps/              # Application configurations
└── platform/          # Platform infrastructure

## Quick Start

1. Browse available services in Backstage
2. Create a new service from templates
3. Service is automatically deployed via ArgoCD

## Support

Reach out to #platform-engineering on Slack
EOF

git add .
git commit -m "chore: initialize IDP repository"
git remote add origin https://github.com/YOUR_ORG/idp-demo
git push -u origin main

Step 4: Install Backstage

npx @backstage/create-app@latest --path idp-backstage

cd idp-backstage

yarn install

yarn dev

Backstage will start at http://localhost:3000.

Step 5: Create Your First Service Template

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: nodejs-service
  title: Create Node.js Microservice
  description: Scaffold a new Node.js microservice with production-ready CI/CD, monitoring, and security controls
  tags:
    - nodejs
    - microservice
    - backend
spec:
  owner: platform-team
  type: service
  
  parameters:
    - title: Basic Service Information
      required:
        - name
        - owner
      properties:
        name:
          title: Service Name
          type: string
          description: Unique name for your service (lowercase, no spaces).
          pattern: '^[a-z0-9-]+$'
          minLength: 3
          maxLength: 30
        
        owner:
          title: Team/Owner
          type: string
          description: Team responsible for this service
          ui:field: OwnerPicker
        
        description:
          title: Service Description
          type: string
          description: What does this service do?
          maxLength: 200
        
        port:
          title: Service Port
          type: number
          default: 3000
          description: Port the service listens on
  
  steps:
    - id: fetch-base
      name: Fetch Service Template
      action: fetch:template
      input:
        url: ./templates/nodejs-service
        values:
          name: ${{ parameters.name }}
          owner: ${{ parameters.owner }}
          description: ${{ parameters.description }}
          port: ${{ parameters.port }}
    
    - id: publish
      name: Publish to GitHub
      action: publish:github
      input:
        allowedHosts: ['github.com']
        description: ${{ parameters.description }}
        repoUrl: github.com?owner=YOUR_ORG&repo=${{ parameters.name }}
        defaultBranch: main
        protectDefaultBranch: true
    
    - id: register
      name: Register Component in Catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps['publish'].output.repoContentsUrl }}
        catalogInfoPath: /catalog-info.yaml
    
    - id: create-argocd-app
      name: Create ArgoCD Application
      action: argocd:create
      input:
        appName: ${{ parameters.name }}
        argoInstanceName: default
        namespace: default
        repoUrl: ${{ steps['publish'].output.repositoryUrl }}
        path: k8s/
  
  output:
    links:
      - title: Repository
        url: ${{ steps['publish'].output.repositoryUrl }}
      - title: Component in Catalog
        icon: catalog
        entityRef: ${{ steps['register'].output.entityRef }}

Step 6: Define Your Service Deployment Manifests

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${{ service_name }}
  namespace: default
  labels:
    app: ${{ service_name }}
    owner: ${{ owner }}
    managed-by: platform
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: ${{ service_name }}
  template:
    metadata:
      labels:
        app: ${{ service_name }}
        owner: ${{ owner }}
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
        prometheus.io/path: "/metrics"
    spec:
      serviceAccountName: ${{ service_name }}
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
      - name: app
        image: gcr.io/YOUR_PROJECT/${{ service_name }}:latest
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: ${{ port }}
          protocol: TCP
        - name: metrics
          containerPort: 9090
          protocol: TCP
        
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        
        startupProbe:
          httpGet:
            path: /health
            port: http
          initialDelaySeconds: 0
          periodSeconds: 10
          timeoutSeconds: 3
          failureThreshold: 30
        
        livenessProbe:
          httpGet:
            path: /health
            port: http
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        
        readinessProbe:
          httpGet:
            path: /ready
            port: http
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 2
        
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          capabilities:
            drop:
              - ALL
        
        env:
        - name: SERVICE_NAME
          value: ${{ service_name }}
        - name: LOG_LEVEL
          value: "info"
        - name: PORT
          value: "${{ port }}"

---
apiVersion: v1
kind: Service
metadata:
  name: ${{ service_name }}
  namespace: default
  labels:
    app: ${{ service_name }}
spec:
  type: ClusterIP
  selector:
    app: ${{ service_name }}
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: http
  - name: metrics
    protocol: TCP
    port: 9090
    targetPort: metrics

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${{ service_name }}
  namespace: default
  labels:
    app: ${{ service_name }}

Step 7: Connect Everything with ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: idp-services
  namespace: argocd
spec:
  project: default
  
  source:
    repoURL: https://github.com/YOUR_ORG/idp-demo
    targetRevision: HEAD
    path: services/
    plugin:
      name: kustomize
  
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    
    syncOptions:
    - CreateNamespace=true
    - RespectIgnoreDifferences=true
    
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: default
  namespace: argocd
spec:
  sourceRepos:
  - 'https://github.com/YOUR_ORG/*'
  - 'https://github.com/YOUR_ORG/idp-demo'
  
  destinations:
  - namespace: 'default'
    server: 'https://kubernetes.default.svc'
  - namespace: 'argocd'
    server: 'https://kubernetes.default.svc'

Apply to your cluster:

kubectl apply -f argocd-app.yaml

Platform Best Practices

1. Golden Path First

Start with a single, opinionated happy path. Don't try to support every possible architecture on day one.

Benefits:

Easier to maintain
Predictable performance
Simpler troubleshooting
New developers get up to speed faster

2. Guard Rails, Not Roadblocks

Use policy enforcement to guide teams toward best practices, not block them.

3. Self-Service First, Escalation Second

Design the platform so 90% of use cases are self-service. Only complex scenarios need human intervention.

Target SLA:

90% of deployments: Self-service, < 5 minutes
9% of deployments: Template with minor modifications, < 15 minutes
1% of deployments: Manual, > 15 minutes

4. Measure and Iterate

Track metrics that matter:

Developer Productivity: Deployment frequency, lead time for changes
Operational Efficiency: Mean time to recovery, change failure rate
Platform Health: Platform uptime, user satisfaction
Adoption: Platform usage metrics, template utilization

5. Internal Marketing

Tools don't adopt themselves. Invest in:

Internal documentation and runbooks
Weekly training sessions
Slack/Discord channels for support
Internal engineering blog or newsletter
Feedback loops (surveys, retrospectives)
Success stories from early adopters

Common Pitfalls to Avoid

Over-Engineering Early

Don't aim for perfect high availability on day one. Start simple and iterate.

Ignoring Developer Feedback

Your platform is only good if developers use it. Regularly gather and act on feedback.

Treating It as Infrastructure-Only

Platform engineering is 40% infrastructure, 60% product management and UX.

Inconsistent Standards

If your platform allows too many variations, you lose the consistency benefits.

Neglecting Security

Bake in security from day one, not as an afterthought.

IDP Maturity Model

Level 0: Manual (No Platform)

Developers provision infrastructure manually
Time to deploy: 1-3 days

Level 1: Documented (Getting Started)

Documented runbooks and procedures
A backstage catalogue exists
Time to deploy: 4-8 hours

Level 2: Automated (This Guide)

GitOps-driven deployments
Self-service portal
Time to deploy: 10-30 minutes

Level 3: Intelligent (Next Steps)

ML-driven resource optimization
Automated incident response
Time to deploy: < 5 minutes

Level 4: Autonomous (Future)

Fully autonomous scaling and healing
Predictive capacity planning
Time to deploy: Automatic on commit

Essential Tools & Technologies

Portal & Discovery

Backstage (Open Source)
Cloudsmith
Spotify's Backstage Plugins

GitOps & Deployment

ArgoCD (Open Source)
Flux (Open Source)
Helm (Open Source)

Infrastructure as Code

Terraform (Open Source)
Pulumi
Crossplane (Open Source)

Policy & Security

OPA/Gatekeeper (Open Source)
Kyverno (Open Source)
Snyk

Observability

Prometheus (Open Source)
Grafana (Open Source)
Loki (Open Source)
Jaeger (Open Source)

Cost Management

Kubecost
Cloudability
Vantage

What's Next?

After implementing this basic IDP:

Expand service templates – Add more patterns
Implement comprehensive observability – Integrate Prometheus and Grafana
Add cost management – Integrate Kubecost
Implement advanced policies – Security scanning and compliance validation
GitOps for infrastructure - Use Terraform + ArgoCD
Multi-cluster support – Expand to multiple regions
Integration marketplace – Allow teams to self-select integrations

Troubleshooting

ArgoCD Application not syncing

kubectl get applications -n argocd
kubectl describe application idp-services -n argocd
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server -f

Backstage template not appearing

curl https://your-git-repo/catalog-info.yaml | yq .
docker logs backstage-container
curl -X POST http://localhost:3000/api/catalog/refresh

Services not deploying

kubectl get pods -n default
kubectl describe deployment service-name -n default
kubectl get endpoints service-name -n default

Conclusion

Platform engineering isn't a destination; it's a journey. Start small with one golden path, measure success, and iterate based on feedback.

The teams that invest in platform engineering early will have a significant competitive advantage:

⚡ Faster deployments (10 minutes vs hours)
😊 Happier developers (reduced cognitive load)
🔒 More secure infrastructure (built-in guardrails)
💰 Better-controlled costs (visibility and governance)

Remember: A great platform is one that developers actually use. Keep iterating based on feedback, and you'll build something truly valuable.

Additional Resources

Backstage Official Documentation: https://backstage.io/docs/overview/what-is-backstage
ArgoCD User Guide: https://argo-cd.readthedocs.io/en/stable/
Gartner Platform Engineering Report: https://www.gartner.com/en/documents/platform-engineering
CNCF Platform Engineering Whitepaper: https://www.cncf.io/blog/2023/10/12/platform-engineering-maturity-model/
Spotify's Backstage Showcase: https://www.youtube.com/watch?v=85TQEpYaTJ0
ArgoCD Best Practices: https://argo-cd.readthedocs.io/en/stable/operator-manual/best\_practices/

Did This Help?

If you found this guide helpful, please share it with your team and follow me for more cloud-native and DevOps content.

What's your experience with platform engineering? Have you built an IDP? Share your thoughts in the comments below!

If you're building an IDP or have questions about the implementation, feel free to connect with me on Twitter or LinkedIn.

About the author: Cloud & DevOps blogger passionate about helping teams build efficient, secure, and scalable infrastructure. Currently exploring platform engineering, FinOps, and cloud-native security.

Tags: #PlatformEngineering #DevOps #Kubernetes #Backstage #ArgoCD #CloudNative #DevSecOps #Infrastructure #DeveloperExperience

Command Palette