Data Integration with Apps
Integrating data from a business application like NetSuite involves three stages: data discovery (identify what you need), data pipelines (fetch it reliably), and data transformation (shape it for your product). Each stage has distinct ownership and requires answering specific technical and business questions before moving forward.
What Are the Steps to Integrate Data from a Business Application?
The integration activity can be split into three sections:
- Data discovery — Identify the entities from the source that constitute the data you need
- Data pipelines — Fetch data from the source system reliably and on schedule
- Data integration / transformation — Model and transform data to work with your product
Stage 1: How Do You Discover What Data You Need?
Ownership: Product Manager and Business Analyst
Data discovery identifies the data elements that constitute the dataset you require. For example, NetSuite exposes over 800 data entities. You need to answer:
- Are you familiar with the application, or will you need a subject-matter expert?
- Which entities, APIs, or database tables constitute the complete dataset (e.g., General Ledger, Sales)?
- How do you define unique records for deduplication?
- What refresh frequency does the source application support?
Stage 2: How Do You Set Up Data Pipelines?
Ownership: Product Manager and Data Engineering Team
The tech team needs answers to the following to build reliable pipelines:
- Which version of the source app is in use? Cloud-based or on-prem?
- How is data exposed — via APIs, databases, emails, or SFTP?
- Does the customer have the right licenses for data sharing?
- How often should data sync — every 5 minutes or every 24 hours?
- What is the estimated data volume?
- How will you verify and reconcile data against the source?
- What are the data retention and security requirements?
Stage 3: How Do You Transform Data for Your Product?
Ownership: Product Manager, App Software Team, Data Engineering Team
With stages 1 and 2 complete, you know what data to fetch and how. In this stage, you transform the data into the structure your product requires. For example, you might combine finance data (general ledger) with operations data and push the merged dataset into your application for end-user consumption.
What Happens After Go-Live? Integration Hygiene
Once the integration is live, maintaining its health requires ongoing attention:
- Pipeline-to-app coordination: If data pipelines fail, the application won't have the latest data. The data engineering and app teams must stay in sync.
- Data validation: Without reconciliation checks, data duplication issues can cause source and destination to diverge quickly.
- Source availability: The data source can go down unexpectedly.
- Schema changes: The source application may change API schemas, which can break downstream pipelines. See Handling Schema Evolution for how DataStori handles this automatically.
How Does DataStori Help?
DataStori is a data ingestion tool that handles Stage 2 — it reduces the time to go live with built-in automations for data ingestion, deduplication, quality checks, and schema evolution. Add a new application in under 30 minutes.
Frequently Asked Questions
How long does a typical data integration take?
Data discovery usually takes 1-2 weeks depending on application complexity. Pipeline setup can take days (with DataStori) to weeks (with custom engineering). Transformation varies by business logic complexity but typically takes 2-4 weeks.
Do I need a NetSuite expert for data discovery?
It depends on your team's familiarity with the application. NetSuite exposes over 800 data entities, so having a subject-matter expert helps identify the correct tables and relationships for your use case.
What happens if the source app changes its API schema?
Schema changes can break downstream pipelines. DataStori handles schema evolution automatically — new columns are added, dropped columns are preserved with null values, and you are alerted to any changes. See our article on schema evolution for details.