How to Fetch Data from an Application
Business applications share data through six primary methods: APIs, databases, webhooks, email reports, cloud storage (SFTP/OneDrive/Google Drive/S3), and custom scripts. APIs are the most reliable and preferred method for automated data pipelines. This article ranks each method and explains the trade-offs.
What Are the Options for Fetching Data?
Depending on the application, it will share data with you in one of the following ways, listed in order of preference for building reliable data pipelines:
- API — Programmatic, structured, and well-documented access to application data
- Database — Direct access to the application's underlying data store
- Webhook — Event-driven push notifications when data changes
- Email reports — Scheduled data exports sent via email
- Cloud storage — Data files deposited in SFTP, OneDrive, Google Drive, or S3
- Custom scripts — Scripts running inside the customer's application environment
How Do These Data Fetch Methods Compare?
| Method | Reliability | Automation | Real-Time | Complexity |
|---|---|---|---|---|
| API | High | Full | Near real-time | Medium |
| Database | High | Full | Yes | Medium-High |
| Webhook | Medium | Event-driven | Yes | Medium |
| Email reports | Low | Partial | No (scheduled) | High (parsing) |
| Cloud storage | Medium | Partial | No (batch) | Low |
| Custom scripts | Variable | Full | Variable | High |
Why Are APIs the Preferred Method?
APIs provide structured, documented, and versioned access to application data. They support pagination, filtering, and incremental fetches, making them ideal for automated data pipelines. Most modern SaaS applications expose REST or GraphQL APIs that can be consumed by ELT tools like DataStori.
When Should You Use Database Access?
Direct database access is useful when the application does not expose a comprehensive API or when you need to query large volumes of historical data. However, it requires database credentials, network access, and careful query design to avoid impacting the application's performance.
What Are the Limitations of Webhooks?
Webhooks push data to you when events occur, which is great for real-time updates. However, they can miss events during downtime, require you to host an endpoint, and don't support historical data fetching. They work best as a supplement to API-based ingestion.
For a deeper look at the full integration workflow from data discovery to transformation, see Data Integration with Apps. For details on how these methods authenticate, see API Authentication Methods.
Frequently Asked Questions
Which data fetch method does DataStori support?
DataStori supports API-based ingestion (REST and GraphQL), database connections, SFTP, email attachment parsing, and cloud storage (S3, Google Drive, SharePoint). APIs are the recommended and most common method.
Can I combine multiple fetch methods for one application?
Yes. Some applications expose certain data only via API and other data via database views or file exports. You can combine methods to get complete coverage of the data you need.