Data Sources
Add, run, monitor, edit and delete the feeds and scrapers that bring articles into your organization.
A data source is a feed or scraper that brings articles in: a website's feed, X accounts, a news wire subscription or a council meeting portal. Data Sources is where you add them, see whether they are delivering, and run, edit or remove them. Everything downstream, from events to pitches, starts from what they collect.
Every member can open the list and a source's page, Viewers included. The Member role or above can
add, edit, enable, disable and run a source (the datasource:create, datasource:update and
datasource:run permissions). Deleting needs the Owner or Admin role (datasource:delete). The page
sits under Discovery in the sidebar.
Add a data source
The first-time walk-through, with an RSS feed, is Add a data source. In short:
- Choose Add Data Source.
- Choose a type. The form offers RSS Feed, X · Follows, X · Thread, xAI Search, Datafactory, DigiChief, AP Media, Reuters, AFP and Council Meetings, each with a one-line description. Each type asks for its own settings; Name is required.
- Optionally choose a Category and a Location.
- Wire and vendor types need a Credential. Choose one from the list, or Create a credential if none exists; see Credentials. The form says you can also attach it later on the data source's page. Authenticating an RSS feed needs the permission to manage credentials, so a Member is asked to have an administrator attach one.
- Choose Create Data Source.
An AFP picture source shows Picture harvesting not ready: you can set it up now and it starts collecting once picture harvesting is released.
Read the list
Each row shows the Name, Type, Category, Location, Status and when it was Updated. The status is one of:
| Status | Meaning |
|---|---|
| Active | Enabled, and harvesting normally. |
| Awaiting first run | Enabled, and no run has completed yet. |
| Errors | Enabled, and its last run reported errors. |
| Stale | Enabled, but it has stopped producing: it has not run within its expected window. |
| Paused | Disabled, so it does not harvest. |
The header counts the sources that need attention, the Errors and Stale ones. Search with Search by name…, or choose + Add filter for Type, Category, Status, Health, Location or Created. Choose Clear filters when nothing matches.
Run, edit or pause a source
Open a source to see its Configuration and Operations. Run now and Delete are also in a row's Data source actions menu; Edit and Disable are on the source's page only:
- Run now starts a harvest straight away; Run started confirms it. Otherwise a source harvests on its own schedule.
- Edit changes its settings, and Save keeps them.
- Disable stops harvesting until you Enable it again, and the status shows Paused.
- Delete removes the source together with its harvested articles and run history, and cannot be undone.
Under Operations, each harvest run shows whether it is Running or Done, when it Started and Finished, and its Stats: Stored, Updated, Not modified, Deduplicated, Rate limited and Errors. No harvest runs yet. means it has not run. View articles on a source's page opens Articles for it. If a source stays Errors or Stale and uses a credential, open Credentials and look at that credential's Health.
Other ways to manage data sources
- CLI:
nuvelics datasourceandnuvelics run. - API: Data sources and Harvest runs.
- AI assistants: the data source tools.
- The data source types and the credentials each needs: Data source types.