Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
152 lines (100 loc) · 5.06 KB

DEVELOPMENT.md

File metadata and controls

152 lines (100 loc) · 5.06 KB
Raw

Development

Setup

First things first, you'll need to fork and clone the project to your local machine.

git clone https://github.com/ecosyste-ms/science.git

The project uses ruby on rails which have a number of system dependencies you'll need to install.

Once you've got all of those installed, from the root directory of the project run the following commands:

bundle install
bundle exec rake db:create
bundle exec rake db:migrate
rails server

You can then load up http://localhost:3000 to access the service.

Docker

Alternatively you can use the existing docker configuration files to run the app in a container.

Run this command from the root directory of the project to start the service.

docker-compose up --build

You can then load up http://localhost:3000 to access the service.

For access the rails console use the following command:

docker-compose exec app rails console

Runing rake tasks in docker follows a similar pattern:

docker-compose exec app rake projects:sync

Importing data

The default set of supported data sources are listed in db/seeds.rb and can be automatically enabled with the following rake command:

rake db:seed

You can then start syncing data for each source with the following command, this may take a while:

rake projects:sync

Tests

The applications tests can be found in test and use the testing framework minitest.

You can run all the tests with:

rails test

Rake tasks

The applications rake tasks can be found in lib/tasks.

You can list all of the available rake tasks with the following command:

rake -T

Data Import & Management Tasks

The platform includes several rake tasks for importing and managing scientific software data from various sources.

Core Import Tasks

JOSS & Academic Papers:

rake projects:import_joss                    # Import all JOSS papers and their software repositories
rake projects:import_papers                  # Import from papers.ecosyste.ms (academic citations)
rake projects:import_all_joss_topics         # Import top 50 topics from JOSS papers (GitHub topics)
rake projects:import_all_joss_keywords       # Import top 50 keywords from JOSS papers (package keywords)

Package Registries:

rake projects:import_cran                    # Import R packages from CRAN with GitHub repos
rake projects:import_bioconductor            # Import bioinformatics packages from Bioconductor
rake projects:import_conda_forge             # Import conda-forge packages with GitHub repos

GitHub Discovery:

rake projects:import_github_topic[topic]     # Import projects by GitHub topic (e.g., science, astronomy)
rake projects:import_github_owner[owner]     # Import all repositories from a GitHub owner/org
rake projects:import_all_github_owners       # Import from all known scientific GitHub owners (min_score=50)
rake projects:import_package_keyword[keyword] # Import packages by keyword

Other Sources:

rake projects:import_ost                     # Import from Open Sustainable Technology
rake projects:discover                       # Auto-discover via topics and keywords

Project Management

rake projects:sync                           # Sync least recently synced projects (500 at a time)
rake projects:sync_dependencies              # Update dependency information across projects
rake projects:fetch_brief                    # Enqueue repository scans on the Brief queue

Brief scanning uses a low-weight Sidekiq queue. See docs/brief-scanning.md for cohort and sharding options.

JOSS Vocabulary Analysis

The application builds a reusable vocabulary model from JOSS project text and stores it in PostgreSQL. See docs/joss-vocabulary.md for the corpus, weighting, scoring, and cache behavior.

bundle exec rake joss_vocabulary:rebuild     # Build and activate a new model
bundle exec rake joss_vocabulary:stats       # Show the active model and source counts

Run bundle exec rake joss_vocabulary:rebuild once after the model-table migration. app.json schedules later rebuilds each Sunday.

Example Import Workflow

# 1. Build the JOSS vocabulary model (one-time setup)
bundle exec rake joss_vocabulary:rebuild

# 2. Import core scientific software
rake projects:import_joss
rake projects:import_papers
rake projects:import_all_joss_topics

# 3. Import from package registries
rake projects:import_cran
rake projects:import_bioconductor
rake projects:import_conda_forge

# 4. Sync and analyze imported projects
rake projects:sync

# 5. Check JOSS vocabulary model statistics
bundle exec rake joss_vocabulary:stats

Deployment

A container-based deployment is highly recommended, we use dokku.com.